From steve at pearwood.info  Wed Jul  1 00:47:01 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Jul 2009 08:47:01 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com>
Message-ID: <200907010847.02606.steve@pearwood.info>

On Wed, 1 Jul 2009 05:19:07 am Tarek Ziad? wrote:
> 2009/6/30 Guido van Rossum <guido at python.org>:
...
> > So what *is* the distutils-sig consensus?
>
> The consensus is to have one and only one way to install
> distributions in Python,

"One and only one way"? Shouldn't that be "one obvious way"?

There is a big difference -- the first implies that Python should 
actively prohibit (somehow!) or at least inhibit any other install 
mechanisms, while the second implies that there should be a preferred, 
obvious, but not compulsory, mechanism.

If you say "one and only one", and I take you at your word, then I can 
only assume that you want to actively prevent me from manually 
dropping .py files into my PYTHONPATH and having them work. I would 
guess that's probably not what you mean, but that's what it sounds like 
you're proposing. I don't like guessing -- would you please clarify 
what you mean?


Some questions and comments regarding the PEP:

Rationale:
http://www.python.org/dev/peps/pep-0376/#id13

"There are too many ways to do it."

Why is having multiple ways of adding distributions a problem to be 
solved? On my Linux system, I can add packages with rpm/yum, or I can 
compile them from source and manage them myself. I possibly even could 
(but never have!) install apt-get and use it to manage packages.

Another rationale which should be added:

There is no standard way of uninstalling distributions.


How distributions are installed:
http://www.python.org/dev/peps/pep-0376/#id14

"The problem is that many people use easy_install (from the setuptools 
project [4]) or pip [5] to install their packages, and these 
third-party tools do not install packages in the same way that 
Distutils does"

Why is that a problem to be solved?


Uninstall information:
http://www.python.org/dev/peps/pep-0376/#id15

"Under some circumstances, you might not be able to know for sure that 
you have removed everything, or that you didn't break another 
distribution by removing a file that is shared among several 
distributions."

I don't see how this proposal will help in the second case. If you 
install distribution Spam, containing file spam.py, and then install 
distribution Ham, which requires spam.py, what is to prevent you from 
removing Spam and breaking Ham?

If you don't propose a solution for the dependency problem, you should 
say so.


The RECORD format:
http://www.python.org/dev/peps/pep-0376/#id19

"Each record is composed of three elements.
...
the MD5 hash of the file, encoded in hex. Notice that pyc and pyo 
generated files will not have a hash because they are automatically 
produced from py files."

What if your distribution is not a source distribution and only provides 
pyc and pyo files?



-- 
Steven D'Aprano

From ziade.tarek at gmail.com  Wed Jul  1 01:34:53 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 01:34:53 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <200907010847.02606.steve@pearwood.info>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com>
	<200907010847.02606.steve@pearwood.info>
Message-ID: <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com>

On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano<steve at pearwood.info> wrote:
> On Wed, 1 Jul 2009 05:19:07 am Tarek Ziad? wrote:
>> 2009/6/30 Guido van Rossum <guido at python.org>:
> ...
>> > So what *is* the distutils-sig consensus?
>>
>> The consensus is to have one and only one way to install
>> distributions in Python,
>
> "One and only one way"? Shouldn't that be "one obvious way"?
>
> There is a big difference -- the first implies that Python should
> actively prohibit (somehow!) or at least inhibit any other install
> mechanisms, while the second implies that there should be a preferred,
> obvious, but not compulsory, mechanism.
>
> If you say "one and only one", and I take you at your word, then I can
> only assume that you want to actively prevent me from manually
> dropping .py files into my PYTHONPATH and having them work. I would
> guess that's probably not what you mean, but that's what it sounds like
> you're proposing. I don't like guessing -- would you please clarify
> what you mean?

one and only one way to install a distribution using its setup.py script,
among the ways provided by easy_install, distutils and pip.
e.g. avoiding having different format/location for the generated .egg.info;

>
>
> Some questions and comments regarding the PEP:
>
> Rationale:
> http://www.python.org/dev/peps/pep-0376/#id13
>
> "There are too many ways to do it."
>
> Why is having multiple ways of adding distributions a problem to be
> solved? On my Linux system, I can add packages with rpm/yum, or I can
> compile them from source and manage them myself. I possibly even could
> (but never have!) install apt-get and use it to manage packages.

If you choose not to manually manage them, the packaging system you use
is installing files in standard places where you expect to find them,
and provides
tools to know what is installed.

Right now we don't have this standard in Python stdlib to interact
with the distributions in a Python installation.  And since packaging systems
that are used out there could use the same standard, since we would like
to make distutils a reference package for this kind of need, it seems
like a good
idea to define this standard in here. then each packaging system is free
to implement its features on the top of it, ala wsgiref.

>
> Another rationale which should be added:
>
> There is no standard way of uninstalling distributions.
>
>
> How distributions are installed:
> http://www.python.org/dev/peps/pep-0376/#id14
>
> "The problem is that many people use easy_install (from the setuptools
> project [4]) or pip [5] to install their packages, and these
> third-party tools do not install packages in the same way that
> Distutils does"
>
> Why is that a problem to be solved?

Being able to retrieve the metadata of an installed distribution, or
the list of its installed files,
no matter which tool installed it.

>
>
> Uninstall information:
> http://www.python.org/dev/peps/pep-0376/#id15
>
> "Under some circumstances, you might not be able to know for sure that
> you have removed everything, or that you didn't break another
> distribution by removing a file that is shared among several
> distributions."
>
> I don't see how this proposal will help in the second case. If you
> install distribution Spam, containing file spam.py, and then install
> distribution Ham, which requires spam.py, what is to prevent you from
> removing Spam and breaking Ham?
>
> If you don't propose a solution for the dependency problem, you should
> say so.

This problem is solved as described later in the PEP, with the API
that allows you to get the
list of the distributions that use a given file. (thanks to the RECORD files)

If Spam and Ham use smap.py, and if you uninstall Spam, this file will
not be removed
because the API will tell you its used in both distributions.

>
>
> The RECORD format:
> http://www.python.org/dev/peps/pep-0376/#id19
>
> "Each record is composed of three elements.
> ...
> the MD5 hash of the file, encoded in hex. Notice that pyc and pyo
> generated files will not have a hash because they are automatically
> produced from py files."
>
> What if your distribution is not a source distribution and only provides
> pyc and pyo files?
>

Good question, I have never created such distribution.
Aren't they read-only files ? What happens if someone change them, do we
really want to keep them since it breaks the distribution ? How they
can be changed in the first place,
without the source ?

Tarek

From ziade.tarek at gmail.com  Wed Jul  1 01:46:00 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 01:46:00 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <h2dqqe$fp1$1@ger.gmane.org>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
Message-ID: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>

On Tue, Jun 30, 2009 at 10:06 PM, Scott David
Daniels<Scott.Daniels at acm.org> wrote:
> Tarek Ziad? wrote:
>>
>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>>>
>>> [1] I'd actually like it if the PEP defined an uninstall command -
>>> something like "python -m distutils.uninstall packagename". It can be
>>> as minimalist as you like, but I'd like to see it present.
>>
>> it's already there:
>>
>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function
>
> That (at least as I read it) is a function, not a command.
> If it is a command, give an example of its use from the command line
> for us poor "don't want to research" people. ?If the following works:
>
> ? ?$ python setup.py uninstall some_package
>
> Then explicitly say so for us poor schlubs.
>

Right, I'll add that. Although it will be a reference implementation only.

From ziade.tarek at gmail.com  Wed Jul  1 01:49:58 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 01:49:58 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4A7F88.7080609@gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<20090630174159.731CE3A4099@sparrow.telecommunity.com>
	<94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com>
	<4A4A7F88.7080609@gmail.com>
Message-ID: <94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com>

On Tue, Jun 30, 2009 at 11:11 PM, Nick Coghlan<ncoghlan at gmail.com> wrote:
>>
>>
>> previous formats will not be supported but that won't break anything
>> of course since the new APIs will work only over the distribution
>> installed with the new version of distutils.
>
> To address PJE's question in the PEP, it may be worth expanding on this
> in the backwards compatibility section explaining how the new distutils
> metadata system avoids getting confused by the old pre-standardisation
> installation formats (e.g. it may be that the directory names and/or
> filenames all deliberately differ from current approaches precisely so
> they can coexist without interfering with each other)
>

I'll work on making it clearer,

> Also, I find the following paragraph (near the start of the section
> linked above) confusing:
>
> [PEP 376]
>> Notice that this change is based on the standard proposed by
>> EggFormats, although this standard proposes two ways to install
>> files:
>>
>> * A self-contained directory that can be zipped or left unzipped and
>> contains the distribution files and the .egg-info directory.
>> * A distinct .egg-info directory located in the site-packages directory.
>
> This is unclear as to what "this standard" is referring to (since the
> PEP itself is proposing a standard, but the sentence is also referring
> to the existing EggFormats de facto standard). If the PEP only supports
> the latter of the two options (which appears to be the case) it should
> say so explicitly.

ok

>
> Another minor nit from the same section:
>
> [PEP 376]
>> Any '-' characters are currently replaced with '_'.
>
> This should say something like "Any '-' characters other than the one in
> 'egg-info' and the one separating the name from the version number are
> included in the replacement of non-alphanumeric characters with '_'"

ok

>
> Other questions/comments:
>
> - What is a "local absolute path"? Absolute path I understand, relative
> path I understand, but "local absolute" is a novel term to me.

local means that the "/" separator that is used in the RECORD file for example,
no matter what platform you are on, is translated using the local separator
(/ or \)

I'll make it clearer,

Regards
Tarek

From ziade.tarek at gmail.com  Wed Jul  1 02:07:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 02:07:11 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
Message-ID: <94bdd2610906301707t12d0e879hd7ccfb47abb8a710@mail.gmail.com>

On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>> We said during the summit at Pycon that we wanted this feature in
>> Distutils, (Guido said so)
>
> "We" in this context denotes the people at the summit. Please remember
> that people who weren't there still have an opinion - and it may well
> differ. I'm not saying that it *does* differ, just that the view of
> the summit should not be viewed as conclusive (unless the way Python
> is developed is very different from how I understand it).

Sorry I didn't mean it this way.  The summit was just the kickoff for this PEP,
but with the strong desire to have a standard + some API for it.

Also, notice that I put in "We" also the people that answered the survey
I did before the summit.
(http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/)

I'll make other fixes in the PEP with the feedback you gave, thanks

Tarek

From pje at telecommunity.com  Wed Jul  1 02:13:19 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 30 Jun 2009 20:13:19 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.co
 m>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com>
	<200907010847.02606.steve@pearwood.info>
	<94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com>
Message-ID: <20090701001021.F1C553A4099@sparrow.telecommunity.com>

At 01:34 AM 7/1/2009 +0200, Tarek Ziad? wrote:
>On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano<steve at pearwood.info> wrote:
> > I don't see how this proposal will help in the second case. If you
> > install distribution Spam, containing file spam.py, and then install
> > distribution Ham, which requires spam.py, what is to prevent you from
> > removing Spam and breaking Ham?
> >
> > If you don't propose a solution for the dependency problem, you should
> > say so.
>
>This problem is solved as described later in the PEP, with the API
>that allows you to get the
>list of the distributions that use a given file. (thanks to the RECORD files)
>
>If Spam and Ham use smap.py, and if you uninstall Spam, this file will
>not be removed
>because the API will tell you its used in both distributions.

That's not the scenario he's talking about.  He's talking about the 
case where Ham has an 'install_requires' of Spam.  That is, a runtime 
dependency, not a shared file.


>Good question, I have never created such distribution.
>Aren't they read-only files ?

setuptools' bdist_egg command has an option to exclude source from an 
.egg, but it doesn't do anything special with permissions.  I don't 
think any other current install tools support source-free installation.


From pje at telecommunity.com  Wed Jul  1 02:21:18 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 30 Jun 2009 20:21:18 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4A7F88.7080609@gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<20090630174159.731CE3A4099@sparrow.telecommunity.com>
	<94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com>
	<4A4A7F88.7080609@gmail.com>
Message-ID: <20090701001819.8ABDF3A4099@sparrow.telecommunity.com>

At 07:11 AM 7/1/2009 +1000, Nick Coghlan wrote:
>To address PJE's question in the PEP, it may be worth expanding on this
>in the backwards compatibility section explaining how the new distutils
>metadata system avoids getting confused by the old pre-standardisation
>installation formats (e.g. it may be that the directory names and/or
>filenames all deliberately differ from current approaches precisely so
>they can coexist without interfering with each other)

In particular, it should be explicitly noted that the API does not 
support .egg files (or directories), and thus cannot substitute for 
the pkg_resources API, as far as detecting installed packages.

That is, users who need to unequivocally detect the presence/version 
of a project on sys.path, etc. will still have to use pkg_resources.

(Because pkg_resources supports the format proposed by the PEP, but 
the PEP does not support .egg files and directories.)


From ziade.tarek at gmail.com  Wed Jul  1 02:19:53 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 02:19:53 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <20090701001021.F1C553A4099@sparrow.telecommunity.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com>
	<200907010847.02606.steve@pearwood.info>
	<94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com>
	<20090701001021.F1C553A4099@sparrow.telecommunity.com>
Message-ID: <94bdd2610906301719u78734951reeb46bef4e1e302a@mail.gmail.com>

2009/7/1 P.J. Eby <pje at telecommunity.com>:
> At 01:34 AM 7/1/2009 +0200, Tarek Ziad? wrote:
>>
>> On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano<steve at pearwood.info>
>> wrote:
>> > I don't see how this proposal will help in the second case. If you
>> > install distribution Spam, containing file spam.py, and then install
>> > distribution Ham, which requires spam.py, what is to prevent you from
>> > removing Spam and breaking Ham?
>> >
>> > If you don't propose a solution for the dependency problem, you should
>> > say so.
>>
>> This problem is solved as described later in the PEP, with the API
>> that allows you to get the
>> list of the distributions that use a given file. (thanks to the RECORD
>> files)
>>
>> If Spam and Ham use smap.py, and if you uninstall Spam, this file will
>> not be removed
>> because the API will tell you its used in both distributions.
>
> That's not the scenario he's talking about. ?He's talking about the case
> where Ham has an 'install_requires' of Spam. ?That is, a runtime dependency,
> not a shared file.

Ah, right sorry I misunderstood... They are no plans to handle
dependency installation / uninstallation / managment  at distutils
level.
so if you remove Ham, it will not check what distributions use it. So
yes, I'll add a note on this,

That said, the APIs will be powerfull enough for a third-party package
managers to handle this
case by throwing for example a warning or an exception.

From kevin at bud.ca  Wed Jul  1 04:12:19 2009
From: kevin at bud.ca (Kevin Teague)
Date: Tue, 30 Jun 2009 19:12:19 -0700
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
Message-ID: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>


On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote:

> On Tue, Jun 30, 2009 at 10:06 PM, Scott David
> Daniels<Scott.Daniels at acm.org> wrote:
>> Tarek Ziad? wrote:
>>>
>>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore at gmail.com>  
>>> wrote:
>>>>
>>>> [1] I'd actually like it if the PEP defined an uninstall command -
>>>> something like "python -m distutils.uninstall packagename". It  
>>>> can be
>>>> as minimalist as you like, but I'd like to see it present.
>>>
>>> it's already there:
>>>
>>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall- 
>>> function
>>
>> That (at least as I read it) is a function, not a command.
>> If it is a command, give an example of its use from the command line
>> for us poor "don't want to research" people.  If the following works:
>>
>>    $ python setup.py uninstall some_package
>>
>> Then explicitly say so for us poor schlubs.
>>
>
> Right, I'll add that. Although it will be a reference implementation  
> only.
>

Uninstall as a command feels a little weird. Since "python setup.py  
[some-command]" implies that the setup.py contains information about  
the distribution that the command is being applied to. So instead of:

$ python setup.py uninstall some_package

It could just be:

$ python setup.py uninstall

Except then you'd need to have a complete distribution kicking around  
with which to run the "python setup.py uninstall" command just to tell  
the uninstall command the distribution name you want to uninstall. But  
then with the other uninstall format you could uninstall any  
distribution from within any other distribution, which is convenient,  
but weird ... e.g.:

$ cd Spam-1.0/
$ python setup.py uninstall Foo

Although even the other version of the command could do weird stuff:

$ cd Spam-1.0/
$ python setup.py install
$ cd ../Spam-2.0/
$ python setup.py uninstall

Which would presumably remove the Spam 1.0 distribution when run from  
the 2.0 version of it! Or perhaps this command should only allow  
uninstall to be run from a distribution whose name and version match  
the one that it was installed from ...

I dunno what the right solution is. My two-cents is either to punt and  
only include an uninstall function as currently proposed, or for only  
supporting some form of the "python setup.py uninstall" style since I  
would guess that the most common use-case for uninstall is: user  
downloads a distribution, runs "python setup.py install", tries out  
the package, decides they don't like package, then runs "python  
setup.py uninstall" to restore their python back to it's original  
state. For doing anything more complex than that, people should be  
encouraged to use another one of the existing tools for managing their  
distributions, IMHO.


From Scott.Daniels at Acm.Org  Wed Jul  1 05:06:34 2009
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Tue, 30 Jun 2009 20:06:34 -0700
Subject: [Python-Dev] PEP 376
In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<h2dqqe$fp1$1@ger.gmane.org>	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
Message-ID: <h2ejem$900$1@ger.gmane.org>

Kevin Teague wrote:
> On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote:
>> On Tue, Jun 30, 2009 at 10:06 PM, Scott David
>> Daniels<Scott.Daniels at acm.org> wrote:
>>> Tarek Ziad? wrote:
>>>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>>>>> [1] I'd actually like it if the PEP defined an uninstall command -
>>>>> something like "python -m distutils.uninstall packagename". It can be
>>>>> as minimalist as you like, but I'd like to see it present.
>>>> it's already there:
>>>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function
>>> That (at least as I read it) is a function, not a command.
>>> If it is a command, give an example of its use from the command line
>>> for us poor "don't want to research" people.  If the following works:
>>>    $ python setup.py uninstall some_package
>>> Then explicitly say so for us poor schlubs.
>> Right, I'll add that. Although it will be a reference implementation 
>> only.
> Uninstall as a command feels a little weird....
> I dunno what the right solution is. My two-cents is either to punt and 
> only include an uninstall function as currently proposed, or for only 
> supporting some form of the "python setup.py uninstall" style....

But for us poor schlubs, we want you brilliant packagers to actually
come to a hard decision.  If you want approval either admit you have no
solution in the PEP (and detail the issues that prevent a decision),
provide a minimally acceptable command, or expect that nobody sees the
value of what you propose....

--Scott David Daniels
Scott.Daniels at Acm.Org


From guido at python.org  Wed Jul  1 06:09:35 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Jun 2009 21:09:35 -0700
Subject: [Python-Dev] PEP 376
In-Reply-To: <ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> 
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> 
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
Message-ID: <ca471dc20906302109q63594f38td9fd12ad119d970f@mail.gmail.com>

2009/6/30 Guido van Rossum <guido at python.org>:
> On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan<ncoghlan at gmail.com> wrote:
>> Martin v. L?wis wrote:
>>>> If no one objects, I'd like to push PEP 376 in the "accepted" status
>>>> and go ahead with its implementation,
>>>> with continuous feedback at Distutils-SIG as we did to build it.
>>>
>>> I think this isn't quite the process. In the past, every PEP required
>>> BDFL pronouncement, which you should now seek.
>>
>> Agreed. While Guido is highly likely to just accept the distutils-sig
>> consensus on something like this, that doesn't eliminate the need for
>> him to actually *say* that he is approving the PEP on that basis.
>
> Sure. :-)
>
> So what *is* the distutils-sig consensus?
>
> And is there consensus outside of it? (Remember the ipaddr debacle.
> It's easy for people to miss an important PEP.)

Looks like the discussion isn't quite over yet...

I did read about a third of the PEP but still feel pretty lost about
how it all fits together; I won't have time to read more until next
week, probably. I made some minor edits for typos and grammar (being
PEP editor and all :-).

I noted an inconsistency: first you say that the RECORD file uses the
excel dialect, but at the end of the same section you say it uses the
default csv settings. Sounds like you need to delete one or the other.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fuzzyman at voidspace.org.uk  Wed Jul  1 10:20:33 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 01 Jul 2009 09:20:33 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<h2dqqe$fp1$1@ger.gmane.org>	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
Message-ID: <4A4B1C51.3060601@voidspace.org.uk>

Kevin Teague wrote:
>
> On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote:
>
>> On Tue, Jun 30, 2009 at 10:06 PM, Scott David
>> Daniels<Scott.Daniels at acm.org> wrote:
>>> Tarek Ziad? wrote:
>>>>
>>>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore at gmail.com> 
>>>> wrote:
>>>>>
>>>>> [1] I'd actually like it if the PEP defined an uninstall command -
>>>>> something like "python -m distutils.uninstall packagename". It can be
>>>>> as minimalist as you like, but I'd like to see it present.
>>>>
>>>> it's already there:
>>>>
>>>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function
>>>
>>> That (at least as I read it) is a function, not a command.
>>> If it is a command, give an example of its use from the command line
>>> for us poor "don't want to research" people.  If the following works:
>>>
>>>    $ python setup.py uninstall some_package
>>>
>>> Then explicitly say so for us poor schlubs.
>>>
>>
>> Right, I'll add that. Although it will be a reference implementation 
>> only.
>>
>
> Uninstall as a command feels a little weird. Since "python setup.py 
> [some-command]" implies that the setup.py contains information about 
> the distribution that the command is being applied to. So instead of:
>
> $ python setup.py uninstall some_package

It could be:

    $ python -m distutils uninstall some_package

Asymmetrical with the install of course.

Michael

>
> It could just be:
>
> $ python setup.py uninstall
>
> Except then you'd need to have a complete distribution kicking around 
> with which to run the "python setup.py uninstall" command just to tell 
> the uninstall command the distribution name you want to uninstall. But 
> then with the other uninstall format you could uninstall any 
> distribution from within any other distribution, which is convenient, 
> but weird ... e.g.:
>
> $ cd Spam-1.0/
> $ python setup.py uninstall Foo
>
> Although even the other version of the command could do weird stuff:
>
> $ cd Spam-1.0/
> $ python setup.py install
> $ cd ../Spam-2.0/
> $ python setup.py uninstall
>
> Which would presumably remove the Spam 1.0 distribution when run from 
> the 2.0 version of it! Or perhaps this command should only allow 
> uninstall to be run from a distribution whose name and version match 
> the one that it was installed from ...
>
> I dunno what the right solution is. My two-cents is either to punt and 
> only include an uninstall function as currently proposed, or for only 
> supporting some form of the "python setup.py uninstall" style since I 
> would guess that the most common use-case for uninstall is: user 
> downloads a distribution, runs "python setup.py install", tries out 
> the package, decides they don't like package, then runs "python 
> setup.py uninstall" to restore their python back to it's original 
> state. For doing anything more complex than that, people should be 
> encouraged to use another one of the existing tools for managing their 
> distributions, IMHO.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk 
>


-- 
http://www.ironpythoninaction.com/


From ziade.tarek at gmail.com  Wed Jul  1 16:32:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 16:32:23 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4B5632.5020209@gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<4A4B5632.5020209@gmail.com>
Message-ID: <94bdd2610907010732p76fc39d7ia516c930fb5da987@mail.gmail.com>

On Wed, Jul 1, 2009 at 2:27 PM, Nick Coghlan<ncoghlan at gmail.com> wrote:
> However, having uninstall as a command supported by setup.py also makes
> a certain amount of sense.
>

Yes, that could be an alias that just calls the uninstall global
function using the distribution
name.

From ncoghlan at gmail.com  Wed Jul  1 14:27:30 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 01 Jul 2009 22:27:30 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<h2dqqe$fp1$1@ger.gmane.org>	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
Message-ID: <4A4B5632.5020209@gmail.com>

Kevin Teague wrote:
> Uninstall as a command feels a little weird. Since "python setup.py
> [some-command]" implies that the setup.py contains information about the
> distribution that the command is being applied to. So instead of:
> 
> $ python setup.py uninstall some_package
> 
> It could just be:
> 
> $ python setup.py uninstall

I was actually thinking of something more along the lines of:

  python -m distutils uninstall <whatever>

(Older Pythons won't reliably let you execute a package like that, but
3.1+ and 2.7+ let you do it by including a __main__ module in the package)

However, having uninstall as a command supported by setup.py also makes
a certain amount of sense.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Wed Jul  1 10:41:14 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 10:41:14 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com>
Message-ID: <94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com>

On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/1 Tarek Ziad? <ziade.tarek at gmail.com>:
>>> That (at least as I read it) is a function, not a command.
>>> If it is a command, give an example of its use from the command line
>>> for us poor "don't want to research" people. ?If the following works:
>>>
>>> ? ?$ python setup.py uninstall some_package
>>>
>>> Then explicitly say so for us poor schlubs.
>>>
>>
>> Right, I'll add that. Although it will be a reference implementation only.
>
> -1. Where does the setup.py file come from? If I have docutils
> installed, and want to remove it, must I download the source again so
> that I can get the setup.py, so I can run the uninstall? This is daft
> - particularly given that the point of PEP 376 is to ensure that all
> of the required information is available from the installed package!
>
> As I suggested before:
>
> ? ?python -m distutils.uninstall packagename

yes sorry if it was unclear, I was not thinking about adding something
based on setup.py,
but just saying that I was going to add this feature in the PEP. and
it will be of the form:

 python -m distutils.uninstall packagename

>
> Calling it a "reference implementation" should not imply that it's not
> built to be usable and complete. If it's there,m people should be able
> to use it.

It will be usable and complete, but very limited. As someone mentioned,
it will not take care of dependencies and prevent you from removing a
distribution that is mentioned in another distrubution in a
setuptools' install_requires
metadata.

That said, when PEP 345 evolves like we have planned to (adding
install_requires in the metadata)
Then we should be able to provide this kind of warning with no pain.

From ziade.tarek at gmail.com  Wed Jul  1 15:34:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 15:34:01 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com>
	<79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com>
Message-ID: <94bdd2610907010634v4581fadfr97b20067658262b7@mail.gmail.com>

Right, the API part is almost empty,

http://docs.python.org/distutils/apiref.html

I'll complete it for the relevant part.

On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/1 Paul Moore <p.f.moore at gmail.com>:
>> One other thought. You haven't documented the DistributionMetaData class. The
>
> Just noticed that it's defined in distutils. But it's not documented there :-(
>
> Maybe just add a bit to the PEP saying that the class exists in
> distutils, give a brief summary, and say that as part of implimenting
> the PEP it will be formally documented. (Even though distutils is a
> mess of undocumented functionality, I don't think that you can base
> new work on undocumented internals - you should document them first,
> or things will never get any better :-()
>
> Paul.
>



-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Wed Jul  1 16:29:56 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 16:29:56 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
Message-ID: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> "The problem is that many people use easy_install or pip to install
> their packages..."
>
> As already noted by others, it's not clear why this is a problem. But
> worse is the fact that the paragraph reads to me as saying that
> easy_install/pip/etc don't follow the *current* standard structure.
> Given this fact (which *is* a problem!) I fail to see why I should
> expect them to follow any *new* standard - so how does this PEP
> actually address the issue? Rather, it seems to say to me that the
> existing tools have a history of ignoring the standard approach, so
> this PEP is going to be useless :-(
>
> [I expect that isn't what you're trying to say, so you may just need
> to clarify your meaning. But I do think it's important to address the
> question of how this PEP is going to ensure that existing tools adapt.
> In particular, setuptools seems to have completely stagnated, so I see
> very little likelihood that setuptools is going to change to conform
> to the PEP - how does that affect things?]

Right it's unclear, I'll work on this part.

To resume :

- Phase 1 : introduction of the egg-info file in distutils

Philipp introduced the creation of a file named xxx.egg-info file in 2006
(see http://bugs.python.org/issue1459476)  alongside
distutils-installed package, that contains
the metadata of the distribution.

- Phase 2:  two new formats in the setuptools project

Then he created two new formats in the setuptools project:

1. an .egg-info directory containing a PKG-INFO file, which is similar
to the previous .egg-info file. This happened
   because other files were added in that directory.

2. an .egg directory, possibly zipped, that is a self-contained
version of the distribution, containing the metadata and all other
files.

Setuptools uses 2. by default. There's an option when you use
setuptools to force 1. (?single-version-externally-managed *or*
--root).

- Phase 3: adopting a unified standard.

It occurs that having a .egg-info directory (1.) is way better than a
single file because it's a place-holder for other files.
It is also adopted by the community when it comes to install
setuptools-based package.:

- This is what "pip" uses to install packages in a more flat manner.
- It's also the policy under debian
(http://wiki.debian.org/DebianPython/NewPolicy)
- and under Fedora
(http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools)

The .egg directory (2.) is more controversial because it a
self-contained directory that doesn't install packages
so it makes two different ways to work with distributions for packagers.

So I have proposed in the PEP to adopt the standalone .egg-info
directory as the new distutils unified standard.

This means that all the third-party tools out there already conform to
that standard, and that packages installed
in other formats will not benefit from the new APIs. which means that
people that want to work with distributions installed as .egg
directories will have to use setuptools APIs.
Which makes sense.

>
> ".egg-info becomes a directory"
>
> Don't refer to the setuptools documentation (EggFormats)! It is
> obscure and user-hostile, as well as not actually being the same as
> the PEP's proposal. Rather, just document the structure of .egg-info.
> If you want, you can then add a cross-reference note, saying something
> like "The setuptools structure, as proposed in the EggFormats
> documentation for that package [ref], is a subset of this standard. In
> order to conform to this PEP, setuptools will have to be amended to
> only install .egg-info directories in the format defined by this PEP".

I'll work that way.

>
> "However, it will impact the setuptools and pip projects, but given
> the fact that..."
>
> Confusing. Will these tools need to change (I believe so) or not? If
> they will need to change, that hardly counts as "no deep consequences"
> - there's the whole backward compatibility issue for them to handle.

I'll add this in a backward compatibility section, as suggested
earlier by someone.

>
> "Adding a RECORD file"
>
> "The RECORD format"
>
> "Adding an INSTALLER file in the .egg-info directory"

These section will have more details about the way they interact with bdist_*

for the rest of the mail, I'll clarify the rest of the PEP using your feedback;

> "New APIs in pkgutil"
>
> You say "the best place to put these APIS seems to be pkgutil". You
> should be more definite - "these APIs will be added to the pkgutil
> module".

ok

> General rule - don't document (and commit yourself to) any internal
> details that the user can't access from the public API. It just makes
> backward compatibility harder to maintain.

The purpose is to provide this documentation to third-party projects that want
to implement a packaging system on the top of these classes.

Maybe this should be removed from the PEP and but into another
document targeted to developers ?

>
> "Adding an uninstall function"
>
> One point - what happens if the uninstall method can't remove a file
> or directory? Will it leave things in a broken/partially installed
> state? This needs to be documented (although huge robustness and
> recovery measures don't need to be taken in a simple reference
> implementation). This is likely to be a somewhat common issue on
> Windows in particular, where you can't delete open files.

Good question. I didn't go that far yet because I didn't implement
that part in the
prototype. Maybe the files could be moved to a temporary place before they are
all deleted, allowing to cancel the uninstallation in case a file
couldn't be moved.

> "Installer marker"
>
> There are times when I would imagine it would be useful to force
> uninstallation of the basic package, even if that leaves stuff behind
> related to a particular installer. A force=True argument would be the
> easiest, but if you'd rather not add that, it would be useful to
> document the workaround:
>
> ? ?inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read()
> ? ?uninstall('docutils', installer=inst)

Ok,

>
> If nothing else, it shows a simple example of how to use the APIs. A
> few more such examples in the PEP might be nice.
>
> But as I said, overall, I'm much happier with the PEP now that I've
> read it through and taken the time to digest it.

Great thanks for the feedback.

Tarek

From p.f.moore at gmail.com  Wed Jul  1 13:44:32 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 12:44:32 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
Message-ID: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>

2009/6/30 Paul Moore <p.f.moore at gmail.com>:
> Thank you. I'll try to make the time to go through the PEP and comment
> more fully.

OK, I've now read the PEP through in full. My comments follow. I have
deliberately avoided mentioning points that others have already
raised, to keep things shorter.

First of all, after I got over the use of setuptools terminology, the
document is very good. It's clear what it's trying to achieve, and
covers most of the points I'd want it to. Occasionally the
presentation is confusing, see below, but I think everything was
there.

I do still feel that the setuptools focus will put some readers off.
The introductory remarks, in particular, assume a reasonable level of
familiarity from readers about setuptools, easy_install, pip and the
like. Not having that familiarity isn't a problem in fact, but the
assumption is there in the tone. (For example, personally, I hate the
unnecessarily cute term "egg", but it's established by now, so that's
a lost cause :-()

Onto the PEP.

"The problem is that many people use easy_install or pip to install
their packages..."

As already noted by others, it's not clear why this is a problem. But
worse is the fact that the paragraph reads to me as saying that
easy_install/pip/etc don't follow the *current* standard structure.
Given this fact (which *is* a problem!) I fail to see why I should
expect them to follow any *new* standard - so how does this PEP
actually address the issue? Rather, it seems to say to me that the
existing tools have a history of ignoring the standard approach, so
this PEP is going to be useless :-(

[I expect that isn't what you're trying to say, so you may just need
to clarify your meaning. But I do think it's important to address the
question of how this PEP is going to ensure that existing tools adapt.
In particular, setuptools seems to have completely stagnated, so I see
very little likelihood that setuptools is going to change to conform
to the PEP - how does that affect things?]

".egg-info becomes a directory"

Don't refer to the setuptools documentation (EggFormats)! It is
obscure and user-hostile, as well as not actually being the same as
the PEP's proposal. Rather, just document the structure of .egg-info.
If you want, you can then add a cross-reference note, saying something
like "The setuptools structure, as proposed in the EggFormats
documentation for that package [ref], is a subset of this standard. In
order to conform to this PEP, setuptools will have to be amended to
only install .egg-info directories in the format defined by this PEP".

"However, it will impact the setuptools and pip projects, but given
the fact that..."

Confusing. Will these tools need to change (I believe so) or not? If
they will need to change, that hardly counts as "no deep consequences"
- there's the whole backward compatibility issue for them to handle.

"Adding a RECORD file"

You say "at installation time" - please clarify. Do you only mean
setup.py install? What about bdist_wininst and bdist_msi? What about
third party bdist style tools? How will they ensure they get a RECORD
file?

"The RECORD format"

The line separator shouldn't be os-dependent. What value is used for a
pure-python (ie, platform independent) package? Unless the file is
generated when the install is done, in which case see the previous
point...

Absolute file paths - this implies that RECORD has to be generated by
the installer (which is the only place that knows absolute paths)
which means that every bdist_xxx installer has to write its own RECORD
file. Does the PEP offer no support for this? In any case, the
bdist_wininst and bdist_msi code (which is core distutils) will need
to be amended to manage RECORD files appropriately.

"Adding an INSTALLER file in the .egg-info directory"

Same question again - does the PEP offer supporting APIs to help
bdist_xxx installers do this? What about the core bdist_xxx commands?

"New APIs in pkgutil"

You say "the best place to put these APIS seems to be pkgutil". You
should be more definite - "these APIs will be added to the pkgutil
module".

The Distribution/ZippedDistribution and
DistributionDir/ZippedDistributionDir pairs seem to imply that users
need to explicitly instantiate these classes (and hence know whether a
distribution is zipped). This is cleared up later, but you should add
a note here - "Users will not need to create instances of these
classes manually, they are returned by the public functions in
pkgutil, such as get_distributions()".

"DistributionDirMap class"

This seems to be an internal implementation detail, and as such should
not be documented. It's never returned from any of the public APIs,
and the only created instance is a hidden internal global instance.
The whole part about purging the cache is also unnecessary - the user
has no interface to allow them to do this, so you don't need to
document that it's possible.

Actually, (Zipped)DistributionDir instances are never returned to the
user via the public API, either - so these don't need to be
documented.

General rule - don't document (and commit yourself to) any internal
details that the user can't access from the public API. It just makes
backward compatibility harder to maintain.

"Adding an uninstall function"

With regard to my earlier comments, my apologies - I misread the text
and assumed the documented function simply returned the files to
remove, but didn't do the removal itself. (Which makes it far more
useless than it actually is).

One point - what happens if the uninstall method can't remove a file
or directory? Will it leave things in a broken/partially installed
state? This needs to be documented (although huge robustness and
recovery measures don't need to be taken in a simple reference
implementation). This is likely to be a somewhat common issue on
Windows in particular, where you can't delete open files.

"Installer marker"

There are times when I would imagine it would be useful to force
uninstallation of the basic package, even if that leaves stuff behind
related to a particular installer. A force=True argument would be the
easiest, but if you'd rather not add that, it would be useful to
document the workaround:

    inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read()
    uninstall('docutils', installer=inst)

If nothing else, it shows a simple example of how to use the APIs. A
few more such examples in the PEP might be nice.

But as I said, overall, I'm much happier with the PEP now that I've
read it through and taken the time to digest it.

Paul.

From p.f.moore at gmail.com  Wed Jul  1 16:56:04 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 15:56:04 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
Message-ID: <79990c6b0907010756x1794cc3at16fa05a8b993ebfa@mail.gmail.com>

2009/7/1 Tarek Ziad? <ziade.tarek at gmail.com>:
> Right it's unclear, I'll work on this part.
>
> To resume :
>
> - Phase 1 : introduction of the egg-info file in distutils
>
> Philipp introduced the creation of a file named xxx.egg-info file in 2006
> (see http://bugs.python.org/issue1459476) ?alongside
> distutils-installed package, that contains
> the metadata of the distribution.
>
> - Phase 2: ?two new formats in the setuptools project
>
> Then he created two new formats in the setuptools project:
>
> 1. an .egg-info directory containing a PKG-INFO file, which is similar
> to the previous .egg-info file. This happened
> ? because other files were added in that directory.
>
> 2. an .egg directory, possibly zipped, that is a self-contained
> version of the distribution, containing the metadata and all other
> files.
>
> Setuptools uses 2. by default. There's an option when you use
> setuptools to force 1. (?single-version-externally-managed *or*
> --root).
>
> - Phase 3: adopting a unified standard.
>
> It occurs that having a .egg-info directory (1.) is way better than a
> single file because it's a place-holder for other files.
> It is also adopted by the community when it comes to install
> setuptools-based package.:
>
> - This is what "pip" uses to install packages in a more flat manner.
> - It's also the policy under debian
> (http://wiki.debian.org/DebianPython/NewPolicy)
> - and under Fedora
> (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools)
>
> The .egg directory (2.) is more controversial because it a
> self-contained directory that doesn't install packages
> so it makes two different ways to work with distributions for packagers.
>
> So I have proposed in the PEP to adopt the standalone .egg-info
> directory as the new distutils unified standard.
>
> This means that all the third-party tools out there already conform to
> that standard, and that packages installed
> in other formats will not benefit from the new APIs. which means that
> people that want to work with distributions installed as .egg
> directories will have to use setuptools APIs.
> Which makes sense.

Ah, that makes a *lot* more sense. So, in a "compatibility" section,
you could point out that the proposed standard is compatible with
setuptools "single version externally managed" format, and that the
setuptools multi-version format (the egg file/directory) is a
non-standard format designed to allow multiple versions to be
installed at once - something out of scope for this PEP.

With that explanation, people like me who glaze over at the
complexities of setuptools scenarios should be able to see what's
going on.

Thanks for clarifying.

>> General rule - don't document (and commit yourself to) any internal
>> details that the user can't access from the public API. It just makes
>> backward compatibility harder to maintain.
>
> The purpose is to provide this documentation to third-party projects that want
> to implement a packaging system on the top of these classes.

Hmm, but you don't explain how they should do that. I certainly
wouldn't know how to. As the public API is pkgutil.get_distribution
(and the like) which isn't a class so cannot be subclassed, it's not
clear how a 3rd party could hook in a subclass of
(Zipped)Distribution. And as there's nothing in the public API that
consumes these classes, creating them by hand is of no use, either.

> Maybe this should be removed from the PEP and but into another
> document targeted to developers ?

No. If it's needed by developers, it should be defined in the PEP, not
elsewhere. But it should be defined in a way that developers can use -
not just for the sake of defining it. Unless there is a concrete use
case (which should be stated as an example in the PEP) then *not*
documenting the classes is of more help to developers, as they don't
have to follow a definition that isn't used.

Paul.

From p.f.moore at gmail.com  Wed Jul  1 14:21:41 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 13:21:41 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
Message-ID: <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com>

2009/7/1 Paul Moore <p.f.moore at gmail.com>:
> 2009/6/30 Paul Moore <p.f.moore at gmail.com>:
>> Thank you. I'll try to make the time to go through the PEP and comment
>> more fully.
>
> OK, I've now read the PEP through in full. My comments follow.

One other thought. You haven't documented the DistributionMetaData class. The

And a minor nit:

"""
Distribution class

A new class called Distribution is created with a the path of the
.egg-info directory provided to the contructor. It reads the metadata
contained in PKG-INFO when it is instanciated.
"""

instantiated

Paul.

From rdmurray at bitdance.com  Wed Jul  1 15:39:09 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 1 Jul 2009 09:39:09 -0400 (EDT)
Subject: [Python-Dev] PEP 376
In-Reply-To: <h2ejem$900$1@ger.gmane.org>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<h2ejem$900$1@ger.gmane.org>
Message-ID: <Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>

On Tue, 30 Jun 2009 at 20:06, Scott David Daniels wrote:
> Kevin Teague wrote:
>>  On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote:
>> >  On Tue, Jun 30, 2009 at 10:06 PM, Scott David
>> >  Daniels<Scott.Daniels at acm.org> wrote:
>> > >  Tarek Ziad? wrote:
>> > > >  On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore<p.f.moore at gmail.com> 
>> > > >  wrote:
>> > > > >  [1] I'd actually like it if the PEP defined an uninstall command -
>> > > > >  something like "python -m distutils.uninstall packagename". It can 
>> > > > >  be
>> > > > >  as minimalist as you like, but I'd like to see it present.
>> > > >  it's already there:
>> > > >  http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function
>> > >  That (at least as I read it) is a function, not a command.
>> > >  If it is a command, give an example of its use from the command line
>> > >  for us poor "don't want to research" people.  If the following works:
>> > >     $ python setup.py uninstall some_package
>> > >  Then explicitly say so for us poor schlubs.
>> >  Right, I'll add that. Although it will be a reference implementation 
>> >  only.
>>  Uninstall as a command feels a little weird....
>>  I dunno what the right solution is. My two-cents is either to punt and
>>  only include an uninstall function as currently proposed, or for only
>>  supporting some form of the "python setup.py uninstall" style....
>
> But for us poor schlubs, we want you brilliant packagers to actually
> come to a hard decision.  If you want approval either admit you have no
> solution in the PEP (and detail the issues that prevent a decision),
> provide a minimally acceptable command, or expect that nobody sees the
> value of what you propose....

I haven't read the PEP in detail since it's outside my area of interest
and knowledge(*), but my understanding of the goal is that the PEP is
providing an _infrastructure_ for system-level package management tools.
The uninstall function is part of that infrastructure, but since
distutils isn't a package manager itself (it's an install program),
distutils as currently organized can't really handle uninstall except
as outlined in a section you may have clipped from the above context
(ie: when setup.py from the original package is available).

A possible implementation at the python/distutils level might be to
have a 'pyuninstall' command installed (like pydoc et al are installed)
which handles uninstallation.

The question is what do the people who do real package management
(linux distribution level package management and the equivalent) think?
I'm guessing they are happy with just having the function for their
package management tools to call when needed, since (I'm hoping) they
are part of the distutils sig....

So, if my understanding of the overall goal is correct, it looks to me
like the PEP is missing a "motivation" section that talks about system
package managers.

--David

(*) I'll make time to read it anyway soon.

From ziade.tarek at gmail.com  Wed Jul  1 10:32:15 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 1 Jul 2009 10:32:15 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4B1C51.3060601@voidspace.org.uk>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<4A4B1C51.3060601@voidspace.org.uk>
Message-ID: <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com>

On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord<fuzzyman at voidspace.org.uk> wrote:
>>
>> Uninstall as a command feels a little weird. Since "python setup.py
>> [some-command]" implies that the setup.py contains information about the
>> distribution that the command is being applied to. So instead of:
>>
>> $ python setup.py uninstall some_package
>
> It could be:
>
> ? $ python -m distutils uninstall some_package
>
> Asymmetrical with the install of course.
>

Yes exactly, I was going to add:

$ python -m distutils.uninstall some_package

The whole point of the RECORD file is to be able to uninstall without
depending on the original
archive used to install

From jcea at jcea.es  Wed Jul  1 19:26:18 2009
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 01 Jul 2009 19:26:18 +0200
Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the
	"select" module]
Message-ID: <4A4B9C3A.4090606@jcea.es>


-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-------------- next part --------------
An embedded message was scrubbed...
From: =?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?= <report at bugs.python.org>
Subject: [issue6397] Implementing Solaris "poll" in the "select" module
Date: Wed, 01 Jul 2009 16:49:15 +0000
Size: 2187
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090701/35c875e9/attachment.eml>

From brett at python.org  Thu Jul  2 01:21:56 2009
From: brett at python.org (Brett Cannon)
Date: Wed, 1 Jul 2009 16:21:56 -0700
Subject: [Python-Dev] I am back
Message-ID: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>

Assuming Mailman flipped the right bits to start delivering mail to me
again, my Python sabbatical is now over.
Anything happen while I was gone that I should be aware of that is not
covered in a PEP?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090701/b62950a0/attachment.htm>

From p.f.moore at gmail.com  Wed Jul  1 14:37:36 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 13:37:36 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com>
Message-ID: <79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com>

2009/7/1 Paul Moore <p.f.moore at gmail.com>:
> One other thought. You haven't documented the DistributionMetaData class. The

Just noticed that it's defined in distutils. But it's not documented there :-(

Maybe just add a bit to the PEP saying that the class exists in
distutils, give a brief summary, and say that as part of implimenting
the PEP it will be formally documented. (Even though distutils is a
mess of undocumented functionality, I don't think that you can base
new work on undocumented internals - you should document them first,
or things will never get any better :-()

Paul.

From ncoghlan at gmail.com  Wed Jul  1 14:37:07 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 01 Jul 2009 22:37:07 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	
	<20090630174159.731CE3A4099@sparrow.telecommunity.com>	
	<94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com>	
	<4A4A7F88.7080609@gmail.com>
	<94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com>
Message-ID: <4A4B5873.1050500@gmail.com>

Tarek Ziad? wrote:
>> - What is a "local absolute path"? Absolute path I understand, relative
>> path I understand, but "local absolute" is a novel term to me.
> 
> local means that the "/" separator that is used in the RECORD file for example,
> no matter what platform you are on, is translated using the local separator
> (/ or \)
> 
> I'll make it clearer,

OK, I understand now. That makes Paul's questions about how to correctly
generate the RECORD file for bdist_xxx installations all the more
relevant though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From eric.pruitt at gmail.com  Thu Jul  2 02:47:15 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Wed, 1 Jul 2009 19:47:15 -0500
Subject: [Python-Dev] Bytes, Strings, Encoding
In-Reply-To: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com>
References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com>
Message-ID: <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com>

Hello,

I am working on the subprocess.Popen module for Python 2.7 and am now moving
my changes over to Python 3.1 however I am having trouble with the whole
byte situation and I can't quite seem to understand how to go back and forth
between bytes and strings. I am also looking for the Python 3k equivalent
for the Python 2.X built-in buffer class.

One version of the file with my modifications can be found here  <
http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9>.
Lines 1 - 15 are me attempting to get around certain changes between
Python 3.0 and Python 2.7. Further down on line 916, we have the function
"send" and "recv" in which I am having the most trouble with bytes and
strings.

Any help is appreciated. Feel free to comment on my blog
http://subdev.blogspot.com/ or reply to the last.

Thanks in advance,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090701/0185442a/attachment-0001.htm>

From benjamin at python.org  Thu Jul  2 03:48:52 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 1 Jul 2009 20:48:52 -0500
Subject: [Python-Dev] Bytes, Strings, Encoding
In-Reply-To: <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com>
References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com>
	<171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com>
Message-ID: <1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com>

2009/7/1 Eric Pruitt <eric.pruitt at gmail.com>:
> Hello,
>
> I am working on the subprocess.Popen module for Python 2.7 and am now moving
> my changes over to Python 3.1 however I am having trouble with the whole
> byte situation and I can't quite seem to understand how to go back and forth
> between bytes and strings. I am also looking for the Python 3k equivalent
> for the Python 2.X built-in buffer class.

Look at the "memoryview" type as a replacement for buffer.
io.Buffered* is not the same.

>
> One version of the file with my modifications can be found here? <
> http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9
>>. Lines 1 - 15 are me attempting to get around certain changes between
> Python 3.0 and Python 2.7. Further down on line 916, we have the function
> "send" and "recv" in which I am having the most trouble with bytes and
> strings.

You translate between bytes and strings through encoding and decoding.
Have you read https://secure.wikimedia.org/wikipedia/en/wiki/Unicode?

>
> Any help is appreciated. Feel free to comment on my blog
> http://subdev.blogspot.com/ or reply to the last.

Have you talked with your mentor about these things?


-- 
Regards,
Benjamin

From guido at python.org  Thu Jul  2 04:05:21 2009
From: guido at python.org (Guido van Rossum)
Date: Wed, 1 Jul 2009 19:05:21 -0700
Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in
	the "select" module]
In-Reply-To: <4A4B9C3A.4090606@jcea.es>
References: <4A4B9C3A.4090606@jcea.es>
Message-ID: <ca471dc20907011905s10596599s17923da6168ebd6@mail.gmail.com>

The select module already supports the poll() system call. Or is there
a special variant that only Solaris has?

2009/7/1 Jesus Cea <jcea at jcea.es>:
>
> --
> Jesus Cea Avion ? ? ? ? ? ? ? ? ? ? ? ? _/_/ ? ? ?_/_/_/ ? ? ? ?_/_/_/
> jcea at jcea.es - http://www.jcea.es/ ? ? _/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/
> jabber / xmpp:jcea at jabber.org ? ? ? ? _/_/ ? ?_/_/ ? ? ? ? ?_/_/_/_/_/
> . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?_/_/ ?_/_/ ? ?_/_/ ? ? ? ? ?_/_/ ?_/_/
> "Things are not so easy" ? ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/
> "My name is Dump, Core Dump" ? _/_/_/ ? ? ? ?_/_/_/ ? ? ?_/_/ ?_/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
>
>
> ---------- Forwarded message ----------
> From:?"Jes?s Cea Avi?n" <report at bugs.python.org>
> To:?jcea at jcea.es
> Date:?Wed, 01 Jul 2009 16:49:15 +0000
> Subject:?[issue6397] Implementing Solaris "poll" in the "select" module
>
> New submission from Jes?s Cea Avi?n <jcea at jcea.es>:
>
> In Python 2.6 we added support for Linux "epoll" and *BSD "kqueue" in
> the select module. I think we should add support for Solaris "poll"
> interface too.
>
> What do you think?.
>
> I volunteer to do the work, if you agree this is a feature we want to
> have. I think so.
>
> ----------
> assignee: jcea
> components: Library (Lib)
> messages: 89989
> nosy: jcea
> severity: normal
> stage: needs patch
> status: open
> title: Implementing Solaris "poll" in the "select" module
> type: feature request
> versions: Python 2.7, Python 3.2
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue6397>
> _______________________________________
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Wed Jul  1 07:30:44 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Jul 2009 07:30:44 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<h2dqqe$fp1$1@ger.gmane.org>	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
Message-ID: <4A4AF484.7070600@v.loewis.de>

> I dunno what the right solution is.

I feel that it is straight forward. Either provide a
/usr/bin/python-uninstall utility, or arrange to make
python -mdistutils.uninstall work, so one would do

python -mdistutils.uninstall some_package

Regards,
Martin

From aahz at pythoncraft.com  Thu Jul  2 04:32:51 2009
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 1 Jul 2009 19:32:51 -0700
Subject: [Python-Dev] I am back
In-Reply-To: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>
Message-ID: <20090702023251.GA28507@panix.com>

On Wed, Jul 01, 2009, Brett Cannon wrote:
>
> Anything happen while I was gone that I should be aware of that is not
> covered in a PEP?

Yes.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha

From pje at telecommunity.com  Thu Jul  2 04:41:52 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 01 Jul 2009 22:41:52 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com
 >
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
Message-ID: <20090702023855.F00293A4109@sparrow.telecommunity.com>

At 04:29 PM 7/1/2009 +0200, Tarek Ziad? wrote:
>- Phase 1 : introduction of the egg-info file in distutils
>
>Philipp introduced the creation of a file named xxx.egg-info file in 2006
>(see http://bugs.python.org/issue1459476)  alongside
>distutils-installed package, that contains
>the metadata of the distribution.
>
>- Phase 2:  two new formats in the setuptools project
>
>Then he created two new formats in the setuptools project:

This is backwards.

The .egg-info file was added to distutils and setuptools *after* 
support for the other formats.  In other words, it was a third format 
added to allow other software (such as setuptools and friends) to 
detect the presence of packages installed via the distutils, and 
thereby prevent other tools from installing duplicates.

(.egg-info directories and .egg files/directories already existed.)


>This means that all the third-party tools out there already conform to
>that standard, and that packages installed
>in other formats will not benefit from the new APIs. which means that
>people that want to work with distributions installed as .egg
>directories will have to use setuptools APIs.
>Which makes sense.

Yes and no.  Not providing uninstall support is reasonable, but the 
PEP also has features to query packages in general.

(There's also no technical reason why comparable manifest and 
uninstall support can't be provided for .egg files and directoriees, 
since they already have an implicit manifest: their 
contents.  However, since I'm not currently possessed of the time to 
provide a patch myself, I'm not going to lobby for this as a feature.)


> > If you want, you can then add a cross-reference note, saying something
> > like "The setuptools structure, as proposed in the EggFormats
> > documentation for that package [ref], is a subset of this standard. In
> > order to conform to this PEP, setuptools will have to be amended to
> > only install .egg-info directories in the format defined by this PEP".
>
>I'll work that way.

(Technically, it's an extension of a subset of EggFormats, since it 
adds RECORD and INSTALLER.)


> >
> > "However, it will impact the setuptools and pip projects, but given
> > the fact that..."
> >
> > Confusing. Will these tools need to change (I believe so) or not? If
> > they will need to change, that hardly counts as "no deep consequences"
> > - there's the whole backward compatibility issue for them to handle.
>
>I'll add this in a backward compatibility section, as suggested
>earlier by someone.

They don't need to change, unless they're going to support being 
uninstalled by the tool (in which case they'll need to add RECORD and 
INSTALLER).


From p.f.moore at gmail.com  Wed Jul  1 10:47:07 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 09:47:07 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com>
	<94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com>
Message-ID: <79990c6b0907010147g4d3b3848i36eafbcd0ba56ef2@mail.gmail.com>

2009/7/1 Tarek Ziad? <ziade.tarek at gmail.com>:
>> As I suggested before:
>>
>> ? ?python -m distutils.uninstall packagename
>
> yes sorry if it was unclear, I was not thinking about adding something
> based on setup.py,
> but just saying that I was going to add this feature in the PEP. and
> it will be of the form:
>
> ?python -m distutils.uninstall packagename
>
>>
>> Calling it a "reference implementation" should not imply that it's not
>> built to be usable and complete. If it's there,m people should be able
>> to use it.
>
> It will be usable and complete, but very limited. As someone mentioned,
> it will not take care of dependencies and prevent you from removing a
> distribution that is mentioned in another distrubution in a
> setuptools' install_requires
> metadata.

Thank you. That is precisely the right level for the PEP, IMHO.

> That said, when PEP 345 evolves like we have planned to (adding
> install_requires in the metadata)
> Then we should be able to provide this kind of warning with no pain.

That's good, but not something that bothers me too much. (I'm still of
the view that dependencies should be handled by documenting them
properly - metadata isn't sufficient on its own).

Paul.

From alexandre at peadrop.com  Thu Jul  2 04:55:49 2009
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Wed, 1 Jul 2009 22:55:49 -0400
Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in
	the "select" module]
In-Reply-To: <ca471dc20907011905s10596599s17923da6168ebd6@mail.gmail.com>
References: <4A4B9C3A.4090606@jcea.es>
	<ca471dc20907011905s10596599s17923da6168ebd6@mail.gmail.com>
Message-ID: <acd65fa20907011955g2b336b14w5298de2b465d1b9e@mail.gmail.com>

On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum<guido at python.org> wrote:
> The select module already supports the poll() system call. Or is there
> a special variant that only Solaris has?
>

I think Jesus refers to /dev/poll?i.e., the interface for
edge-triggered polling on Solaris. This is the Solaris equivalent of
FreeBSD's kqueue and Linux's epoll.

-- Alexandre

From stephen at xemacs.org  Thu Jul  2 05:44:57 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 02 Jul 2009 12:44:57 +0900
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
Message-ID: <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp>

Tarek Ziad? writes:
 > On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore at gmail.com> wrote:

 > > General rule - don't document (and commit yourself to) any
 > > internal details that the user can't access from the public
 > > API. It just makes backward compatibility harder to maintain.
 > 
 > The purpose is to provide this documentation to third-party
 > projects that want to implement a packaging system on the top of
 > these classes.

That's a judgment you must make.  However, Paul's opinion seems to be
that it is internal, and not needed by third-parties who are working
"on the top" of these classes.  If upon consideration you agree, you
should take those "details" out of the PEP proper.  If you disagree,
you should promote them to the "official"/public API.

The point of a PEP is not to construct a duck; it is to explain what
"quack" means.

 > Maybe this should be removed from the PEP and but into another
 > document targeted to developers ?

Yes.  In the reference implementation (aka prototype), which should
have its own documentation as usual.  The reference implementation for
the PEP shows *how* these things can be done.  It must do that, or the
PEP has no force.  However, the reference implementation need not be
"industrial strength"; the actual implementation that eventually goes
into the stdlib may very well be an optimized and enhanced version
with many data structures and algorithms that differ from the
reference implementation.

Another general principle: even in the draft PEP, say "is", not "will
be".  If you're wrong, and that won't work; if it's insufficiently
precise; if you find a more elegant way to express the solution; if
you simply haven't thought carefully about something yet; if, for
whatever reason, you may need to change the PEP, then you change it in
the next draft.  That's why we have a review cycle, so you can change
it.  If you yourself have a question, or the draft is incomplete at
some point, then you can explain in a note (either a footnote or
parentheses).  You can even mark a whole section as "speculative" (eg,
"nothing in this section has been implemented yet, so everything is
subject to change").  But a rule should be stated as a rule, and
precisely, so that reviewers can criticize it accurately.

If you're uncomfortable saying "is", then maybe that part of the draft
isn't ready for public review yet.  On the other hand, you can use the
reviewers' knowledge here: write "A is B", then as a note "(In some
cases A is actually C, and this should be treated specially.  But
how?)"


From p.f.moore at gmail.com  Wed Jul  1 10:35:35 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jul 2009 09:35:35 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
Message-ID: <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com>

2009/7/1 Tarek Ziad? <ziade.tarek at gmail.com>:
>> That (at least as I read it) is a function, not a command.
>> If it is a command, give an example of its use from the command line
>> for us poor "don't want to research" people. ?If the following works:
>>
>> ? ?$ python setup.py uninstall some_package
>>
>> Then explicitly say so for us poor schlubs.
>>
>
> Right, I'll add that. Although it will be a reference implementation only.

-1. Where does the setup.py file come from? If I have docutils
installed, and want to remove it, must I download the source again so
that I can get the setup.py, so I can run the uninstall? This is daft
- particularly given that the point of PEP 376 is to ensure that all
of the required information is available from the installed package!

As I suggested before:

    python -m distutils.uninstall packagename

Calling it a "reference implementation" should not imply that it's not
built to be usable and complete. If it's there,m people should be able
to use it.

Paul.

From p.f.moore at gmail.com  Thu Jul  2 09:59:20 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Jul 2009 08:59:20 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<h2ejem$900$1@ger.gmane.org>
	<Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>
Message-ID: <79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com>

2009/7/1 R. David Murray <rdmurray at bitdance.com>:
> I haven't read the PEP in detail since it's outside my area of interest
> and knowledge(*), but my understanding of the goal is that the PEP is
> providing an _infrastructure_ for system-level package management tools.
> The uninstall function is part of that infrastructure, but since
> distutils isn't a package manager itself (it's an install program),
> distutils as currently organized can't really handle uninstall except
> as outlined in a section you may have clipped from the above context
> (ie: when setup.py from the original package is available).

>From a Windows user's POV, "install program" = "package manager". And
an install program needs an uninstall feature. I know Linux users with
their advanced dependency-managing package managers feel that this is
a stone-age view, and they may be right, but the PEP needs to take the
Windows situation into account.

> The question is what do the people who do real package management
> (linux distribution level package management and the equivalent) think?
> I'm guessing they are happy with just having the function for their
> package management tools to call when needed, since (I'm hoping) they
> are part of the distutils sig....

Don't forget that the maintainers of the bdist_wininst, bdist_msi and
bdist_rpm code *are* the distutils maintainers - so to that extent,
the PEP has to say how *those* aspects of package managers are
covered. Unless another PEP is accepted saying that support for
bdist_xxx is being dropped [1], this part of distutils cannot be
ignored.

>
> So, if my understanding of the overall goal is correct, it looks to me
> like the PEP is missing a "motivation" section that talks about system
> package managers.

Possibly. If so, though, it must discuss the above 3 cases which are
part of core distutils.

Paul.

[1] A PEP I plan on strongly opposing!

From ziade.tarek at gmail.com  Thu Jul  2 10:55:51 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 10:55:51 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <20090702023855.F00293A4109@sparrow.telecommunity.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
	<20090702023855.F00293A4109@sparrow.telecommunity.com>
Message-ID: <94bdd2610907020155w43aee580wac3604f828cb6b54@mail.gmail.com>

On Thu, Jul 2, 2009 at 4:41 AM, P.J. Eby<pje at telecommunity.com> wrote:
> Yes and no. ?Not providing uninstall support is reasonable, but the PEP also
> has features to query packages in general.
>
> (There's also no technical reason why comparable manifest and uninstall
> support can't be provided for .egg files and directoriees, since they
> already have an implicit manifest: their contents. ?However, since I'm not
> currently possessed of the time to provide a patch myself, I'm not going to
> lobby for this as a feature.)
>

Yes that's not technical : I am in favor of keeping a single format. I
don't see but confusion in keeping several formats,

From ziade.tarek at gmail.com  Thu Jul  2 11:01:34 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 11:01:34 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>
	<877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com>

On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull<stephen at xemacs.org> wrote:
> That's a judgment you must make. ?However, Paul's opinion seems to be
> that it is internal, and not needed by third-parties who are working
> "on the top" of these classes. ?If upon consideration you agree, you
> should take those "details" out of the PEP proper. ?If you disagree,
> you should promote them to the "official"/public API.
>
> The point of a PEP is not to construct a duck; it is to explain what
> "quack" means.

Yes, while the APIs I have written in the prototype+PEP helped us
claryfing what we wanted,
I agree they would be better in a second document.

They are two target audience, the users of distutils and the builders
of package managers,
so removing this details from the PEP will also make it easier to read
for the first crowd.

>
> Another general principle: even in the draft PEP, say "is", not "will
> be".

Ok I'll fix that. That's a French stuff : in french, "will be" isn't
speculative at all.

Thx

From jcea at jcea.es  Thu Jul  2 11:52:32 2009
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 02 Jul 2009 11:52:32 +0200
Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in
 the "select" module]
In-Reply-To: <acd65fa20907011955g2b336b14w5298de2b465d1b9e@mail.gmail.com>
References: <4A4B9C3A.4090606@jcea.es>	<ca471dc20907011905s10596599s17923da6168ebd6@mail.gmail.com>
	<acd65fa20907011955g2b336b14w5298de2b465d1b9e@mail.gmail.com>
Message-ID: <4A4C8360.7000701@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexandre Vassalotti wrote:
> On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum<guido at python.org> wrote:
>> The select module already supports the poll() system call. Or is there
>> a special variant that only Solaris has?
>>
> 
> I think Jesus refers to /dev/poll?i.e., the interface for
> edge-triggered polling on Solaris. This is the Solaris equivalent of
> FreeBSD's kqueue and Linux's epoll.

Yes. In python 2.6, Linux and *BSD benefice of access to high performing
"poll"-like interfaces, non portable. Solaris has "/dev/poll" too, so we
can provide another non-portable interface :). In fact I think it could
be interesting to investigate the direct use of "/dev/poll" in Solaris
under "select.poll" implementation, if available, in a transparent way.
With this could be possible to improve efficiency without adding another
non-portable interface.

But before investigating options, I want to know if there is interest on
it. I think so, because Solaris/OpenSolaris is an important platform, I
will take care of the code, and we already have non-portable interfaces:
"select.epoll" for Linux and "select.kqueue" for *BSD.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSkyDXZlgi5GaxT1NAQJSawP/SUf2AVfzYbqv8F3zZjNBspFu5c3bYVEK
6A2YSW0WIrNiQtcI9Ka99dCCzHI6cwohlDhxQddJEG4w30hewsFBFJqftSpPLhkv
ToSUNDYe37QmRgVrcns//repWoaMwsrt8MMQU6MLb9pyaoT4Pffe1vcbqg+/XgUN
EJENI7BjSOw=
=BTMr
-----END PGP SIGNATURE-----

From ben+python at benfinney.id.au  Thu Jul  2 12:16:41 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 02 Jul 2009 20:16:41 +1000
Subject: [Python-Dev] PEP 376
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<h2ejem$900$1@ger.gmane.org>
	<Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>
	<79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com>
Message-ID: <873a9ftmnq.fsf@benfinney.id.au>

Paul Moore <p.f.moore at gmail.com> writes:

> 2009/7/1 R. David Murray <rdmurray at bitdance.com>:
> > The uninstall function is part of that infrastructure, but since
> > distutils isn't a package manager itself (it's an install program),
> > distutils as currently organized can't really handle uninstall
> > except as outlined in a section you may have clipped from the above
> > context (ie: when setup.py from the original package is available).
> 
> >From a Windows user's POV, "install program" = "package manager". And
> an install program needs an uninstall feature. I know Linux users with
> their advanced dependency-managing package managers feel that this is
> a stone-age view, and they may be right, but the PEP needs to take the
> Windows situation into account.

Actually, your view seems quite reasonable to me: as a GNU user with my
advanced dependency-managing package manager, I feel just as strongly
that a program that installs is implicitly promising that it can also
uninstall cleanly.

I think that calling distutils an ?install program? is confusing. When
I discuss distutils, I don't call it a program at all; it's a library
(or perhaps ?framework?) that provides part of the job of package
installation.

-- 
 \      ?I think it would be a good idea.? ?Mahatma Gandhi (when asked |
  `\                          what he thought of Western civilization) |
_o__)                                                                  |
Ben Finney


From p.f.moore at gmail.com  Thu Jul  2 13:26:00 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Jul 2009 12:26:00 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <873a9ftmnq.fsf@benfinney.id.au>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<h2ejem$900$1@ger.gmane.org>
	<Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>
	<79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com>
	<873a9ftmnq.fsf@benfinney.id.au>
Message-ID: <79990c6b0907020426x5165e0bfl886e876dbae47d57@mail.gmail.com>

2009/7/2 Ben Finney <ben+python at benfinney.id.au>:
> I think that calling distutils an ?install program? is confusing. When
> I discuss distutils, I don't call it a program at all; it's a library
> (or perhaps ?framework?) that provides part of the job of package
> installation.

I agree, but I consider the fact that distutils "provides" python
setup.py install means that it "is an install program" in some
people's minds. These are the same people who complain about the lack
of a distutils uninstall feature.

Personally, I want distutils to provide all the individual bits that I
get from bdist_wininst, so that I can put together an alternative for
specialised situations (notably, non-site installs of some packages).
So uninstall and being able to list installed packages are important
to me - hence my interest in this PEP.

Paul.

From ncoghlan at gmail.com  Thu Jul  2 13:32:57 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Jul 2009 21:32:57 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<h2dqqe$fp1$1@ger.gmane.org>	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>	<4A4B1C51.3060601@voidspace.org.uk>
	<94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com>
Message-ID: <4A4C9AE9.5000806@gmail.com>

Tarek Ziad? wrote:
> On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord<fuzzyman at voidspace.org.uk> wrote:
>>> Uninstall as a command feels a little weird. Since "python setup.py
>>> [some-command]" implies that the setup.py contains information about the
>>> distribution that the command is being applied to. So instead of:
>>>
>>> $ python setup.py uninstall some_package
>> It could be:
>>
>>   $ python -m distutils uninstall some_package
>>
>> Asymmetrical with the install of course.
>>
> 
> Yes exactly, I was going to add:
> 
> $ python -m distutils.uninstall some_package

A directly executable submodule is an even better idea than making
distutils itself executable. Definitely worth mentioning in the PEP in
the section on uninstallation support.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Thu Jul  2 13:40:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 13:40:10 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4C9AE9.5000806@gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<4A4B1C51.3060601@voidspace.org.uk>
	<94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com>
	<4A4C9AE9.5000806@gmail.com>
Message-ID: <94bdd2610907020440t747e94fema65b940be596db29@mail.gmail.com>

On Thu, Jul 2, 2009 at 1:32 PM, Nick Coghlan<ncoghlan at gmail.com> wrote:
>>
>> Yes exactly, I was going to add:
>>
>> $ python -m distutils.uninstall some_package
>
> A directly executable submodule is an even better idea than making
> distutils itself executable. Definitely worth mentioning in the PEP in
> the section on uninstallation support.
>

The PEP is updated for this, and takes into account several feedback
from this thread,
(like removing implementation details)

I am working now on describing the behavior with the bdist_* commands

From ncoghlan at gmail.com  Thu Jul  2 13:39:48 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Jul 2009 21:39:48 +1000
Subject: [Python-Dev] I am back
In-Reply-To: <20090702023251.GA28507@panix.com>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>
	<20090702023251.GA28507@panix.com>
Message-ID: <4A4C9C84.1030702@gmail.com>

Aahz wrote:
> On Wed, Jul 01, 2009, Brett Cannon wrote:
>> Anything happen while I was gone that I should be aware of that is not
>> covered in a PEP?
> 
> Yes.

I agree with Aahz ;)

Cheers,
Nick.

P.S. Welcome back :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ben+python at benfinney.id.au  Thu Jul  2 14:24:14 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 02 Jul 2009 22:24:14 +1000
Subject: [Python-Dev] PEP 376
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<h2dqqe$fp1$1@ger.gmane.org>
	<94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com>
	<8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca>
	<h2ejem$900$1@ger.gmane.org>
	<Pine.LNX.4.64.0907010920320.6306@kimball.webabinitio.net>
	<79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com>
	<873a9ftmnq.fsf@benfinney.id.au>
	<79990c6b0907020426x5165e0bfl886e876dbae47d57@mail.gmail.com>
Message-ID: <87y6r7s26p.fsf@benfinney.id.au>

Paul Moore <p.f.moore at gmail.com> writes:

> 2009/7/2 Ben Finney <ben+python at benfinney.id.au>:
> > I think that calling distutils an ?install program? is confusing.
> > When I discuss distutils, I don't call it a program at all; it's a
> > library (or perhaps ?framework?) that provides part of the job of
> > package installation.
> 
> I agree, but I consider the fact that distutils "provides" python
> setup.py install means that it "is an install program" in some
> people's minds. These are the same people who complain about the lack
> of a distutils uninstall feature.

Well, I don't suffer from that confusion between the library and a
specific program, yet I still want distutils to support uninstall :-)
and hence am interested in seeing the fate of this PEP.

-- 
 \     ?Men never do evil so completely and cheerfully as when they do |
  `\        it from religious conviction.? ?Blaise Pascal (1623?1662), |
_o__)                                                   Pens?es, #894. |
Ben Finney


From ziade.tarek at gmail.com  Thu Jul  2 14:34:59 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 14:34:59 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
Message-ID: <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>

On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> "Adding a RECORD file"
>
> You say "at installation time" - please clarify. Do you only mean
> setup.py install? What about bdist_wininst and bdist_msi? What about
> third party bdist style tools? How will they ensure they get a RECORD
> file?
>
> "The RECORD format"
>
> The line separator shouldn't be os-dependent. What value is used for a
> pure-python (ie, platform independent) package? Unless the file is
> generated when the install is done, in which case see the previous
> point...
>
> Absolute file paths - this implies that RECORD has to be generated by
> the installer (which is the only place that knows absolute paths)
> which means that every bdist_xxx installer has to write its own RECORD
> file. Does the PEP offer no support for this? In any case, the
> bdist_wininst and bdist_msi code (which is core distutils) will need
> to be amended to manage RECORD files appropriately.

Since all bdist_* commands are using the install command in a
temporary build directory to create
a binary distribution, the RECORD file (and the INSTALLER) will be
pre-generated then installed
alongside the packages/modules.

For absolute paths now that gets installed, what would be the
difference between the pre-generated
RECORD file and the RECORD file installed on the win32 target system, if any ?

From p.f.moore at gmail.com  Thu Jul  2 14:39:29 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Jul 2009 13:39:29 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
Message-ID: <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>

2009/7/2 Tarek Ziad? <ziade.tarek at gmail.com>:
> For absolute paths now that gets installed, what would be the
> difference between the pre-generated
> RECORD file and the RECORD file installed on the win32 target system, if any ?

When is an absolute path generated? If you can give me a small sample
of a distribution that installs a file in an absolute path, I'll do
some testing on Windows.

But the immediate answer is that there are *no* reliable "absolute"
paths on Windows, so you're not looking at the likes of /usr/doc, but
rather paths that are relative to sys.prefix, but not to the package
directory. In that case, the key point is that if the installer is
built on a system where Python is installed in a different directory
than the system where the installer is run, paths need to be
relocated. (E.g., C:\Python27 vs D:\Apps\Python27).

Paul.

From anderson.lizardo at openbossa.org  Thu Jul  2 15:03:48 2009
From: anderson.lizardo at openbossa.org (Anderson Lizardo)
Date: Thu, 2 Jul 2009 09:03:48 -0400
Subject: [Python-Dev] "re" module and Python 1.6 (GPL incompatible) license?
Message-ID: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com>

Hi,

I noticed that some files of the "re" module still have the (GPL
incompatible) 1.6 license notice. Is that on purpose or
unintentionally forgotten?

If that is on purpose, does that mean that some GPL Python code cannot
use the "re" module?

For reference, here are the links to these files (from trunk):

http://svn.python.org/view/python/trunk/Lib/re.py?view=markup
http://svn.python.org/view/python/trunk/Modules/_sre.c?view=markup

Additionally, the files below do not have a complete license header,
or they just point to one of the files above:

http://svn.python.org/view/python/trunk/Lib/sre_compile.py?view=markup
http://svn.python.org/view/python/trunk/Lib/sre_constants.py?view=markup
http://svn.python.org/view/python/trunk/Lib/sre_parse.py?view=markup
http://svn.python.org/view/python/trunk/Modules/sre.h?view=markup
http://svn.python.org/view/python/trunk/Modules/sre_constants.h?view=markup

PS: if this gets fixed somehow, can the fix be backported to the
Python 2.5.x releases? Let me know what needs to be done to get this
fixed on next release.

Thanks,
-- 
Anderson Lizardo
OpenBossa Labs - INdT
Manaus - Brazil

From ziade.tarek at gmail.com  Thu Jul  2 15:45:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 15:45:23 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
	<79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
Message-ID: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>

On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/2 Tarek Ziad? <ziade.tarek at gmail.com>:
>> For absolute paths now that gets installed, what would be the
>> difference between the pre-generated
>> RECORD file and the RECORD file installed on the win32 target system, if any ?
>
> When is an absolute path generated? If you can give me a small sample
> of a distribution that installs a file in an absolute path, I'll do
> some testing on Windows.

Try this setup.py file:

"""
from distutils.core import setup

setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
"""

with this MANIFEST.in file:
""
include data
"""

and add a 'data' file alongside

If you install it, data is copied in /tmp. If you create a bdist
distribution it will be created in the root of the dumb directory
which is used to generate the binary distro.

Of course you'll have to change '/tmp' to 'c:\tmp'

(which makes me realizes that there's no way to force the installation
data_files in another drive under windows:
  the binary distribution will be the same, no matter what drive you
use in the absolute path used in data_files
I'll add an issue on this)

>
> But the immediate answer is that there are *no* reliable "absolute"
> paths on Windows, so you're not looking at the likes of /usr/doc, but
> rather paths that are relative to sys.prefix, but not to the package
> directory. In that case, the key point is that if the installer is
> built on a system where Python is installed in a different directory
> than the system where the installer is run, paths need to be
> relocated. (E.g., C:\Python27 vs D:\Apps\Python27).

I get the point: they are three levels we should handle in the RECORD file

1. absolute paths
2. paths relative to sys.prefix or sys.exec_prefix
3. paths relative to the directory where the .egg-info directory is located

the RECORD file doesn't handle 2. indeed. For instance you can add a script:

setup(
..
scripts=['myscript.py']
..)

that will get installed in : sys.prefix + 'Scripts/myscript.py'  under
win32 for instance

So a possible solution is to add 2. in the RECORD files by using a
notation such as
"$PREFIX" "$EXEC_PREFIX" when we detect that the file is under on of
these paths.

The query functions will then be able to use sys.prefix and
sys.exec_prefix for recompose
the absolute pat on the target system


>
> Paul.
>



-- 
Tarek Ziad? | http://ziade.org

From solipsis at pitrou.net  Thu Jul  2 16:18:42 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 2 Jul 2009 14:18:42 +0000 (UTC)
Subject: [Python-Dev] PEP 376
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<ca471dc20906302109q63594f38td9fd12ad119d970f@mail.gmail.com>
Message-ID: <loom.20090702T141741-4@post.gmane.org>

Guido van Rossum <guido <at> python.org> writes:
> 
> I noted an inconsistency: first you say that the RECORD file uses the
> excel dialect, but at the end of the same section you say it uses the
> default csv settings. Sounds like you need to delete one or the other.

If I remember correctly, the default csv dialect is just the Excel dialect, so
this would be two different ways of saying the same things.

Regards

Antoine.



From p.f.moore at gmail.com  Thu Jul  2 16:35:18 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Jul 2009 15:35:18 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
	<79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
	<94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
Message-ID: <79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com>

2009/7/2 Tarek Ziad? <ziade.tarek at gmail.com>:
> Try this setup.py file:
>
> """
> from distutils.core import setup
>
> setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
> """
>
> with this MANIFEST.in file:
> ""
> include data
> """
>
> and add a 'data' file alongside

Ta. I'll give it a go tonight. But haven't you made the point yourself
by saying I'll need to change the directory? This will fail for me as
I don't have a "/tmp" directory. So I'd expect a bdist installer
(*any* bdist installer) to fail, as it's got no way to tell where to
put that file. I can only imagine this type of thing being done by
someone packaging for Unix, with no interest in cross-platform
portability (I suspect MAC OS directory names differ from Unix as
well).

So maybe absolute filenames should be banned in distutils? (OK, less
radically, maybe someone should ask where that functionality is used
in practice, and based on the answers come up with a more portable
approach).

Paul.

From ziade.tarek at gmail.com  Thu Jul  2 16:50:51 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 2 Jul 2009 16:50:51 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
	<79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
	<94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
	<79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com>
Message-ID: <94bdd2610907020750lc49ddd3kf21f7dceda13bc2b@mail.gmail.com>

On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> Ta. I'll give it a go tonight. But haven't you made the point yourself
> by saying I'll need to change the directory? This will fail for me as
> I don't have a "/tmp" directory. So I'd expect a bdist installer
> (*any* bdist installer) to fail, as it's got no way to tell where to
> put that file. I can only imagine this type of thing being done by
> someone packaging for Unix, with no interest in cross-platform
> portability (I suspect MAC OS directory names differ from Unix as
> well).

It will fail if your path starts with "/" and you are under win32
because there's
a converter that will raise an error in that case at installation time.

But you can use some code in your setup.py to provide the right directory
name when install is called. I don't see it as a problem.

I'm just afraid it's impossible to use efficiently under win32 because
of the drive
letter. But that's rather a bug.

>
> So maybe absolute filenames should be banned in distutils? (OK, less
> radically, maybe someone should ask where that functionality is used
> in practice, and based on the answers come up with a more portable
> approach).
>

I'll try to see if I can collect that info out of PyPI with a script,
to list the real-world usages.
Maybe Martin you have a simple way to do this on PyPI ?

Tarek

From skip at pobox.com  Thu Jul  2 18:07:13 2009
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 2 Jul 2009 11:07:13 -0500
Subject: [Python-Dev] I am back
In-Reply-To: <20090702023251.GA28507@panix.com>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>
	<20090702023251.GA28507@panix.com>
Message-ID: <19020.56113.916994.809864@montanaro.dyndns.org>

    Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote:

    >> Anything happen while I was gone that I should be aware of that is
    >> not covered in a PEP?

    Aahz> Yes.

In particular, Brett, you probably didn't hear that the King of Pop died
last week.  It was hardly mentioned in the major news outlets...

Skip

From josiah.carlson at gmail.com  Thu Jul  2 18:30:04 2009
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 2 Jul 2009 09:30:04 -0700
Subject: [Python-Dev] Bytes, Strings, Encoding
In-Reply-To: <1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com>
References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com>
	<171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com>
	<1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com>
Message-ID: <e6511dbf0907020930w3e2b9243o8a2a8610475e6c21@mail.gmail.com>

On Wed, Jul 1, 2009 at 6:48 PM, Benjamin Peterson<benjamin at python.org> wrote:
> 2009/7/1 Eric Pruitt <eric.pruitt at gmail.com>:
>> Hello,
>>
>> I am working on the subprocess.Popen module for Python 2.7 and am now moving
>> my changes over to Python 3.1 however I am having trouble with the whole
>> byte situation and I can't quite seem to understand how to go back and forth
>> between bytes and strings. I am also looking for the Python 3k equivalent
>> for the Python 2.X built-in buffer class.
>
> Look at the "memoryview" type as a replacement for buffer.
> io.Buffered* is not the same.

FYI, it is used slightly differently.  Rather than x = buffer(y,
offset), you use x = memoryview(y)[offset:].

>> One version of the file with my modifications can be found here? <
>> http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9
>>>. Lines 1 - 15 are me attempting to get around certain changes between
>> Python 3.0 and Python 2.7. Further down on line 916, we have the function
>> "send" and "recv" in which I am having the most trouble with bytes and
>> strings.
>
> You translate between bytes and strings through encoding and decoding.
> Have you read https://secure.wikimedia.org/wikipedia/en/wiki/Unicode?
>
>>
>> Any help is appreciated. Feel free to comment on my blog
>> http://subdev.blogspot.com/ or reply to the last.
>
> Have you talked with your mentor about these things?
>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
>

From stefan_ml at behnel.de  Thu Jul  2 19:06:53 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Jul 2009 19:06:53 +0200
Subject: [Python-Dev] I am back
In-Reply-To: <19020.56113.916994.809864@montanaro.dyndns.org>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>	<20090702023251.GA28507@panix.com>
	<19020.56113.916994.809864@montanaro.dyndns.org>
Message-ID: <h2ipfd$p0b$1@ger.gmane.org>

skip at pobox.com wrote:
>     Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote:
> 
>     >> Anything happen while I was gone that I should be aware of that is
>     >> not covered in a PEP?
> 
>     Aahz> Yes.
> 
> In particular, Brett, you probably didn't hear that the King of Pop died
> last week.  It was hardly mentioned in the major news outlets...

*shrug* who cares about foreign politics anyway. I don't even know where
Pop is, and Google Maps doesn't seem to either.

Stefan


From python at mrabarnett.plus.com  Thu Jul  2 19:27:25 2009
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 02 Jul 2009 18:27:25 +0100
Subject: [Python-Dev] I am back
In-Reply-To: <h2ipfd$p0b$1@ger.gmane.org>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>	<20090702023251.GA28507@panix.com>	<19020.56113.916994.809864@montanaro.dyndns.org>
	<h2ipfd$p0b$1@ger.gmane.org>
Message-ID: <4A4CEDFD.8070704@mrabarnett.plus.com>

Stefan Behnel wrote:
> skip at pobox.com wrote:
>>     Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote:
>>
>>     >> Anything happen while I was gone that I should be aware of that is
>>     >> not covered in a PEP?
>>
>>     Aahz> Yes.
>>
>> In particular, Brett, you probably didn't hear that the King of Pop died
>> last week.  It was hardly mentioned in the major news outlets...
> 
> *shrug* who cares about foreign politics anyway. I don't even know where
> Pop is, and Google Maps doesn't seem to either.
> 
I believe that "Pop" is an abbreviation for "Populous". You can read
about it here:

http://en.wikipedia.org/wiki/Populous

:-)

From Scott.Daniels at Acm.Org  Thu Jul  2 20:01:15 2009
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Thu, 02 Jul 2009 11:01:15 -0700
Subject: [Python-Dev] PEP 376
In-Reply-To: <loom.20090702T141741-4@post.gmane.org>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<ca471dc20906302109q63594f38td9fd12ad119d970f@mail.gmail.com>
	<loom.20090702T141741-4@post.gmane.org>
Message-ID: <h2is83$1og$1@ger.gmane.org>

Antoine Pitrou wrote:
> Guido van Rossum <guido <at> python.org> writes:
>> I noted an inconsistency: first you say that the RECORD file uses the
>> excel dialect, but at the end of the same section you say it uses the
>> default csv settings. Sounds like you need to delete one or the other.
> 
> If I remember correctly, the default csv dialect is just the Excel dialect, so
> this would be two different ways of saying the same things.

Right, but Guido's point is, decide which one is the is the definition
and stick to  talking about it in that form.

--Scott David Daniels
Scott.Daniels at Acm.Org


From brett at python.org  Thu Jul  2 19:54:53 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 2 Jul 2009 10:54:53 -0700
Subject: [Python-Dev] I am back
In-Reply-To: <4A4C9C84.1030702@gmail.com>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com> 
	<20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com>
Message-ID: <bbaeab100907021054y39c0ac01t203df8da145daf44@mail.gmail.com>

On Thu, Jul 2, 2009 at 04:39, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Aahz wrote:
> > On Wed, Jul 01, 2009, Brett Cannon wrote:
> >> Anything happen while I was gone that I should be aware of that is not
> >> covered in a PEP?
> >
> > Yes.
>
> I agree with Aahz ;)
>

Ah, the smart ass remarks never cease. =)


>
> Cheers,
> Nick.
>
> P.S. Welcome back :)


Thanks!

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090702/93a86b4a/attachment-0001.htm>

From tjreedy at udel.edu  Thu Jul  2 20:00:09 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 02 Jul 2009 14:00:09 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <h2is83$1og$1@ger.gmane.org>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<ca471dc20906302109q63594f38td9fd12ad119d970f@mail.gmail.com>	<loom.20090702T141741-4@post.gmane.org>
	<h2is83$1og$1@ger.gmane.org>
Message-ID: <h2isj9$oi$1@ger.gmane.org>

Scott David Daniels wrote:
> Antoine Pitrou wrote:
>> Guido van Rossum <guido <at> python.org> writes:
>>> I noted an inconsistency: first you say that the RECORD file uses the
>>> excel dialect, but at the end of the same section you say it uses the
>>> default csv settings. Sounds like you need to delete one or the other.
>>
>> If I remember correctly, the default csv dialect is just the Excel 
>> dialect, so
>> this would be two different ways of saying the same things.
> 
> Right, but Guido's point is, decide which one is the is the definition
> and stick to  talking about it in that form.

Since I think it more likely that Excel will change than cvs default, I 
think the latter should be used.


From tjreedy at udel.edu  Thu Jul  2 20:44:36 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 02 Jul 2009 14:44:36 -0400
Subject: [Python-Dev] I am back
In-Reply-To: <bbaeab100907021054y39c0ac01t203df8da145daf44@mail.gmail.com>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com>
	<20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com>
	<bbaeab100907021054y39c0ac01t203df8da145daf44@mail.gmail.com>
Message-ID: <h2iv6k$an5$1@ger.gmane.org>

Brett Cannon wrote:
> 
> 
> On Thu, Jul 2, 2009 at 04:39, Nick Coghlan <ncoghlan at gmail.com 
> <mailto:ncoghlan at gmail.com>> wrote:
> 
>     Aahz wrote:
>      > On Wed, Jul 01, 2009, Brett Cannon wrote:
>      >> Anything happen while I was gone that I should be aware of that
>     is not
>      >> covered in a PEP?
>      >
>      > Yes.
> 
>     I agree with Aahz ;)
> 
> 
> Ah, the smart ass remarks never cease. =)

The biggest news is the release of 3.1, but that might have been 
recorded in the relevant PEP, and it certainly hard to miss.

In any case, welcome back.

Terry


From brett at python.org  Thu Jul  2 21:04:45 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 2 Jul 2009 12:04:45 -0700
Subject: [Python-Dev] semi-regular check that all committers are subscribed
	to python-committers
Message-ID: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com>

If you have commit privileges on Python but are not subscribed to
python-committers, please let me know and I will subscribe you. Membership
on python-committers is mandatory as that's where we announce branch
freezes, etc.
-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090702/fad0b8ee/attachment.htm>

From martin at v.loewis.de  Thu Jul  2 21:33:50 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Jul 2009 21:33:50 +0200
Subject: [Python-Dev] "re" module and Python 1.6 (GPL incompatible)
	license?
In-Reply-To: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com>
References: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com>
Message-ID: <4A4D0B9E.3070301@v.loewis.de>

> PS: if this gets fixed somehow, can the fix be backported to the
> Python 2.5.x releases?

Definitely not - it's not security critical.

Regards,
Martin

From dirkjan at ochtman.nl  Thu Jul  2 21:56:34 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 2 Jul 2009 21:56:34 +0200
Subject: [Python-Dev] semi-regular check that all committers are
	subscribed to python-committers
In-Reply-To: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com>
References: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com>
Message-ID: <ea2499da0907021256sbd69effo7239cd7c89e8fba6@mail.gmail.com>

On Thu, Jul 2, 2009 at 21:04, Brett Cannon<brett at python.org> wrote:
> If you have commit privileges on Python but are not subscribed to
> python-committers, please let me know and I will subscribe you. Membership
> on python-committers is mandatory as that's where we announce branch
> freezes, etc.

Would it be helpful if I was subscribed? I'm not strictly a committer,
but what with the Mercurial migration I imagine it won't be long
before the first cset bearing my name appears in the Python repository
(plus I just saw there was some discussion about the hg migration
there which I could weigh in on).

Cheers,

Dirkjan

From p.f.moore at gmail.com  Thu Jul  2 22:06:53 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Jul 2009 21:06:53 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
	<79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
	<94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
Message-ID: <79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.com>

2009/7/2 Tarek Ziad? <ziade.tarek at gmail.com>:
> On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>> 2009/7/2 Tarek Ziad? <ziade.tarek at gmail.com>:
>>> For absolute paths now that gets installed, what would be the
>>> difference between the pre-generated
>>> RECORD file and the RECORD file installed on the win32 target system, if any ?
>>
>> When is an absolute path generated? If you can give me a small sample
>> of a distribution that installs a file in an absolute path, I'll do
>> some testing on Windows.
>
> Try this setup.py file:
>
> """
> from distutils.core import setup
>
> setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
> """
>
> with this MANIFEST.in file:
> ""
> include data
> """
>
> and add a 'data' file alongside
>
> If you install it, data is copied in /tmp. If you create a bdist
> distribution it will be created in the root of the dumb directory
> which is used to generate the binary distro.
>
> Of course you'll have to change '/tmp' to 'c:\tmp'

>python setup.py bdist_wininst
running bdist_wininst
running build
installing to build\bdist.win32\wininst
running install_data
creating build
creating build\bdist.win32
creating build\bdist.win32\wininst
creating build\bdist.win32\wininst\DATA
Traceback (most recent call last):
  File "setup.py", line 3, in <module>
    setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
  File "C:\Apps\Python26\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Apps\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Apps\Python26\lib\distutils\command\bdist_wininst.py", line
171, in run
    install.run()
  File "C:\Apps\Python26\lib\distutils\command\install.py", line 589, in run
    self.run_command(cmd_name)
  File "C:\Apps\Python26\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Apps\Python26\lib\distutils\command\install_data.py", line 62, in run
    dir = convert_path(f[0])
  File "C:\Apps\Python26\lib\distutils\util.py", line 164, in convert_path
    raise ValueError, "path '%s' cannot be absolute" % pathname
ValueError: path '/tmp' cannot be absolute

Looks like bdist_wininst doesn't support absolute paths. Which isn't
surprising, really.

Same with bdist_msi.

Actually, with setup.py install:

>python setup.py install
running install
running build
running install_data
Traceback (most recent call last):
  File "setup.py", line 3, in <module>
    setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
  File "C:\Apps\Python26\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Apps\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Apps\Python26\lib\distutils\command\install.py", line 589, in run
    self.run_command(cmd_name)
  File "C:\Apps\Python26\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Apps\Python26\lib\distutils\command\install_data.py", line 62, in run
    dir = convert_path(f[0])
  File "C:\Apps\Python26\lib\distutils\util.py", line 164, in convert_path
    raise ValueError, "path '%s' cannot be absolute" % pathname
ValueError: path '/tmp' cannot be absolute

So how do absolute paths work??? It sure ain't like this!

Changing '/tmp' to c:/tmp' gives this:

>python setup.py bdist_wininst
running bdist_wininst
running build
installing to build\bdist.win32\wininst
running install_data
creating build\bdist.win32\wininst\DATA\c:tmp
error: could not create 'build\bdist.win32\wininst\DATA\c:tmp': The
directory name is invalid

Changing it to 'c:\\tmp' gives (sort of) success:

python setup.py bdist_wininst
running bdist_wininst
running build
installing to build\bdist.win32\wininst
running install_data
creating build\bdist.win32\wininst\tmp
copying data -> build\bdist.win32\wininst\tmp
running install_egg_info
Creating build\bdist.win32\wininst\PURELIB\
Writing build\bdist.win32\wininst\PURELIB\foo-1.0-py2.6.egg-info
creating 'c:\users\gustav\appdata\local\temp\tmpcs9rmi.zip' and adding '.' to it
adding 'PURELIB\foo-1.0-py2.6.egg-info'
adding 'tmp\data'
creating dist
removing 'build\bdist.win32\wininst' (and everything under it)

The installer packs a *relative* path tmp/data - which isn't right.

bdist_msi copies the data file to c:\tmp - when building the msi!.
This is very wrong. I can't introspect a msi file to see what ends up
in it, but if I install it, no c:\tmp\data is installed, so it does
look like it's screwed up.

Frankly, handling of absolute paths looks so broken, that I'd
explicitly state that it's not supported.

I'd raise bugs for some of these things, but to be honest, I'm not at
all sure what I'd expect to be correct behaviour (on Windows). In my
view absolute paths shouldn't be supported - I can see that might be
different on Linux, but I don't see how it's ever going to be
cross-platform.

Not sure where that leaves things. Absolute path handling seems broken
enough (on Windows) that whatever the PEP says isn't much of an issue,
as it's not usable anyway. One definite point, though - given the
behaviour of bdist_msi, it thought it had installed something that it
simply didn't - so there's a serious risk that the distutils uninstall
function could uninstall something that the installer never installed
- that's far worse than simply deleting the files and messing up the
installer metadata.

Maybe the answer is that distutils reject *all* uses of absolute paths
on Windows. The PEP can then say whatever you want as far as I'm
concerned, as it doesn't affect me. If the existing bugs don't get
fixed, though, I'd say that the RECORD file should be explicitly
disallowed from containing absolute paths on Windows (for safety
reasons if nothing else).

Paul.

From brett at python.org  Thu Jul  2 22:08:13 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 2 Jul 2009 13:08:13 -0700
Subject: [Python-Dev] semi-regular check that all committers are
	subscribed to python-committers
In-Reply-To: <ea2499da0907021256sbd69effo7239cd7c89e8fba6@mail.gmail.com>
References: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com> 
	<ea2499da0907021256sbd69effo7239cd7c89e8fba6@mail.gmail.com>
Message-ID: <bbaeab100907021308i64cb8262ma72e673ab33c7464@mail.gmail.com>

On Thu, Jul 2, 2009 at 12:56, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:

> On Thu, Jul 2, 2009 at 21:04, Brett Cannon<brett at python.org> wrote:
> > If you have commit privileges on Python but are not subscribed to
> > python-committers, please let me know and I will subscribe you.
> Membership
> > on python-committers is mandatory as that's where we announce branch
> > freezes, etc.
>
> Would it be helpful if I was subscribed? I'm not strictly a committer,
> but what with the Mercurial migration I imagine it won't be long
> before the first cset bearing my name appears in the Python repository
> (plus I just saw there was some discussion about the hg migration
> there which I could weigh in on).


 Maybe you should just get the commit privileges now. Any objections?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090702/5352b9de/attachment.htm>

From dirkjan at ochtman.nl  Thu Jul  2 22:10:49 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 2 Jul 2009 22:10:49 +0200
Subject: [Python-Dev] semi-regular check that all committers are
	subscribed to python-committers
In-Reply-To: <bbaeab100907021308i64cb8262ma72e673ab33c7464@mail.gmail.com>
References: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com>
	<ea2499da0907021256sbd69effo7239cd7c89e8fba6@mail.gmail.com>
	<bbaeab100907021308i64cb8262ma72e673ab33c7464@mail.gmail.com>
Message-ID: <ea2499da0907021310n232d666eg6f8565eaf7f61afc@mail.gmail.com>

On Thu, Jul 2, 2009 at 22:08, Brett Cannon<brett at python.org> wrote:
> ?Maybe you should just get the commit privileges now. Any objections?

Not from me, obviously.

Cheers,

Dirkjan

From pje at telecommunity.com  Thu Jul  2 22:59:11 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 02 Jul 2009 16:59:11 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.co
 m>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>
	<4A4A05DE.2040804@gmail.com>
	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>
	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>
	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>
	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>
	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>
	<94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com>
	<79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com>
	<94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com>
	<79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.com>
Message-ID: <20090702205612.57D8A3A4109@sparrow.telecommunity.com>

At 09:06 PM 7/2/2009 +0100, Paul Moore wrote:
>Maybe the answer is that distutils reject *all* uses of absolute paths
>on Windows. The PEP can then say whatever you want as far as I'm
>concerned, as it doesn't affect me. If the existing bugs don't get
>fixed, though, I'd say that the RECORD file should be explicitly
>disallowed from containing absolute paths on Windows (for safety
>reasons if nothing else).

In non bdist cases (e.g. setup.py with manual options, but also 
easy_install, pip, etc.) you can specify where data files and scripts 
go on Windows, and they don't have to be relative locations.  So, 
absolute paths need to be supported, despite the existence of broken 
bdist targets on Windows.


From dirkjan at ochtman.nl  Thu Jul  2 22:42:32 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 2 Jul 2009 22:42:32 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
Message-ID: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>

In response to some rumblings on python-committers and just to request
more feedback, a progress report. I know it's long, I've tried to put
to keep it concise and chunked, though.

- First of all, I've got the basic conversion down, I've done it a few
times now, with progressively better results. You can view some
results at http://hg.python.org/, which has a preliminary cpython
repository. *** The changeset hashes for that repo will change, so you
won't be able to commit or pull from it in the future.***

- Second of all, some planning. I've thought about it a bit, and I
think we should aim for going live with hg on August 1. Given that I'm
on vacation from 8-18 July (and I'm not sure whether I'll be able to
actually work on it during that time, though I imagine I'll be able to
spend some time on it at least), that's quite ambitious, so I'm going
to say it's okay if it slips by a few days. Putting a deadline out
there is a good thing, anyway.

- Third of all, to make this possible, it would be helpful if I got
more feedback on the PEP. Last time I raised it, there was virtually
nothing. This time, I'll include it inline so there's hopefully less
of a barrier to reviewing it.

- Fourth, Mercurial 1.3 was just released! Bet you didn't see that
coming. It's looking like a pretty good release, with an experimental
version of the much-coveted subrepository support (like
svn:externals). This also means that the latest version of
hgsubversion, the tool I used for the conversion, will be more
accessible for converting other projects. You know you want to!

- Fifth, here's a list of things, off the top of my head, that still need doing:

* Get agreement on branch strategy and branch processing (list of
branches + proposed handling at
http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
REVIEW
* Get agreement on tag processing (first come up with a list)
* Set up hg-ssh infra (should be easy)
* Set up hooks (should be mostly straightforward)
* Set up roundup integration (should be made easier by quick revision
map hgweb extension)
* Write docs

- Sixth (this is the good part), less obvious things that have been
done or don't need doing:

* .hgignore generation (I've been convinced it's too hard, the current
version will do)
* revlog reordering (it's painless and a big win)

I'll get through all of these myself, but obviously any help would be
welcome. For any hg users, writing docs should be an easy start. For
others, please review the PEP (below), the branch map in
http://hg.python.org/pymigr/file/tip/all-branches.txt and the author
map at http://hg.python.org/pymigr/file/tip/author-map (not that much
has changed since the start, so if you've looked at it already, feel
free to skip this part). Right now I'm a little stuck on branch
processing, because it's a long running script that needs a bunch of
debugging, but I'll get going on that again.

I think that's all I can think of for now, I'll update the PEP with
new bits soon. Here it is, ready for your review:

==============================================================

Motivation

After having decided to switch to the Mercurial DVCS, the actual
migration still has to be performed. In the case of an important piece
of infrastructure like the version control system for a large,
distributed project like Python, this is a significant effort. This
PEP is an attempt to describe the steps that must be taken for further
discussion. It's somewhat similar to PEP 347 [1], which discussed the
migration to SVN.

To make the most of hg, I (Dirkjan) would like to make a high-fidelity
conversion, such that (a) as much of the svn metadata as possible is
retained, and (b) all metadata is converted to formats that are common
in Mercurial. This way, tools written for Mercurial can be optimally
used. In order to do this, I want to use the hgsubversion [2] software
to do an initial conversion. This hg extension is focused on providing
high-quality conversion from Subversion to Mercurial for use in
two-way correspondence, meaning it doesn't throw away as much
available metadata as other solutions.

Such a conversion also seems like a good time to reconsider the
contents of the repository and determine if some things are still
valuable. In this spirit, the following sections also propose
discarding some of the older metadata.
Timeline

TBD; needs fully working hgsubversion and consensus on this document.
Transition plan
Branch strategy

Mercurial has two basic ways of using branches: cloned branches, where
each branch is kept in a separate repository, and named branches,
where each revision keeps metadata to note on which branch it belongs.
The former makes it easier to distinguish branches, at the expense of
requiring more disk space on the client. The latter makes it a little
easier to switch between branches, but often has somewhat unintuitive
results for people (though this has been getting better in recent
versions of Mercurial).

I'm still a bit on the fence about whether Python should adopt cloned
branches and named branches. Since it usually makes more sense to tag
releases on the maintenance branch, for example, mainline history
would not contain release tags if we used cloned branches. Also,
Mercurial 1.2 and 1.3 have the necessary tools to make named branches
less painful (because they can be properly closed and closed heads are
no longer considered in relevant cases).

A disadvantage might be that the used clones will be a good bit larger
(since they essentially contain all other branches as well). This can
me mitigated by keeping non-release (feature) branches in separate
clones. Also note that it's still possible to clone a single named
branch from a combined clone, by specifying the branch as in hg clone
http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a
separate clone problably also makes sense.

XXX To do: size comparison for selected separation scenarios.
Converting branches

There are quite a lot of branches in SVN's branches directory. I
propose to clean this up a bit, by employing the following the
strategy:

    * Keep all release (maintenance) branches
    * Discard branches that haven't been touched in 18 months, unless
somone indicates there's still interest in such a branch
    * Keep branches that have been touched in the last 18 months,
unless someone indicates the branch can be deprecated

Converting tags

The SVN tags directory contains a lot of old stuff. Some of these are
not, in fact, full tags, but contain only a smaller subset of the
repository. I think we should keep all release tags, and consider
other tags for inclusion based on requests from the developer
community. I'd like to consider unifying the release tag naming scheme
to make some things more consistent, if people feel that won't create
too many problems. For example, Mercurial itself just uses '1.2.1' as
a tag, where CPython would currently use r121.
Author map

In order to provide user names the way they are common in hg (in the
'First Last <user at example.org>' format), we need an author map to map
cvs and svn user names to real names and their email addresses. I have
a complete version of such a map in my migration tools repository [3].
The email addresses in it might be out of date; that's bound to
happen, although it would be nice to try and have as many people as
possible review it for addresses that are out of date. The current
version also still seems to contain some encoding problems.
Generating .hgignore

The .hgignore file can be used in Mercurial repositories to help
ignore files that are not eligible for version control. It does this
by employing several possible forms of pattern matching. The current
Python repository already includes a rudimentary .hgignore file to
help with using the hg mirrors.

It might be useful to have the .hgignore be generated automatically
from svn:ignore properties. This would make sure all historic
revisions also have useful ignore information (though one could argue
ignoring isn't really relevant to just checking out an old revision).
Revlog reordering

As an optional optimization technique, we should consider trying a
reordering pass on the revlogs (internal Mercurial files) resulting
from the conversion. In some cases this results in dramatic decreases
in on-disk repository size.
Other repositories

Richard Tew has indicated that he'd like the Stackless repository to
also be converted. What other projects in the svn.python.org
repository should be converted? Do we want to convert the peps
repository? distutils? others?
Infrastructure
hg-ssh

Developers should access the repositories through ssh, similar to the
current setup. Public keys can be used to grant people access to a
shared hg@ account. A hgwebdir instance should also be set up for easy
browsing and read-only access. If we're using ssh, developers should
trivially be able to start new clones (for longer-term features that
profit from a separate branch).
Hooks

A number of hooks is currently in use. The hg equivalents for these
should be developed and deployed. The following hooks are being used:

    * check whitespace: a hook to reject commits in case the
whitespace doesn't match the rules for the Python codebase. Should be
straightforward to re-implement from the current version. We can also
offer a whitespace hook for use with client-side repositories that
people can use; it could either warn about whitespace issues and/or
truncate trailing whitespace from changed lines. Open issue: do we
check only the tip after each push, or do we check every commit in a
changegroup?
    * commit mails: we can leverage the notify extension for this
    * buildbots: both the regular and the community build masters must
be notified. Fortunately buildbot includes support for hg. I've also
implemented this for Mercurial itself, so I don't expect problems
here.
    * check contributors: in the current setup, all changesets bear
the username of committers, who must have signed the contributor
agreement. In a DVCS, the committers are not necessarily the same
people who push, and so we can't check if the committer is a
contributor. We could use a hook to check if the committer is a
contributor if we keep a list of registered contributors.

hgwebdir

A more or less stock hgwebdir installation should be set up. We might
want to come up with a style to match the Python website. It may also
be useful to build a quick extension to augment the URL rev parser so
that it can also take r[0-9]+ args and come up with the matching hg
revision.
After migration
Where to get code

It needs to be decided where the hg repositories will live. I'd like
to propose to keep the hgwebdir instance at hg.python.org. This is an
accepted standard for many organizations, and an easy parallel to
svn.python.org. The 2.7 (trunk) repo might live at
http://hg.python.org/main/, for example, with py3k at
http://hg.python.org/py3k/. For write access, developers will have to
use ssh, which could be ssh://hg at hg.python.org/main/. A demo
installation will be set up with a preliminary conversion so people
can experiment and review; it can live at
http://hg.python.org/example/.

code.python.org was also proposed as the hostname. Personally, I think
that using the VCS name in the hostname is good because it prevents
confusion: it should be clear that you can't use svn or bzr for
hg.python.org.

hgwebdir can already provide tarballs for every changeset. I think
this obviates the need for daily snapshots; we can just point users to
tip.tar.gz instead, meaning they will get the latest. If desired, we
could even use buildbot results to point to the last good changeset.
Python-specific documentation

hg comes with good built-in documentation (available through hg help)
and a wiki [4] that's full of useful information and recipes. In
addition to that, the parts of the developer FAQ [5] concerning
version control will gain a section on using hg for Python
development. Some of the text will be dependent on the outcome of
debate about this PEP (for example, the branching strategy).

Think first, commit later?

In recent history, old versions of Python have been maintained by a
select group of people backporting patches from trunk to release
branches. While this may not scale so well as the development pace
grows, it also runs into some problems with the current crop of
distributed versioning tools. These tools (I believe similar problems
would exist for either git, bzr, or hg, though some may cope better
than others) are based on the idea of a Directed Acyclic Graph (or
DAG), meaning they keep track of relations of changesets.

Mercurial itself has a stable branch which is a ''strict'' subset of
the unstable branch. This means that generally all fixes for the
stable branch get committed against the tip of the stable branch, then
they get merged into the unstable branch (which already contains the
parent of the new cset). This provides a largely frictionless
environment for moving changes from stable to unstable branches.
Mistakes, where a change that should go on stable goes on unstable
first, do happen, but they're usually easy to fix. That can be done by
copying the change over to the stable branch, then trivial-merging
with unstable -- meaning the merge in fact ignores the parent from the
stable branch).

This strategy means a little more work for regular committers, because
they have to think about whether their change should go on stable or
unstable; they may even have to ask someone else (the RM) before
committing. But it also relieves a dedicated group of committers of
regular backporting duty, in addition to making it easier to work with
the tool.

Now would be a good time to consider changing strategies in this
regard, although it would be relatively easy to switch to such a model
later on.
The future of Subversion

What happens to the Subversion repositories after the migration? Since
the svn server contains a bunch of repositories, not just the CPython
one, it will probably live on for a bit as not every project may want
to migrate or it takes longer for other projects to migrate. To
prevent people from staying behind, we may want to remove migrated
projects from the repository.
Build identification

Python currently provides the sys.subversion tuple to allow Python
code to find out exactly what version of Python it's running against.
The current version looks something like this:

    * ('CPython', 'tags/r262', '71600')
    * ('CPython', 'trunk', '73128M')

Another value is returned from Py_GetBuildInfo() in the C API, and
available to Python code as part of sys.version:

    * 'r262:71600, Jun 2 2009, 09:58:33'
    * 'trunk:73128M, Jun 2 2009, 01:24:14'

I propose that the revision identifier will be the short version of
hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
(instead of 'M') if the working directory from which it was built was
modified. This mirrors the output of the hg id command, which is
intended for this kind of usage.

For the tag/branch identifier, I propose that hg will check for tags
on the currently checked out revision, use the tag if there is one
('tip' doesn't count), and uses the branch name otherwise.
sys.subversion becomes

    * ('CPython', '2.6.2', 'dd3ebf81af43')
    * ('CPython', 'default', 'af694c6a888c+')

and the build info string becomes

    * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33'
    * 'default:af694c6a888c+, Jun 2 2009, 01:24:14'

This reflects that the default branch in hg is called 'default'
instead of Subversion's 'trunk', and reflects the proposed new tag
format.
References
[1]	http://www.python.org/dev/peps/pep-0347/
[2]	http://bitbucket.org/durin42/hgsubversion/
[3]	http://hg.xavamedia.nl/cpython/pymigr/
[4]	http://www.selenic.com/mercurial/wiki/
[5]	http://www.python.org/dev/faq/#version-control

=====================================================

Cheers,

Dirkjan

From greg at krypto.org  Fri Jul  3 00:41:48 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 2 Jul 2009 15:41:48 -0700
Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in
	the "select" module]
In-Reply-To: <4A4C8360.7000701@jcea.es>
References: <4A4B9C3A.4090606@jcea.es>
	<ca471dc20907011905s10596599s17923da6168ebd6@mail.gmail.com> 
	<acd65fa20907011955g2b336b14w5298de2b465d1b9e@mail.gmail.com> 
	<4A4C8360.7000701@jcea.es>
Message-ID: <52dc1c820907021541l3ac2301ej6efa032f50e19429@mail.gmail.com>

On Thu, Jul 2, 2009 at 2:52 AM, Jesus Cea<jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alexandre Vassalotti wrote:
>> On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum<guido at python.org> wrote:
>>> The select module already supports the poll() system call. Or is there
>>> a special variant that only Solaris has?
>>>
>>
>> I think Jesus refers to /dev/poll?i.e., the interface for
>> edge-triggered polling on Solaris. This is the Solaris equivalent of
>> FreeBSD's kqueue and Linux's epoll.
>
> Yes. In python 2.6, Linux and *BSD benefice of access to high performing
> "poll"-like interfaces, non portable. Solaris has "/dev/poll" too, so we
> can provide another non-portable interface :). In fact I think it could
> be interesting to investigate the direct use of "/dev/poll" in Solaris
> under "select.poll" implementation, if available, in a transparent way.
> With this could be possible to improve efficiency without adding another
> non-portable interface.
>
> But before investigating options, I want to know if there is interest on
> it. I think so, because Solaris/OpenSolaris is an important platform, I
> will take care of the code, and we already have non-portable interfaces:
> "select.epoll" for Linux and "select.kqueue" for *BSD.

+1  I'm sure *Solaris users would appreciate this.  Go for it.

>
> - --
> Jesus Cea Avion ? ? ? ? ? ? ? ? ? ? ? ? _/_/ ? ? ?_/_/_/ ? ? ? ?_/_/_/
> jcea at jcea.es - http://www.jcea.es/ ? ? _/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/
> jabber / xmpp:jcea at jabber.org ? ? ? ? _/_/ ? ?_/_/ ? ? ? ? ?_/_/_/_/_/
> . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?_/_/ ?_/_/ ? ?_/_/ ? ? ? ? ?_/_/ ?_/_/
> "Things are not so easy" ? ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/
> "My name is Dump, Core Dump" ? _/_/_/ ? ? ? ?_/_/_/ ? ? ?_/_/ ?_/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iQCVAwUBSkyDXZlgi5GaxT1NAQJSawP/SUf2AVfzYbqv8F3zZjNBspFu5c3bYVEK
> 6A2YSW0WIrNiQtcI9Ka99dCCzHI6cwohlDhxQddJEG4w30hewsFBFJqftSpPLhkv
> ToSUNDYe37QmRgVrcns//repWoaMwsrt8MMQU6MLb9pyaoT4Pffe1vcbqg+/XgUN
> EJENI7BjSOw=
> =BTMr
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>

From greg at krypto.org  Fri Jul  3 00:47:48 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 2 Jul 2009 15:47:48 -0700
Subject: [Python-Dev] pthread sem PyThread_acquire_lock
In-Reply-To: <4A4931E7.1020108@v.loewis.de>
References: <4A466846.1050904@aim.com> <4A468507.1060503@v.loewis.de> 
	<4A46A40B.5080202@aim.com>
	<536685ea0906291233r1b3e8dc4ndf25e7f404a87e22@mail.gmail.com> 
	<4A4931E7.1020108@v.loewis.de>
Message-ID: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com>

On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.
>
> This is my understanding as well - so I don't think Python actually
> "swallows" the signal.
>
>> A great example is reading from a socket. Whether or not it can be
>> interrupted depends on the platform, so catching Ctrl+C often requires
>> a timeout loop.
>>
>> Also, remember that signals are asynchronous in the sense that they
>> are handled outside the normal execution flow of a program. Checking
>> for EINTR probably isn't the best way to determine if a signal has
>> been sent to the program.
>
> I think it would be reasonable to support "asynchronous" exceptions,
> and Python supports SIGINT fairly well most of the time.
>
> It might be possible to support keyboard interrupts throughout
> the system, but changing Python to do so could also cause
> incompatibilities. So any change must be done with greatest care,
> but simultaneously, should also try to arrange to cover all cases.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org

If you want signals to actually be handled in a timely manner, its
best to leave the main thread of the program alone as a signal
handling thread that just spends its time in a loop of time.sleep(123)
calls rather than blocking on any sort of lock.  Spawn other threads
to do the actual work in your program.  Signals are delivered
indirectly in the existing CPython implementation by setting an
internal flag that the main interpreter thread polls on occasion so
blocking calls that do not interrupt and return early being called
from the main thread will effectively block signals.

From exarkun at divmod.com  Fri Jul  3 01:04:51 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 2 Jul 2009 19:04:51 -0400
Subject: [Python-Dev] pthread sem PyThread_acquire_lock
In-Reply-To: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com>
Message-ID: <20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com>

On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" <greg at krypto.org> wrote:
>On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>>> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers.
>>
>> This is my understanding as well - so I don't think Python actually
>> "swallows" the signal.
>>
>>> A great example is reading from a socket. Whether or not it can be
>>> interrupted depends on the platform, so catching Ctrl+C often requires
>>> a timeout loop.
>>>
>>> Also, remember that signals are asynchronous in the sense that they
>>> are handled outside the normal execution flow of a program. Checking
>>> for EINTR probably isn't the best way to determine if a signal has
>>> been sent to the program.
>>
>> I think it would be reasonable to support "asynchronous" exceptions,
>> and Python supports SIGINT fairly well most of the time.
>>
>> It might be possible to support keyboard interrupts throughout
>> the system, but changing Python to do so could also cause
>> incompatibilities. So any change must be done with greatest care,
>> but simultaneously, should also try to arrange to cover all cases.
>>
>> Regards,
>> Martin
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>If you want signals to actually be handled in a timely manner, its
>best to leave the main thread of the program alone as a signal
>handling thread that just spends its time in a loop of time.sleep(123)
>calls rather than blocking on any sort of lock.  Spawn other threads
>to do the actual work in your program.  Signals are delivered
>indirectly in the existing CPython implementation by setting an
>internal flag that the main interpreter thread polls on occasion so
>blocking calls that do not interrupt and return early being called
>from the main thread will effectively block signals.

Yes, this is all true now.  The question is why the implementation
works that way, and whether it is desirable to keep it working that
way.

Considering *some* of the lock implementations make themselves not
interruptable by threads while others don't bother, it seems like
*some* change to the status quo is desirable.

Jean-Paul

From benjamin at python.org  Fri Jul  3 01:26:58 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 2 Jul 2009 18:26:58 -0500
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <1afaf6160907021626w297c1727kcbbf1a0c586f07ac@mail.gmail.com>

2009/7/2 Dirkjan Ochtman <dirkjan at ochtman.nl>:
> In response to some rumblings on python-committers and just to request
> more feedback, a progress report. I know it's long, I've tried to put
> to keep it concise and chunked, though.

Thanks very much for working on this, Dirkjan. It may seem rather
thankless now, but I'm sure once we wish to Mercurial, the praise will
flow. :)

>
> - Second of all, some planning. I've thought about it a bit, and I
> think we should aim for going live with hg on August 1. Given that I'm
> on vacation from 8-18 July (and I'm not sure whether I'll be able to
> actually work on it during that time, though I imagine I'll be able to
> spend some time on it at least), that's quite ambitious, so I'm going
> to say it's okay if it slips by a few days. Putting a deadline out
> there is a good thing, anyway.

Sounds good.


> - Fifth, here's a list of things, off the top of my head, that still need doing:
>
> * Get agreement on branch strategy and branch processing (list of
> branches + proposed handling at
> http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
> REVIEW

The io-c branch doesn't need to stay.

> * Get agreement on tag processing (first come up with a list)
> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs
>
> - Sixth (this is the good part), less obvious things that have been
> done or don't need doing:

I suppose this includes modifying sys.subversion as described in the PEP?

> The SVN tags directory contains a lot of old stuff. Some of these are
> not, in fact, full tags, but contain only a smaller subset of the
> repository. I think we should keep all release tags, and consider
> other tags for inclusion based on requests from the developer
> community. I'd like to consider unifying the release tag naming scheme
> to make some things more consistent, if people feel that won't create
> too many problems. For example, Mercurial itself just uses '1.2.1' as
> a tag, where CPython would currently use r121.

+1 to unifying tag name style to the current cpython procedure.

> Author map
>
> In order to provide user names the way they are common in hg (in the
> 'First Last <user at example.org>' format), we need an author map to map
> cvs and svn user names to real names and their email addresses. I have
> a complete version of such a map in my migration tools repository [3].
> The email addresses in it might be out of date; that's bound to
> happen, although it would be nice to try and have as many people as
> possible review it for addresses that are out of date. The current
> version also still seems to contain some encoding problems.
> Generating .hgignore

What effect will the encoding problems have? Does hg require ASCII characters?

> Richard Tew has indicated that he'd like the Stackless repository to
> also be converted. What other projects in the svn.python.org
> repository should be converted? Do we want to convert the peps
> repository? distutils? others?

I think everything should be converted unless there's a reason not to.
(such as the maintainer indicates she doesn't what to migrate)


> A number of hooks is currently in use. The hg equivalents for these
> should be developed and deployed. The following hooks are being used:
>
> ? ?* check whitespace: a hook to reject commits in case the
> whitespace doesn't match the rules for the Python codebase. Should be
> straightforward to re-implement from the current version. We can also
> offer a whitespace hook for use with client-side repositories that
> people can use; it could either warn about whitespace issues and/or
> truncate trailing whitespace from changed lines. Open issue: do we
> check only the tip after each push, or do we check every commit in a
> changegroup?

It might as well be on every commit because it will have to normalized
on push anyway.

> code.python.org was also proposed as the hostname. Personally, I think
> that using the VCS name in the hostname is good because it prevents
> confusion: it should be clear that you can't use svn or bzr for
> hg.python.org.

+1 for hg.python.org

>
> Think first, commit later?
>
> In recent history, old versions of Python have been maintained by a
> select group of people backporting patches from trunk to release
> branches. While this may not scale so well as the development pace
> grows, it also runs into some problems with the current crop of
> distributed versioning tools. These tools (I believe similar problems
> would exist for either git, bzr, or hg, though some may cope better
> than others) are based on the idea of a Directed Acyclic Graph (or
> DAG), meaning they keep track of relations of changesets.

The problem is that Python is much more complicated than the average
project. We have many commits that are only applicable one maintenance
branch, or just 2.x, or just 3.x; the trunk and py3k will never be
subsets of each other. Regardless of where we make commits initially,
we need to ability to manage special cases like this easily.



-- 
Regards,
Benjamin

From skippy.hammond at gmail.com  Fri Jul  3 01:01:24 2009
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Fri, 03 Jul 2009 09:01:24 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <4A4D3C44.60504@gmail.com>

On 3/07/2009 6:42 AM, Dirkjan Ochtman wrote:
> In response to some rumblings on python-committers and just to request
> more feedback, a progress report. I know it's long, I've tried to put
> to keep it concise and chunked, though.

Although this has come up in the past, I don't recall a resolution.

What is your plan to handle svn:eol-style?  We have some files in the 
tree which need that support and it isn't clear to me how that would 
work with the existing win32text extension provided with current 
mercurial releases.  (I've an outstanding patch to hg which should 
address some of these issues, but without the 'rules' being versioned I 
fear that would still fall short.)

Even more generally, how will you suggest Windows users work?  Will 
local files, in general, have windows line endings or unix?  If the 
latter, will there be hooks in-place to prevent editors on Windows 
'accidently' mixing eol styles?  If so, this cycles back to the first 
question - how would we know which files get treated that way?

Thanks,

Mark

From dinov at microsoft.com  Fri Jul  3 01:45:39 2009
From: dinov at microsoft.com (Dino Viehland)
Date: Thu, 2 Jul 2009 23:45:39 +0000
Subject: [Python-Dev] [IronPython] Exception for setting attributes of
	built-in type
In-Reply-To: <5b0248170906141209y4398defanacfd0bb62504e550@mail.gmail.com>
References: <5b0248170906141209y4398defanacfd0bb62504e550@mail.gmail.com>
Message-ID: <1A472770E042064698CB5ADC83A12ACD034067D5@TK5EX14MBXC116.redmond.corp.microsoft.com>

Seo wrote:
> Exception for setting attributes of built-in type differs between
> CPython and IronPython. This is not purely theoretical, as
> zope.interface tries to set Implements declaration as __implemented__
> attribute of built-in type object, and excepts TypeError.
>
> Python 2.6.1
> >>> object.flag = True
> TypeError: can't set attributes of built-in/extension type 'object'
>
> IronPython 2.6
> >>> object.flag = True
> AttributeError: 'object' object has no attribute 'flag'
>
> I was surprised that CPython raises TypeError. Library Reference seems
> to mention it here:
>
> exception AttributeError
> Raised when an attribute reference or assignment fails. (When an
> object does not support attribute references or attribute assignments
> at all, TypeError is raised.)
> http://docs.python.org/library/exceptions.html
>
> What does it mean that "an object does not support attribute
> references or attribute assignments at all"?

Even though Guido said we didn't have to match it I've updated IronPython
so that we'll throw the same exception now.  The change will be in
tomorrow's source push to CodePlex.  We've matched specific exceptions
in the past and this one is at least documented :)

From brett at python.org  Fri Jul  3 03:20:13 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 2 Jul 2009 18:20:13 -0700
Subject: [Python-Dev] I am back
In-Reply-To: <h2iv6k$an5$1@ger.gmane.org>
References: <bbaeab100907011621l70d46f83y230d8b1c76ae7063@mail.gmail.com> 
	<20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com> 
	<bbaeab100907021054y39c0ac01t203df8da145daf44@mail.gmail.com> 
	<h2iv6k$an5$1@ger.gmane.org>
Message-ID: <bbaeab100907021820l44685664of81833e7321c4b4c@mail.gmail.com>

On Thu, Jul 2, 2009 at 11:44, Terry Reedy <tjreedy at udel.edu> wrote:

> Brett Cannon wrote:
>
>>
>>
>> On Thu, Jul 2, 2009 at 04:39, Nick Coghlan <ncoghlan at gmail.com <mailto:
>> ncoghlan at gmail.com>> wrote:
>>
>>    Aahz wrote:
>>     > On Wed, Jul 01, 2009, Brett Cannon wrote:
>>     >> Anything happen while I was gone that I should be aware of that
>>    is not
>>     >> covered in a PEP?
>>     >
>>     > Yes.
>>
>>    I agree with Aahz ;)
>>
>>
>> Ah, the smart ass remarks never cease. =)
>>
>
> The biggest news is the release of 3.1, but that might have been recorded
> in the relevant PEP, and it certainly hard to miss.
>

Yeah, it hit Twitter immediately thanks to Benjamin.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090702/bb3c1140/attachment.htm>

From guido at python.org  Fri Jul  3 05:49:05 2009
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Jul 2009 20:49:05 -0700
Subject: [Python-Dev] pthread sem PyThread_acquire_lock
In-Reply-To: <20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com>
References: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com> 
	<20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com>
Message-ID: <ca471dc20907022049n21ddd52duf26bed436ae4557c@mail.gmail.com>

On Thu, Jul 2, 2009 at 4:04 PM, Jean-Paul Calderone<exarkun at divmod.com> wrote:
> On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" <greg at krypto.org>
>> If you want signals to actually be handled in a timely manner, its
>> best to leave the main thread of the program alone as a signal
>> handling thread that just spends its time in a loop of time.sleep(123)
>> calls rather than blocking on any sort of lock. ?Spawn other threads
>> to do the actual work in your program. ?Signals are delivered
>> indirectly in the existing CPython implementation by setting an
>> internal flag that the main interpreter thread polls on occasion so
>> blocking calls that do not interrupt and return early being called
>> from the main thread will effectively block signals.
>
> Yes, this is all true now. ?The question is why the implementation
> works that way, and whether it is desirable to keep it working that
> way.
>
> Considering *some* of the lock implementations make themselves not
> interruptable by threads while others don't bother, it seems like
> *some* change to the status quo is desirable.

I'm probably to blame for this, but IIRC when we did it, lock
acquisitions were not interruptible by signals. However, that was
before posix threads were even standardized. Nowadays I believe we use
pthreads everywhere except on Windows. But I know nearly nothing about
pthreads. If pthreads mutex acquisition is (or can easily be made)
interruptible by signals I'd be all for adding signal handling to
them, assuming it doesn't slow things down much. Also, if there are
ways to do the "try to acquire a lock or wait until a given timeout"
natively and while handling signals, that would be a great
improvement.

I'll probably regret saying this within an hour though,

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From dirkjan at ochtman.nl  Fri Jul  3 13:28:02 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Fri, 3 Jul 2009 13:28:02 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4D3C44.60504@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4D3C44.60504@gmail.com>
Message-ID: <ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>

On Fri, Jul 3, 2009 at 01:01, Mark Hammond<skippy.hammond at gmail.com> wrote:
> Although this has come up in the past, I don't recall a resolution.
>
> What is your plan to handle svn:eol-style? ?We have some files in the tree
> which need that support and it isn't clear to me how that would work with
> the existing win32text extension provided with current mercurial releases.
> ?(I've an outstanding patch to hg which should address some of these issues,
> but without the 'rules' being versioned I fear that would still fall short.)

What files would need what? Are there any files that really need to be
\r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember
one file was discussed separately, but I think the outcome there was
that it could just always be \r\n (since it wasn't used at all on
non-Windows platforms). Anyway, knowing specific requirements (or
where to find them) would help here.

> Even more generally, how will you suggest Windows users work? ?Will local
> files, in general, have windows line endings or unix? ?If the latter, will
> there be hooks in-place to prevent editors on Windows 'accidently' mixing
> eol styles? ?If so, this cycles back to the first question - how would we
> know which files get treated that way?

There will be a server-side hook to check whitespace. People will also
be able to install it for commit-time.

I think just using \n by default everywhere is a good default (though
I almost always use Windows client machine, I do all nearly all of my
development through a terminal on several Linux boxen), except where
it isn't. People who want to use can set up the win32text stuff to get
\r\n on Windows if they feel they need that -- we can provide
information about that in the dev FAQ (although it would be nice if
someone else who was more familiar with it -- like yourself! -- would
write it).

Cheers,

Dirkjan

From lukasz.langa at stxnext.pl  Fri Jul  3 14:57:37 2009
From: lukasz.langa at stxnext.pl (=?UTF-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 3 Jul 2009 13:57:37 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
Message-ID: <A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>


On 2009-06-22 at 14:23, Tarek Ziad? wrote:

> Hello,
>
> We have polished out PEP 376 and its code prototype at Distutils-SIG.
> It seems to fullfill now all the requirements,
> so I am mailing it here again, for a new round of feedback, if needed.


Hello.
I have read the current version of the PEP from trunk and would like  
to make some comments about it.

The .egg-info as a directory
============================
At our company, after much fiddling about the current distutils and  
setuptools, we have developed a solution that enables us to serve  
pseudo-eggs. What I mean by that is that by plugging into the PEP 302  
infrastructure, we were able to make non-filesystem-based repositories  
behave like `.egg` files. There are a couple of use cases for that  
implementation, the most important ones being:
* being able to deploy encrypted and sealed blobs with a binary  
loader, to discourage fiddling around the source code by its users
* being able to serve modules from a database back-end instead of the  
filesystem

This solution depends on the `EGG-INFO` to be fetched from the `.egg`  
itself. We wouldn't want to use a static filesystem-based folder  
because it breaks the encryption use-case by revealing the  
`SOURCES.txt` information. It also breaks the database back-end  
usecase by presenting static versioning information whereas the whole  
point of using a database backed "egg" is to have the newest version  
installed at all times.

Moreover, the proposed ``egginfo_dirname()`` routine is a step-back  
from the ``pkg_resources`` approach where we don't enforce resources  
to reside on a traditional filesystem.

The uninstall feature
=====================
This is a great feature but I don't seem to understand why there seems  
to be a consensus here that it's good for it not to even warn about  
dependencies, let alone manage them. ``easy_install`` and co. does  
manage dependencies while installing, why shouldn't it be symmetrical  
in that regard? Moreover, wouldn't dependency management be useful  
while using ``easy_install`` to *upgrade* an installation of a package?

The second issue is that while having ``python -m  
distutils.uninstall`` is the preferred way to go, I would argue that  
enabling an ``uninstall`` entry-point in ``setup.py`` is desirable as  
well, if only for retaining an intuitive symmetrical system where I  
can do ``python setup.py install`` and ``python setup.py uninstall``  
as well. I'm not forced to download the sources just to uninstall the  
package (``distutils.uninstall`` covers this use case) but if I do  
have the source code available, it would feel very natural to use the  
``setup.py`` provided.

There already is the precedent of ``setup.py develop --uninstall``.  
Having a ``setup.py uninstall`` could handle this use case as well.

More high-level remarks
=======================
This isn't probably the best place to cover this but may serve as a  
rationale for the above .egg-info problems we have with the proposed  
changes.

It's great that you push the metadata functionality to core distutils.  
At the same time, it would be feasible to think over the whole `egg`  
concept. We would argue that having the `egg` format as a container  
more-less agnostic to the underlying data storage structure would be  
**very** useful. Imagine installable `egg` support packages which  
would enable you to:
* use other compression formats beside ZIP
* use other means of module storage beside the filesystem
* use sealed and encrypted archives for deployment
* use programmatic `egg`s for licensing, etc.
* ``easy_install`` from different protocols, archive formats, etc.

Decoupling the `egg` format from the concrete "zip-based" or  
"directory-based" implementations is a step forward in that direction.  
In that regard, the solution PEP 376 is proposing isn't ultimately  
solving the issues we're having.

Thanks for your time.

-- 
Best regards,
?ukasz Langa
Senior Developer
STX-Next Sp. z o.o.
tel: +48 791 080 144
skype: lukaszlanga


From stephen at xemacs.org  Fri Jul  3 15:29:41 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 03 Jul 2009 22:29:41 +0900
Subject: [Python-Dev]  Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>

Dirkjan Ochtman writes:

 > Mercurial has two basic ways of using branches: cloned branches, where
 > each branch is kept in a separate repository, and named branches,
 > where each revision keeps metadata to note on which branch it belongs.
 > The former makes it easier to distinguish branches, at the expense of
 > requiring more disk space on the client. The latter makes it a little
 > easier to switch between branches, but often has somewhat unintuitive
 > results for people (though this has been getting better in recent
 > versions of Mercurial).

I'll have to try them again now that 1.3 is out, but I found Mercurial
named branches fundamentally unsuited to release management.

 > I'm still a bit on the fence about whether Python should adopt cloned
 > branches and named branches. Since it usually makes more sense to tag
 > releases on the maintenance branch, for example, mainline history
 > would not contain release tags if we used cloned branches.

Ditto named branches.  The problem is that (unless the internal
implementation has changed very recently) a Mercurial revision can be
on exactly one named branch (or on the trunk).

 > A disadvantage might be that the used clones will be a good bit larger
 > (since they essentially contain all other branches as well). This can
 > me mitigated by keeping non-release (feature) branches in separate
 > clones.

Which defeats the purpose of having named branches, really.  (I mean
the version control purpose; obviously it still can save disk space.)

 > Also note that it's still possible to clone a single named
 > branch from a combined clone, by specifying the branch as in hg clone
 > http://hg.python.org/main/#2.6-maint.

Unless you're really short on space, though, that's not a big deal.
What would be more important to me (not that I matter for the purpose
of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
the other way around: pulling an external branch into a named branch.
I have a feeling that working with such a repository with others would
be a little difficult.

 > too many problems. For example, Mercurial itself just uses '1.2.1' as
 > a tag, where CPython would currently use r121.

Stick with the CPython notation.  At XEmacs, continuity of tags has
made our beta testers happy.  (Well, the two of them who bothered to
mention it, anyway. :-)

 > code.python.org was also proposed as the hostname. Personally, I think
 > that using the VCS name in the hostname is good because it prevents
 > confusion: it should be clear that you can't use svn or bzr for
 > hg.python.org.

Agreed, although "can't" is a little too strong.  It might work (there
are a lot of places where http://ftp.example.com works just fine, for
example), but we don't want people to expect it to, and
"http://REPOHOST.python.org/" should take your browser or your client
to the official repo (which will be the hg repo), not to some index of
repos that happen to live on the same host.

 > Mercurial itself has a stable branch which is a ''strict'' subset of
 > the unstable branch.

As others (MvL, I think) have commented, this isn't really relevant to
Python which generally has four mainlines going at once.  I don't see
why the requirements are going to change with the shift to hg, and I
see no reason why hg won't handle the existing workflow just fine.

Note that PEP 374 was written on the assumption that the existing
workflow will *not* change until the committers have gotten used to
Mercurial, and then it will change in the natural way.  Ie, some one
will say, "you know, this bit of red tape isn't needed any more if we
do X, so let's do X", and after a cascade of 100*"+1 " we do X. :-)

Other than that, looks to be shaping up well.  (Note: I don't have any
comments on subversion-specific aspects, as XEmacs went directly from
CVS to Mercurial).

From mhammond at skippinet.com.au  Fri Jul  3 15:31:17 2009
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Fri, 03 Jul 2009 23:31:17 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A4D3C44.60504@gmail.com>
	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
Message-ID: <4A4E0825.5020407@skippinet.com.au>

On 3/07/2009 9:28 PM, Dirkjan Ochtman wrote:
> On Fri, Jul 3, 2009 at 01:01, Mark Hammond<skippy.hammond at gmail.com>  wrote:
>    
>> Although this has come up in the past, I don't recall a resolution.
>>
>> What is your plan to handle svn:eol-style?  We have some files in the tree
>> which need that support and it isn't clear to me how that would work with
>> the existing win32text extension provided with current mercurial releases.
>>   (I've an outstanding patch to hg which should address some of these issues,
>> but without the 'rules' being versioned I fear that would still fall short.)
>>      
>
> What files would need what? Are there any files that really need to be
> \r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember
> one file was discussed separately, but I think the outcome there was
> that it could just always be \r\n (since it wasn't used at all on
> non-Windows platforms). Anyway, knowing specific requirements (or
> where to find them) would help here.
>
>    
>> Even more generally, how will you suggest Windows users work?  Will local
>> files, in general, have windows line endings or unix?  If the latter, will
>> there be hooks in-place to prevent editors on Windows 'accidently' mixing
>> eol styles?  If so, this cycles back to the first question - how would we
>> know which files get treated that way?
>>      
>
> There will be a server-side hook to check whitespace. People will also
> be able to install it for commit-time.
>
> I think just using \n by default everywhere is a good default (though
> I almost always use Windows client machine, I do all nearly all of my
> development through a terminal on several Linux boxen), except where
> it isn't.
So we must work without effective EOL support?  I fear we will end up 
like the mozilla hg repo with some files in windows line endings and 
some with linux.  While my editing tools are good enough to preserve 
existing EOL styles, I've found myself accidentally checking in new \r\n 
terminated files in a repo which otherwise uses \n line endings.  IMO, 
SVN's EOL support was better than no EOL support.

>   People who want to use can set up the win32text stuff to get
> \r\n on Windows if they feel they need that -- we can provide
> information about that in the dev FAQ (although it would be nice if
> someone else who was more familiar with it -- like yourself! -- would
> write it).
>    
This is exactly why I was asking for your advice - I can't work out how 
to work effectively with win32text as it stands myself, so remain stuck 
accidently checking in files with inappropriate line endings and stuck 
working out how to move pywin32's CVS repo with abandoning the very 
primitive EOL safety it offers...

Cheers,

Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/02e9d3db/attachment-0001.htm>

From dirkjan at ochtman.nl  Fri Jul  3 15:43:54 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Fri, 3 Jul 2009 15:43:54 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E0825.5020407@skippinet.com.au>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4D3C44.60504@gmail.com>
	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au>
Message-ID: <ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>

On Fri, Jul 3, 2009 at 15:31, Mark Hammond<mhammond at skippinet.com.au> wrote:
> So we must work without effective EOL support?? I fear we will end up like
> the mozilla hg repo with some files in windows line endings and some with
> linux.? While my editing tools are good enough to preserve existing EOL
> styles, I've found myself accidentally checking in new \r\n terminated files
> in a repo which otherwise uses \n line endings.? IMO, SVN's EOL support was
> better than no EOL support.

This is why we'll have hooks -- to prevent you from pushing changesets
with inappropriate, to say the least, and, if you're willing to do a
little bit of extra work, to prevent you from committing them.

> This is exactly why I was asking for your advice - I can't work out how to
> work effectively with win32text as it stands myself, so remain stuck
> accidently checking in files with inappropriate line endings and stuck
> working out how to move pywin32's CVS repo with abandoning the very
> primitive EOL safety it offers...

As long as the difference between \r\n- and \n-based files is clear
and can be reasoned about, I don't see why having some of both (I'm
assuming an overwhelming majority will have one, and only a few the
other) is a big problem. But feel free to enlighten me!

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Fri Jul  3 15:49:31 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Fri, 3 Jul 2009 15:49:31 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>

On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull<stephen at xemacs.org> wrote:
> I'll have to try them again now that 1.3 is out, but I found Mercurial
> named branches fundamentally unsuited to release management.

Can you explain why, please? It's not clear from what you say below.

> Ditto named branches. ?The problem is that (unless the internal
> implementation has changed very recently) a Mercurial revision can be
> on exactly one named branch (or on the trunk).

That's still true.

> Which defeats the purpose of having named branches, really. ?(I mean
> the version control purpose; obviously it still can save disk space.)

Why does it defeat the purpose? What, in your opinion, is the purpose?

> Unless you're really short on space, though, that's not a big deal.
> What would be more important to me (not that I matter for the purpose
> of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
> the other way around: pulling an external branch into a named branch.
> I have a feeling that working with such a repository with others would
> be a little difficult.

Can you give an example?

> Stick with the CPython notation. ?At XEmacs, continuity of tags has
> made our beta testers happy. ?(Well, the two of them who bothered to
> mention it, anyway. :-)

Right; Benjamin also mentioned that processing the tags just to be
consistent with the recent tagging scheme would probably be the best
solution.

> As others (MvL, I think) have commented, this isn't really relevant to
> Python which generally has four mainlines going at once. ?I don't see
> why the requirements are going to change with the shift to hg, and I
> see no reason why hg won't handle the existing workflow just fine.

It will handle it, for sure, but I think it would all go easier if we
could work with stricter subset branches (and leave the effective
cherrypicking for the occasional problem).

Cheers,

Dirkjan

From p.f.moore at gmail.com  Fri Jul  3 15:54:13 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 3 Jul 2009 14:54:13 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
Message-ID: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>

2009/7/3 ?ukasz Langa <lukasz.langa at stxnext.pl>:
>
> On 2009-06-22 at 14:23, Tarek Ziad? wrote:
>
>> Hello,
>>
>> We have polished out PEP 376 and its code prototype at Distutils-SIG.
>> It seems to fullfill now all the requirements,
>> so I am mailing it here again, for a new round of feedback, if needed.
>
>
> Hello.
> I have read the current version of the PEP from trunk and would like to make
> some comments about it.
>
> The .egg-info as a directory
> ============================
> At our company, after much fiddling about the current distutils and
> setuptools, we have developed a solution that enables us to serve
> pseudo-eggs. What I mean by that is that by plugging into the PEP 302
> infrastructure, we were able to make non-filesystem-based repositories
> behave like `.egg` files. There are a couple of use cases for that
> implementation, the most important ones being:
> * being able to deploy encrypted and sealed blobs with a binary loader, to
> discourage fiddling around the source code by its users
> * being able to serve modules from a database back-end instead of the
> filesystem
>
> This solution depends on the `EGG-INFO` to be fetched from the `.egg`
> itself. We wouldn't want to use a static filesystem-based folder because it
> breaks the encryption use-case by revealing the `SOURCES.txt` information.
> It also breaks the database back-end usecase by presenting static versioning
> information whereas the whole point of using a database backed "egg" is to
> have the newest version installed at all times.

This is a good point. Distutils only installs files in the filesystem
- it has no facilities for installing packages based on any other sort
of PEP 302 based importers. Hence, PEP 376 in principle should only
relate to filesystem-based distributions. But it also mentions
zipfile-based distributions: "Notice that the API is organized in five
classes that work with directories and Zip files (so it works with
files included in Zip files, see PEP 273 for more details [8])."

This is wrong. The PEP should either (1) restrict itself to filesystem
based implementations (leaving the problem of other PEP 302 loaders to
systems that manage these) or (2) be defined in a sufficiently general
way that it can be implemented for *any* PEP 302 based loader - which
probably means extending the PEP 302 protocols - and supplying zipfile
functions as an example of how this is used.

I believe that (1) is unlikely to be sufficient for real world use.
Zip files (eggs, py2exe embedded modules, etc) are far too important a
real world use case to ignore. The problem with (2) is that it
requires significant extra work. But special-casing zip files (as the
present implementation appears to do) will break as soon as any other
PEP 302 compliant format becomes popular.

> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the
> ``pkg_resources`` approach where we don't enforce resources to reside on a
> traditional filesystem.

On the other hand, pkgutil.get_data is the standard library means of
reading resources from a package. This is PEP 302 compliant now. This
new PEP doesn't affect that.

> The uninstall feature
> =====================
> This is a great feature but I don't seem to understand why there seems to be
> a consensus here that it's good for it not to even warn about dependencies,
> let alone manage them. ``easy_install`` and co. does manage dependencies
> while installing, why shouldn't it be symmetrical in that regard? Moreover,
> wouldn't dependency management be useful while using ``easy_install`` to
> *upgrade* an installation of a package?

easy_install is not a standard library feature - the fact that it has
no uninstall facility is not relevant.

python setup.py install is the standard library feature. The new
uninstall is intended to correspond to that.

> More high-level remarks
> =======================
> This isn't probably the best place to cover this but may serve as a
> rationale for the above .egg-info problems we have with the proposed
> changes.
>
> It's great that you push the metadata functionality to core distutils. At
> the same time, it would be feasible to think over the whole `egg` concept.
> We would argue that having the `egg` format as a container more-less
> agnostic to the underlying data storage structure would be **very** useful.
> Imagine installable `egg` support packages which would enable you to:
> * use other compression formats beside ZIP
> * use other means of module storage beside the filesystem
> * use sealed and encrypted archives for deployment
> * use programmatic `egg`s for licensing, etc.

PEP 302 gives you (in theory) all of this now. I'd recommend you look
at it - and at Brett's importlib, which will make experimenting with
such things far easier.

What PEP 302 doesn't provide is package management. But Python itself
doesn't provide package management, except in the form of distutils
setup.py install - which is solely filesystem based.

Maybe there's a case for extending PEP 302 and distutils to allow
integrated management of other forms of importer format, but that's a
huge other project, which no-one to my knowledge is even looking at.

> * ``easy_install`` from different protocols, archive formats, etc.

easy_install is not a standard library feature, so is not relevant here.

> Decoupling the `egg` format from the concrete "zip-based" or
> "directory-based" implementations is a step forward in that direction. In
> that regard, the solution PEP 376 is proposing isn't ultimately solving the
> issues we're having.

Eggs are fundamentally a PEP 302 zip file format. There are some extra
bits of metadata for setuptools/easy_install in there (as I understand
things) but essentially they are zip files. When you say "decoupling
the egg format", I assume you mean "decoupling the egg metadata" -
which is fine, but to properly decouple, you need API level access to
the metadata. PEP 376 offers read-only access, but as you rightly
point out, it is only for filesystem data (and some form of zip file,
which appears to be limited in some way, as it isn't PEP 302 based,
and the actual format isn't defined anywhere).

The basic point here is that PEP 376 needs to define precisely how
pkgutil.get_distributions() scans sys.path looking for ".egg-info
directories". What does it do for sys.path entries that don't
correspond to filesystem directories? (Note - these may or may not be
zip files. Even if they are zip files, an earlier entry on
sys.path_hooks could have taken precedence. At the very least, you
should only process path entries as zip files if their importer - in
sys.path_importer_cache or via an explicit path hook scan - is a
zipimporter object.).

To be honest, this is a major can of worms. But if PEP 376 is not
going to support PEP 302, then it must state that fact explicitly, to
avoid giving people false expectations - particularly with Brett's
importlib in Python 3.1, which will make it far easier for people to
experiment with new packaging formats such as the ones Lukasz mentions
above. And it MUST fail gracefully in the face of unsupported importer
types.

Paul.

From ncoghlan at gmail.com  Fri Jul  3 16:04:57 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 00:04:57 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E0825.5020407@skippinet.com.au>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au>
Message-ID: <4A4E1009.4050901@gmail.com>

Mark Hammond wrote:
> So we must work without effective EOL support?  I fear we will end up
> like the mozilla hg repo with some files in windows line endings and
> some with linux.  While my editing tools are good enough to preserve
> existing EOL styles, I've found myself accidentally checking in new \r\n
> terminated files in a repo which otherwise uses \n line endings.  IMO,
> SVN's EOL support was better than no EOL support.

If Mercury doesn't have automatic whitespace conversion along the lines
of svn:eol-style, then the existing white-space checking script probably
needs to be updated to enforce the appropriate line endings for all files.

If we default to Unix line endings for most files, then the checking
script can be made aware of which files should always have Windows line
endings (I believe the various Visual Studio files need them, batch
files are probably best left with Windows line endings in the
repository, and I expect there are other files in PC and PCbuild that
need them as well).

However, I expect that would still be painful to work with for Windows
developers, even if it prevented any line ending problems from making
their way into the main repository. I believe that is where the
win32text extensions can help. Looking at the Wiki page for win32text
[1], I believe it would be a matter of configuring the extension to
encode and decode all files with the extensions:

*.py
*.pyw
*.h
*.c
*.in
*.rst
*.asdl

That said, I don't see a way to tell win32text to also translate files
which don't have an extension at all (e.g. NEWS or ACKS), and there
doesn't seem to be a way to tell it to *skip* files matching certain
patterns (if there was, we could just tell it to ignore extensions like
.bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert
everything else)

Cheers,
Nick.

[1] http://mercurial.selenic.com/wiki/Win32TextExtension

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Fri Jul  3 16:28:02 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 00:28:02 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
Message-ID: <4A4E1572.3000801@gmail.com>

Paul Moore wrote:
> To be honest, this is a major can of worms. But if PEP 376 is not
> going to support PEP 302, then it must state that fact explicitly, to
> avoid giving people false expectations - particularly with Brett's
> importlib in Python 3.1, which will make it far easier for people to
> experiment with new packaging formats such as the ones Lukasz mentions
> above. And it MUST fail gracefully in the face of unsupported importer
> types.

importlib is in 2.7 as well.

I agree that even if the reference implementation of PEP 376 only
handles normal directories and zipfiles, the PEP itself needs to explain
how the developer of a custom PEP 302 importer or loader can hook into
the mechanism in order to provide metadata that distutils can
understand. Or, as you suggest, state explicitly that the proposal at
this stage is specifically limited to filesystem and zipfile packages
and defer extension to arbitrary PEP 302 importers and loaders to a
later PEP.

To be honest, I'd personally be OK with the latter strategy - while
other PEP 302 loaders and importers do exist (as Lukasz'z post shows),
it would be unfortunate to unduly hold up improved installation metadata
for the vast majority of typical use cases while we try to figure out
ways to support the more esoteric use cases. Supporting both would
obviously be better, but given the choice between the status quo and
partial support, I believe this is a case where the partial support
would still be worthwhile.

I will note (and I believe this is also the main point that Lukasz was
making) that having the distribution metadata outside the distribution
as currently proposed in PEP 376 is going to make any eventual PEP 302
integration much harder - 302 importers only provide a mechanism for
accessing files inside the distribution, not "adjacent" to them, so the
mechanism in the PEP doesn't generalise properly.

I suspect this limitation of the PEP 302 APIs is the origin of the
setuptools format that embeds the metadata inside the distribution - it
lets you get at the metadata without having to assume that it exists
directly on the filesystem anywhere.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Fri Jul  3 16:31:31 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 3 Jul 2009 16:31:31 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
Message-ID: <94bdd2610907030731s395a962byced058b97c2ef6d@mail.gmail.com>

2009/7/3 Paul Moore <p.f.moore at gmail.com>:
> This is a good point. Distutils only installs files in the filesystem
> - it has no facilities for installing packages based on any other sort
> of PEP 302 based importers. Hence, PEP 376 in principle should only
> relate to filesystem-based distributions. But it also mentions
> zipfile-based distributions: "Notice that the API is organized in five
> classes that work with directories and Zip files (so it works with
> files included in Zip files, see PEP 273 for more details [8])."
>
> This is wrong. The PEP should either (1) restrict itself to filesystem
> based implementations (leaving the problem of other PEP 302 loaders to
> systems that manage these) or (2) be defined in a sufficiently general
> way that it can be implemented for *any* PEP 302 based loader - which
> probably means extending the PEP 302 protocols - and supplying zipfile
> functions as an example of how this is used.
>
> I believe that (1) is unlikely to be sufficient for real world use.
> Zip files (eggs, py2exe embedded modules, etc) are far too important a
> real world use case to ignore. The problem with (2) is that it
> requires significant extra work. But special-casing zip files (as the
> present implementation appears to do) will break as soon as any other
> PEP 302 compliant format becomes popular.
>> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the
>> ``pkg_resources`` approach where we don't enforce resources to reside on a
>> traditional filesystem.
>
> On the other hand, pkgutil.get_data is the standard library means of
> reading resources from a package. This is PEP 302 compliant now. This
> new PEP doesn't affect that.
>

Right. While it would be feasible to make pgutil works with PEP 302 loaders,
we would still need to define in a generic way the content of the RECORD files.

Right now it works for directory and zipped files since it's expressed
with '/' separated paths.

And if I understand PEP 302 right, any backend would be able to handle
these paths
no matter how they are stored, as long as the implement APIs like get_data()

> What PEP 302 doesn't provide is package management. But Python itself
> doesn't provide package management, except in the form of distutils
> setup.py install - which is solely filesystem based.
>
> Maybe there's a case for extending PEP 302 and distutils to allow
> integrated management of other forms of importer format, but that's a
> huge other project, which no-one to my knowledge is even looking at.

Sounds like a fully-featured packaging managment system, which is
imho, out of scope.

And I don't see PEP 376 making it impossible for someone to build such a
packaging system on the top of distutils.

I've started one myself for the sake of experimentation, with built-in
multiversion
support, for a full replacement of site-packages.

> Eggs are fundamentally a PEP 302 zip file format. There are some extra
> bits of metadata for setuptools/easy_install in there (as I understand
> things) but essentially they are zip files. When you say "decoupling
> the egg format", I assume you mean "decoupling the egg metadata" -
> which is fine, but to properly decouple, you need API level access to
> the metadata. PEP 376 offers read-only access, but as you rightly
> point out, it is only for filesystem data (and some form of zip file,
> which appears to be limited in some way, as it isn't PEP 302 based,
> and the actual format isn't defined anywhere).

And also PEP 376 goal is to define a single standard location of
egg-info files for filesystem data.

The zip form was built so it could work with zipped site-packages directories,
like what the py2app project uses.

>
> The basic point here is that PEP 376 needs to define precisely how
> pkgutil.get_distributions() scans sys.path looking for ".egg-info
> directories". What does it do for sys.path entries that don't
> correspond to filesystem directories? (Note - these may or may not be
> zip files. Even if they are zip files, an earlier entry on
> sys.path_hooks could have taken precedence. At the very least, you
> should only process path entries as zip files if their importer - in
> sys.path_importer_cache or via an explicit path hook scan - is a
> zipimporter object.).

I'll add more details on that part. right now it visits directories
and zip files.

Tarek

From ziade.tarek at gmail.com  Fri Jul  3 16:44:34 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 3 Jul 2009 16:44:34 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4E1572.3000801@gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<4A4E1572.3000801@gmail.com>
Message-ID: <94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com>

On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan<ncoghlan at gmail.com> wrote:
> I will note (and I believe this is also the main point that Lukasz was
> making) that having the distribution metadata outside the distribution
> as currently proposed in PEP 376 is going to make any eventual PEP 302
> integration much harder - 302 importers only provide a mechanism for
> accessing files inside the distribution, not "adjacent" to them, so the
> mechanism in the PEP doesn't generalise properly.
>
> I suspect this limitation of the PEP 302 APIs is the origin of the
> setuptools format that embeds the metadata inside the distribution - it
> lets you get at the metadata without having to assume that it exists
> directly on the filesystem anywhere.

Maybe we could rework the pkgutil classes for PEP 376 so they look
like an implementation
of the PEP 302 protocol for directories and zip files, with an extra
"get_metadata()" API
and state that it could be an extension for PEP 302 later.

From g.brandl at gmx.net  Fri Jul  3 17:17:02 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 03 Jul 2009 17:17:02 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <h2l7e7$mt$1@ger.gmane.org>

Dirkjan Ochtman schrieb:

> - Fifth, here's a list of things, off the top of my head, that still need doing:
> 
> * Get agreement on branch strategy and branch processing (list of
> branches + proposed handling at
> http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
> REVIEW

Do you have a key to the second column in that file? E.g. the difference
between "strip" and "discard" is not clear to me. "strip partial"?

Why are there branch names starting with "../"?

[PEP 385]

> ==============================================================
> 
> Motivation
> 
> After having decided to switch to the Mercurial DVCS, the actual
> migration still has to be performed. In the case of an important piece
> of infrastructure like the version control system for a large,
> distributed project like Python, this is a significant effort. This
> PEP is an attempt to describe the steps that must be taken for further
> discussion. It's somewhat similar to PEP 347 [1], which discussed the
> migration to SVN.
> 
> To make the most of hg, I (Dirkjan) would like to make a high-fidelity
> conversion, such that (a) as much of the svn metadata as possible is
> retained, and (b) all metadata is converted to formats that are common
> in Mercurial. This way, tools written for Mercurial can be optimally
> used. In order to do this, I want to use the hgsubversion [2] software
> to do an initial conversion. This hg extension is focused on providing
> high-quality conversion from Subversion to Mercurial for use in
> two-way correspondence, meaning it doesn't throw away as much
> available metadata as other solutions.
> 
> Such a conversion also seems like a good time to reconsider the
> contents of the repository and determine if some things are still
> valuable. In this spirit, the following sections also propose
> discarding some of the older metadata.
> Timeline
> 
> TBD; needs fully working hgsubversion and consensus on this document.
> Transition plan
> Branch strategy
> 
> Mercurial has two basic ways of using branches: cloned branches, where
> each branch is kept in a separate repository, and named branches,
> where each revision keeps metadata to note on which branch it belongs.
> The former makes it easier to distinguish branches, at the expense of
> requiring more disk space on the client. The latter makes it a little
> easier to switch between branches, but often has somewhat unintuitive
> results for people (though this has been getting better in recent
> versions of Mercurial).
> 
> I'm still a bit on the fence about whether Python should adopt cloned
> branches and named branches. Since it usually makes more sense to tag
> releases on the maintenance branch, for example, mainline history
> would not contain release tags if we used cloned branches. Also,
> Mercurial 1.2 and 1.3 have the necessary tools to make named branches
> less painful (because they can be properly closed and closed heads are
> no longer considered in relevant cases).
> 
> A disadvantage might be that the used clones will be a good bit larger
> (since they essentially contain all other branches as well). This can
> me mitigated by keeping non-release (feature) branches in separate
> clones. Also note that it's still possible to clone a single named
> branch from a combined clone, by specifying the branch as in hg clone
> http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a
> separate clone problably also makes sense.

* Does it work with "hg pull" etc. too, afterwards?
* Does it support more than one branch?

> XXX To do: size comparison for selected separation scenarios.
> Converting branches
> 
> There are quite a lot of branches in SVN's branches directory. I
> propose to clean this up a bit, by employing the following the
> strategy:
> 
>     * Keep all release (maintenance) branches
>     * Discard branches that haven't been touched in 18 months, unless
> somone indicates there's still interest in such a branch
>     * Keep branches that have been touched in the last 18 months,
> unless someone indicates the branch can be deprecated

I would just kill all feature branches unless someone indicates it is
still used.  There are very few active feature branches.

(I guess in the case a branch gets killed erroneously it could still be
re-created after the conversion?)

> Converting tags
> 
> The SVN tags directory contains a lot of old stuff. Some of these are
> not, in fact, full tags, but contain only a smaller subset of the
> repository. I think we should keep all release tags, and consider
> other tags for inclusion based on requests from the developer
> community. I'd like to consider unifying the release tag naming scheme
> to make some things more consistent, if people feel that won't create
> too many problems. For example, Mercurial itself just uses '1.2.1' as
> a tag, where CPython would currently use r121.

+1 for readable tag names.
+1 for throwing out old questionable tag names.

> Generating .hgignore
> 
> The .hgignore file can be used in Mercurial repositories to help
> ignore files that are not eligible for version control. It does this
> by employing several possible forms of pattern matching. The current
> Python repository already includes a rudimentary .hgignore file to
> help with using the hg mirrors.
> 
> It might be useful to have the .hgignore be generated automatically
> from svn:ignore properties. This would make sure all historic
> revisions also have useful ignore information (though one could argue
> ignoring isn't really relevant to just checking out an old revision).

I guess that's not necessary.  People can just add stuff to .hgignore
when they see something that should be there.

> hg-ssh
> 
> Developers should access the repositories through ssh, similar to the
> current setup. Public keys can be used to grant people access to a
> shared hg@ account. A hgwebdir instance should also be set up for easy
> browsing and read-only access. If we're using ssh, developers should
> trivially be able to start new clones (for longer-term features that
> profit from a separate branch).

+1.

> Hooks
> 
> A number of hooks is currently in use. The hg equivalents for these
> should be developed and deployed. The following hooks are being used:
> 
>     * check whitespace: a hook to reject commits in case the
> whitespace doesn't match the rules for the Python codebase. Should be
> straightforward to re-implement from the current version. We can also
> offer a whitespace hook for use with client-side repositories that
> people can use; it could either warn about whitespace issues and/or
> truncate trailing whitespace from changed lines. Open issue: do we
> check only the tip after each push, or do we check every commit in a
> changegroup?

Only checking the tip would make it possible for people to revert their
whitespace commits, but then -- if they have the local hook -- they
shouldn't do that anyway.

>     * commit mails: we can leverage the notify extension for this

As long as it can send diffs...

>     * check contributors: in the current setup, all changesets bear
> the username of committers, who must have signed the contributor
> agreement. In a DVCS, the committers are not necessarily the same
> people who push, and so we can't check if the committer is a
> contributor. We could use a hook to check if the committer is a
> contributor if we keep a list of registered contributors.

That gets very ugly as soon as you start pulling from repos that just
fix a small typo or so.

> code.python.org was also proposed as the hostname. Personally, I think
> that using the VCS name in the hostname is good because it prevents
> confusion: it should be clear that you can't use svn or bzr for
> hg.python.org.

Yes, and it mirrors svn.python.org.

> Mercurial itself has a stable branch which is a ''strict'' subset of
> the unstable branch. This means that generally all fixes for the
> stable branch get committed against the tip of the stable branch, then
> they get merged into the unstable branch (which already contains the
> parent of the new cset). This provides a largely frictionless
> environment for moving changes from stable to unstable branches.
> Mistakes, where a change that should go on stable goes on unstable
> first, do happen, but they're usually easy to fix. That can be done by
> copying the change over to the stable branch, then trivial-merging
> with unstable -- meaning the merge in fact ignores the parent from the
> stable branch).
> 
> This strategy means a little more work for regular committers, because
> they have to think about whether their change should go on stable or
> unstable; they may even have to ask someone else (the RM) before
> committing. But it also relieves a dedicated group of committers of
> regular backporting duty, in addition to making it easier to work with
> the tool.

Strong +1 for that.

> I propose that the revision identifier will be the short version of
> hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
> (instead of 'M') if the working directory from which it was built was
> modified. This mirrors the output of the hg id command, which is
> intended for this kind of usage.
> 
> For the tag/branch identifier, I propose that hg will check for tags
> on the currently checked out revision, use the tag if there is one
> ('tip' doesn't count), and uses the branch name otherwise.
> sys.subversion becomes
> 
>     * ('CPython', '2.6.2', 'dd3ebf81af43')
>     * ('CPython', 'default', 'af694c6a888c+')
> 
> and the build info string becomes
> 
>     * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33'
>     * 'default:af694c6a888c+, Jun 2 2009, 01:24:14'
>
> This reflects that the default branch in hg is called 'default'
> instead of Subversion's 'trunk', and reflects the proposed new tag
> format.

Looks good to me.

cheers,
Georg


From status at bugs.python.org  Fri Jul  3 18:07:18 2009
From: status at bugs.python.org (Python tracker)
Date: Fri,  3 Jul 2009 18:07:18 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20090703160718.2935E78611@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (06/26/09 - 07/03/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2231 open (+26) / 16000 closed (+38) / 18231 total (+64)

Open issues with patches:   890

Average duration of open issues: 661 days.
Median duration of open issues: 413 days.

Open Issues Breakdown
   open  2203 (+26)
pending    28 ( +0)

Issues Created Or Reopened (65)
_______________________________

Remove checks for win NT                                         06/29/09
CLOSED http://bugs.python.org/issue4856    reopened amaury.forgeotdarc            
       patch                                                                   

Rstrip Incorrectly Strips Some Strings                           06/26/09
CLOSED http://bugs.python.org/issue6346    created  mkubilus                      
                                                                               

hpux11.00-parisc: dtoa.c: "Failed to find an exact-width 32-bit  06/26/09
CLOSED http://bugs.python.org/issue6347    created  srid                          
       patch                                                                   

solaris/aix: Py_Initialize: can't initialize sys standard stream 06/26/09
       http://bugs.python.org/issue6348    created  srid                          
                                                                               

email.Maildir cannot roundtrip messages.                         06/27/09
CLOSED http://bugs.python.org/issue6349    created  bremner                       
                                                                               

Example at end of HTMLParser documentation uses old-style print  06/27/09
CLOSED http://bugs.python.org/issue6350    created  MLModel                       
                                                                               

Compiler warning in _cursesmodule.c                              06/28/09
       http://bugs.python.org/issue6351    created  hagen                         
                                                                               

Compiler warning in unicodeobject.c                              06/28/09
       http://bugs.python.org/issue6352    created  hagen                         
       patch                                                                   

"L" integer suffix in Python 3.1 tutorial                        06/28/09
CLOSED http://bugs.python.org/issue6353    created  delroth                       
       patch                                                                   

Old floating point representation in 3.1 tutorial                06/28/09
CLOSED http://bugs.python.org/issue6354    created  delroth                       
                                                                               

bogus NULL check in PyCapsule                                    06/28/09
CLOSED http://bugs.python.org/issue6355    created  bkramer                       
       patch                                                                   

Segfault of sys.setrecursionlimit	if limit exceeds the value 180 06/28/09
CLOSED http://bugs.python.org/issue6356    created  str1442                       
                                                                               

tempfile.NamedTemporaryFile does not accept the delete= paramete 06/28/09
CLOSED http://bugs.python.org/issue6357    created  stutzbach                     
                                                                               

os.popen exit code inconsistent                                  06/28/09
       http://bugs.python.org/issue6358    created  abbeyj                        
                                                                               

pyexpat.c calls trace function incorrectly for exceptions        06/28/09
       http://bugs.python.org/issue6359    created  nedbat                        
                                                                               

Simplify string decoding in xmlrpc.client.                       06/29/09
       http://bugs.python.org/issue6360    created  alexandre.vassalotti          
       patch                                                                   

I/O object wrappers shouldn't close their underlying file when d 06/29/09
       http://bugs.python.org/issue6361    created  alexandre.vassalotti          
                                                                               

multiprocessing: handling of errno after signals in sem_acquire( 06/29/09
       http://bugs.python.org/issue6362    created  ryles                         
                                                                               

__future__ statements break doctest                              06/29/09
CLOSED http://bugs.python.org/issue6363    created  pooryorick                    
                                                                               

freeze tool broken in Python 3.x                                 06/29/09
       http://bugs.python.org/issue6364    created  hagen                         
                                                                               

distutils duplicates package directory for C extensions in 3.1 f 06/29/09
CLOSED http://bugs.python.org/issue6365    created  scoder                        
                                                                               

rare assertion failure in test_multiprocessing                   06/29/09
       http://bugs.python.org/issue6366    created  pitrou                        
                                                                               

Change the instruction pointer in python                         06/29/09
CLOSED http://bugs.python.org/issue6367    created  vinoth                        
                                                                               

Fix unused variable warning introduced by commit r73603.         06/29/09
CLOSED http://bugs.python.org/issue6368    created  vshenoy                       
       patch                                                                   

binhex buggy in py3k                                             06/29/09
CLOSED http://bugs.python.org/issue6369    created  pitrou                        
       patch                                                                   

Bad performance of colllections.Counter at initialisation from a 06/29/09
CLOSED http://bugs.python.org/issue6370    created  SilentGhost                   
                                                                               

Minor internal link errors in Optparse doc                       06/29/09
CLOSED http://bugs.python.org/issue6371    created  pakal                         
                                                                               

Make all switches keyword-only                                   06/29/09
CLOSED http://bugs.python.org/issue6372    created  ptn                           
                                                                               

SystemError in encoder                                           06/29/09
CLOSED http://bugs.python.org/issue6373    created  glynnc                        
       patch                                                                   

Confused by subprocess API documentation                         06/29/09
CLOSED http://bugs.python.org/issue6374    created  pakal                         
       patch                                                                   

compile failure in Modules/python.c                              06/29/09
       http://bugs.python.org/issue6375    created  adamg                         
       patch                                                                   

description of Decinal.logical_invert is incorrect               06/29/09
CLOSED http://bugs.python.org/issue6376    created  mft                           
                                                                               

distutils compiler switch ignored                                06/29/09
       http://bugs.python.org/issue6377    created  atuining                      
       patch                                                                   

Patch to make 'idle.bat' run idle.pyw using appropriate Python i 06/29/09
       http://bugs.python.org/issue6378    created  srid                          
       patch                                                                   

Add first() function that would take first matching case from an 06/29/09
CLOSED http://bugs.python.org/issue6379    created  gandalf                       
                                                                               

Deadlock during the "import" in the fork()'ed child process if f 06/29/09
       http://bugs.python.org/issue6380    created  hdn                           
                                                                               

test_urllib2_localnet sporadic failures closing socket           06/30/09
       http://bugs.python.org/issue6381    created  r.david.murray                
                                                                               

test_socketserver fails on trunk in test_ForkingTCPServer        06/30/09
       http://bugs.python.org/issue6382    created  r.david.murray                
                                                                               

error in unicodedata.numeric(u"\u2187") and 2188                 06/30/09
CLOSED http://bugs.python.org/issue6383    created  vernondcole                   
       patch, needs review                                                     

Permalink to built-in exception class hierarchy documentation    06/30/09
CLOSED http://bugs.python.org/issue6384    created  MLModel                       
                                                                               

two lines of typographical inconsistency in doc of exception cla 06/30/09
CLOSED http://bugs.python.org/issue6385    created  MLModel                       
                                                                               

importing yields unexpected results when initial script is a sym 06/30/09
       http://bugs.python.org/issue6386    created  jbeulich                      
                                                                               

floor division gives different result for int versus float.      06/30/09
CLOSED http://bugs.python.org/issue6387    created  drj                           
                                                                               

platform.python_implementation does not work on IronPython       06/30/09
CLOSED http://bugs.python.org/issue6388    created  tbleier                       
                                                                               

os.chmod() documentation refers to non-existent documentation in 06/30/09
CLOSED http://bugs.python.org/issue6389    created  mhearne808                    
       easy                                                                    

File reads past EOF in "w+b" mode                                06/30/09
CLOSED http://bugs.python.org/issue6390    created  bmearns                       
                                                                               

Incorrect description of unittest.TestCase.run                   06/30/09
CLOSED http://bugs.python.org/issue6391    created  zhouer                        
                                                                               

IDLE 3.1 will not open                                           07/01/09
CLOSED http://bugs.python.org/issue6392    created  matthall                      
                                                                               

OS X: python3 from python-3.1.dmg crashes at startup             07/01/09
       http://bugs.python.org/issue6393    created  marketdickinson               
                                                                               

getppid support in os module on Windows                          07/01/09
       http://bugs.python.org/issue6394    created  janglin                       
       patch                                                                   

Add Pickle Support to the codecs Module                          07/01/09
       http://bugs.python.org/issue6395    created  ThomasH                       
                                                                               

No conversion specifier in the string, no __getitem__ method in  07/02/09
       http://bugs.python.org/issue6396    reopened amaury.forgeotdarc            
                                                                               

Implementing Solaris "poll" in the "select" module               07/01/09
       http://bugs.python.org/issue6397    created  jcea                          
                                                                               

README typo                                                      07/01/09
       http://bugs.python.org/issue6398    created  nmichaels                     
       patch                                                                   

Error reporting by logging.config.fileConfig()                   07/02/09
       http://bugs.python.org/issue6399    created  mike                          
                                                                               

incorrect commands import                                        07/02/09
CLOSED http://bugs.python.org/issue6400    created  alejolp                       
                                                                               

Unexpected module garbage collection                             07/02/09
CLOSED http://bugs.python.org/issue6401    created  j_pok                         
                                                                               

Crash after attempt to set the error indicator via PyErr_SetStri 07/02/09
CLOSED http://bugs.python.org/issue6402    created  iankko                        
                                                                               

distutils builds extension modules to root package directory     07/02/09
CLOSED http://bugs.python.org/issue6403    created  ivank                         
                                                                               

3.1 NEWS.txt points to alpha 2 file                              07/02/09
CLOSED http://bugs.python.org/issue6404    created  srid                          
                                                                               

Redundant redeclarations in descrobject.h                        07/03/09
CLOSED http://bugs.python.org/issue6405    created  flub                          
       patch                                                                   

NameError on 2to3 tool                                           07/03/09
CLOSED http://bugs.python.org/issue6406    created  alejolp                       
       patch                                                                   

multiprocessing Pool should allow custom task queue              07/03/09
       http://bugs.python.org/issue6407    created  masher                        
                                                                               

2to3: Local package import                                       07/03/09
CLOSED http://bugs.python.org/issue6408    created  alejolp                       
                                                                               

2to3 generates malformed diffs                                   07/03/09
       http://bugs.python.org/issue6409    created  exarkun                       
       patch                                                                   



Issues Now Closed (79)
______________________

Timeout in CGIXMLRPCRequestHandler under IIS                      641 days
       http://bugs.python.org/issue1214    alexandre.vassalotti          
       patch                                                                   

eliminate recursion in pickling                                   460 days
       http://bugs.python.org/issue2480    alexandre.vassalotti          
                                                                               

optparse quirks                                                   236 days
       http://bugs.python.org/issue4278    amaury.forgeotdarc            
                                                                               

3.0 file.read dreadfully slow                                     210 days
       http://bugs.python.org/issue4533    pitrou                        
       patch                                                                   

SystemError if invalid arguments passed to range() and step=-1    207 days
       http://bugs.python.org/issue4536    georg.brandl                  
       patch, needs review                                                     

directory permission error with make install in 3.0               206 days
       http://bugs.python.org/issue4601    amaury.forgeotdarc            
       patch                                                                   

print_function and unicode_literals don't work together           204 days
       http://bugs.python.org/issue4618    benjamin.peterson             
                                                                               

Issue with RotatingFileHandler logging handler on Windows          23 days
       http://bugs.python.org/issue4749    eantelman                     
                                                                               

Remove checks for win NT                                            0 days
       http://bugs.python.org/issue4856    ocean-city                    
       patch                                                                   

Multiprocessing.Queue created by sub-process fails when used in   146 days
       http://bugs.python.org/issue5155    jnoller                       
                                                                               

Formatting error in "findertools" header                          135 days
       http://bugs.python.org/issue5254    ezio.melotti                  
                                                                               

multiprocessing.process using os.close(sys.stdin.fileno) instead  131 days
       http://bugs.python.org/issue5313    jnoller                       
       patch                                                                   

multiprocessing hangs when Pool used within Process               130 days
       http://bugs.python.org/issue5331    jnoller                       
                                                                               

Allow intermixing of keyword arguments and vargarg arguments      124 days
       http://bugs.python.org/issue5383    amaury.forgeotdarc            
                                                                               

Item 'Python x.x.x' in Add/Remove Programs list still lacks an i  115 days
       http://bugs.python.org/issue5390    loewis                        
                                                                               

uuid module generates incorrect values for uuid3 (and possibly u  119 days
       http://bugs.python.org/issue5415    amaury.forgeotdarc            
                                                                               

multiprocessing/connection.py wrong pipe name under win32         104 days
       http://bugs.python.org/issue5503    jnoller                       
                                                                               

optparse: clarify option concatenation in docs                     99 days
       http://bugs.python.org/issue5555    ajs                           
                                                                               

multiprocessing.connection.Client API documentation incorrect      78 days
       http://bugs.python.org/issue5740    jnoller                       
       needs review                                                            

timeit documentation                                               57 days
       http://bugs.python.org/issue5896    ezio.melotti                  
                                                                               

Risk of confusion in multiprocessing module - daemonic processes   58 days
       http://bugs.python.org/issue5906    jnoller                       
                                                                               

re-usable generators / generator expressions should return itera   52 days
       http://bugs.python.org/issue5973    Jae                           
                                                                               

Fix O(n**2) performance problem in socket._fileobject              33 days
       http://bugs.python.org/issue6117    krisvale                      
       patch, patch, easy, needs review                                        

Confusing DeprecationWarning                                       37 days
       http://bugs.python.org/issue6119    benjamin.peterson             
                                                                               

Parallel regression testing                                        30 days
       http://bugs.python.org/issue6152    pitrou                        
       patch                                                                   

Python 3.1 rc1 will not build on OS X 10.5.7 with macports libin   30 days
       http://bugs.python.org/issue6154    ygale                         
       patch                                                                   

[AIX] Patch to correct the AIX C/C++ linker argument used for 'r   27 days
       http://bugs.python.org/issue6164    tarek                         
       patch                                                                   

Mac 'make frameworkinstall' error: [...]/Python.framework/Versio   25 days
       http://bugs.python.org/issue6170    srid                          
                                                                               

add disable_nagle_algorithm to SocketServer.TCPServer              21 days
       http://bugs.python.org/issue6192    krisvale                      
       patch, patch, easy                                                      

TCP Sockets not closed by TCPServer and StreamRequestHandler       19 days
       http://bugs.python.org/issue6248    krisvale                      
       patch, patch, needs review                                              

distutils upload command doesn't work with http proxy              13 days
       http://bugs.python.org/issue6286    tarek                         
                                                                               

Tix needs TCL package require statement                             8 days
       http://bugs.python.org/issue6309    eantelman                     
                                                                               

Add a "swap" method to list                                         8 days
       http://bugs.python.org/issue6326    pitrou                        
       patch, patch                                                            

mmap.read() crashes when passed a negative argument                 4 days
       http://bugs.python.org/issue6344    ocean-city                    
       patch                                                                   

Rstrip Incorrectly Strips Some Strings                              0 days
       http://bugs.python.org/issue6346    ezio.melotti                  
                                                                               

hpux11.00-parisc: dtoa.c: "Failed to find an exact-width 32-bit     4 days
       http://bugs.python.org/issue6347    marketdickinson               
       patch                                                                   

email.Maildir cannot roundtrip messages.                            1 days
       http://bugs.python.org/issue6349    r.david.murray                
                                                                               

Example at end of HTMLParser documentation uses old-style print     0 days
       http://bugs.python.org/issue6350    ezio.melotti                  
                                                                               

"L" integer suffix in Python 3.1 tutorial                           3 days
       http://bugs.python.org/issue6353    ezio.melotti                  
       patch                                                                   

Old floating point representation in 3.1 tutorial                   1 days
       http://bugs.python.org/issue6354    marketdickinson               
                                                                               

bogus NULL check in PyCapsule                                       0 days
       http://bugs.python.org/issue6355    benjamin.peterson             
       patch                                                                   

Segfault of sys.setrecursionlimit	if limit exceeds the value 180    0 days
       http://bugs.python.org/issue6356    loewis                        
                                                                               

tempfile.NamedTemporaryFile does not accept the delete= paramete    1 days
       http://bugs.python.org/issue6357    stutzbach                     
                                                                               

__future__ statements break doctest                                 4 days
       http://bugs.python.org/issue6363    benjamin.peterson             
                                                                               

distutils duplicates package directory for C extensions in 3.1 f    0 days
       http://bugs.python.org/issue6365    tarek                         
                                                                               

Change the instruction pointer in python                            1 days
       http://bugs.python.org/issue6367    vinoth                        
                                                                               

Fix unused variable warning introduced by commit r73603.            0 days
       http://bugs.python.org/issue6368    ocean-city                    
       patch                                                                   

binhex buggy in py3k                                                2 days
       http://bugs.python.org/issue6369    pitrou                        
       patch                                                                   

Bad performance of colllections.Counter at initialisation from a    0 days
       http://bugs.python.org/issue6370    rhettinger                    
                                                                               

Minor internal link errors in Optparse doc                          1 days
       http://bugs.python.org/issue6371    georg.brandl                  
                                                                               

Make all switches keyword-only                                      0 days
       http://bugs.python.org/issue6372    ptn                           
                                                                               

SystemError in encoder                                              0 days
       http://bugs.python.org/issue6373    amaury.forgeotdarc            
       patch                                                                   

Confused by subprocess API documentation                            1 days
       http://bugs.python.org/issue6374    georg.brandl                  
       patch                                                                   

description of Decinal.logical_invert is incorrect                  1 days
       http://bugs.python.org/issue6376    georg.brandl                  
                                                                               

Add first() function that would take first matching case from an    0 days
       http://bugs.python.org/issue6379    rhettinger                    
                                                                               

error in unicodedata.numeric(u"\u2187") and 2188                    1 days
       http://bugs.python.org/issue6383    amaury.forgeotdarc            
       patch, needs review                                                     

Permalink to built-in exception class hierarchy documentation       1 days
       http://bugs.python.org/issue6384    georg.brandl                  
                                                                               

two lines of typographical inconsistency in doc of exception cla    0 days
       http://bugs.python.org/issue6385    georg.brandl                  
                                                                               

floor division gives different result for int versus float.         0 days
       http://bugs.python.org/issue6387    r.david.murray                
                                                                               

platform.python_implementation does not work on IronPython          0 days
       http://bugs.python.org/issue6388    benjamin.peterson             
                                                                               

os.chmod() documentation refers to non-existent documentation in    2 days
       http://bugs.python.org/issue6389    r.david.murray                
       easy                                                                    

File reads past EOF in "w+b" mode                                   0 days
       http://bugs.python.org/issue6390    amaury.forgeotdarc            
                                                                               

Incorrect description of unittest.TestCase.run                      0 days
       http://bugs.python.org/issue6391    ezio.melotti                  
                                                                               

IDLE 3.1 will not open                                              0 days
       http://bugs.python.org/issue6392    amaury.forgeotdarc            
                                                                               

incorrect commands import                                           0 days
       http://bugs.python.org/issue6400    alejolp                       
                                                                               

Unexpected module garbage collection                                0 days
       http://bugs.python.org/issue6401    pitrou                        
                                                                               

Crash after attempt to set the error indicator via PyErr_SetStri    0 days
       http://bugs.python.org/issue6402    benjamin.peterson             
                                                                               

distutils builds extension modules to root package directory        1 days
       http://bugs.python.org/issue6403    tarek                         
                                                                               

3.1 NEWS.txt points to alpha 2 file                                 0 days
       http://bugs.python.org/issue6404    benjamin.peterson             
                                                                               

Redundant redeclarations in descrobject.h                           0 days
       http://bugs.python.org/issue6405    benjamin.peterson             
       patch                                                                   

NameError on 2to3 tool                                              0 days
       http://bugs.python.org/issue6406    alejolp                       
       patch                                                                   

2to3: Local package import                                          0 days
       http://bugs.python.org/issue6408    alejolp                       
                                                                               

Tk.quit leads to crash in python.exe                             2169 days
       http://bugs.python.org/issue775544  gpolo                         
                                                                               

fcntl.ioctl have a bit problem.                                  1413 days
       http://bugs.python.org/issue1262856 amaury.forgeotdarc            
                                                                               

traceback module can return undecodable byte strings             1392 days
       http://bugs.python.org/issue1284496 amaury.forgeotdarc            
                                                                               

Trailing slash redirection for SimpleHTTPServer                  1364 days
       http://bugs.python.org/issue1314572 josiahcarlson                 
       patch, easy                                                             

Polymorphic getters / setters                                    1285 days
       http://bugs.python.org/issue1388872 amaury.forgeotdarc            
                                                                               

Stepping into a generator throw does not work                     962 days
       http://bugs.python.org/issue1592241 amaury.forgeotdarc            
                                                                               

pickle - cannot unpickle circular deps with custom __hash__       704 days
       http://bugs.python.org/issue1761028 alexandre.vassalotti          
                                                                               



Top Issues Most Discussed (10)
______________________________

  7 getppid support in os module on Windows                            2 days
open    http://bugs.python.org/issue6394   

  7 error in unicodedata.numeric(u"\u2187") and 2188                   1 days
closed  http://bugs.python.org/issue6383   

  7 Green-box doc glitch: winhelp version only                       126 days
open    http://bugs.python.org/issue5388   

  7 Remove checks for win NT                                           0 days
closed  http://bugs.python.org/issue4856   

  6 incorrect commands import                                          0 days
closed  http://bugs.python.org/issue6400   

  6 tempfile.NamedTemporaryFile does not accept the delete= paramet    1 days
closed  http://bugs.python.org/issue6357   

  6 Cumulative patcc:h to http and xmlrpc                             22 days
open    http://bugs.python.org/issue6267   

  5 floor division gives different result for int versus float.        0 days
closed  http://bugs.python.org/issue6387   

  5 SystemError in encoder                                             0 days
closed  http://bugs.python.org/issue6373   

  5 Make all switches keyword-only                                     0 days
closed  http://bugs.python.org/issue6372   




From p.f.moore at gmail.com  Fri Jul  3 18:45:07 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 3 Jul 2009 17:45:07 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<4A4E1572.3000801@gmail.com>
	<94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com>
Message-ID: <79990c6b0907030945u1ed6e358o7847a08e39fa0c58@mail.gmail.com>

2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
> On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan<ncoghlan at gmail.com> wrote:
>> I will note (and I believe this is also the main point that Lukasz was
>> making) that having the distribution metadata outside the distribution
>> as currently proposed in PEP 376 is going to make any eventual PEP 302
>> integration much harder - 302 importers only provide a mechanism for
>> accessing files inside the distribution, not "adjacent" to them, so the
>> mechanism in the PEP doesn't generalise properly.
>>
>> I suspect this limitation of the PEP 302 APIs is the origin of the
>> setuptools format that embeds the metadata inside the distribution - it
>> lets you get at the metadata without having to assume that it exists
>> directly on the filesystem anywhere.
>
> Maybe we could rework the pkgutil classes for PEP 376 so they look
> like an implementation
> of the PEP 302 protocol for directories and zip files, with an extra
> "get_metadata()" API
> and state that it could be an extension for PEP 302 later.

That sounds like an excellent approach And given that PEP 302 is
nothing more than an API spec, it's nice and easy to extend PEP 302 to
say that importers can/should implement this (after all, the only 2
"real" cases in the wild are done, via PEP 376).

Warning: I've not thought all this through fully, so I may be missing
some subtleties! The ultimate spec needs to be clearly worded, and as
a consequence of this discussion it occurs to me that the current PEP
376 is very far from clear in defining exactly what the naming, layout
and location of egg-info directories is - it makes a lot of
assumptions that may break for more general path importers

I'll do some thinking, and maybe come up with some examples of PEP 302
edge cases, so that we can at least be sure that they've not been
ignored (rejected as to stupid to care about, on the other hand, I'm
happy with :-))

Paul.

From pje at telecommunity.com  Fri Jul  3 19:02:40 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 03 Jul 2009 13:02:40 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.co
 m>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
Message-ID: <20090703165942.38CFA3A4109@sparrow.telecommunity.com>

At 02:54 PM 7/3/2009 +0100, Paul Moore wrote:
>Eggs are fundamentally a PEP 302 zip file format. There are some 
>extra bits of metadata for setuptools/easy_install in there (as I 
>understand things) but essentially they are zip files. When you say 
>"decoupling the egg format", I assume you mean "decoupling the egg 
>metadata" - which is fine, but to properly decouple, you need API 
>level access to the metadata. PEP 376 offers read-only access, but 
>as you rightly point out, it is only for filesystem data (and some 
>form of zip file, which appears to be limited in some way, as it 
>isn't PEP 302 based, and the actual format isn't defined anywhere). 
>The basic point here is that PEP 376 needs to define precisely how 
>pkgutil.get_distributions() scans sys.path looking for ".egg-info 
>directories". What does it do for sys.path entries that don't 
>correspond to filesystem directories? (Note - these may or may not 
>be zip files. Even if they are zip files, an earlier entry on 
>sys.path_hooks could have taken precedence. At the very least, you 
>should only process path entries as zip files if their importer - in 
>sys.path_importer_cache or via an explicit path hook scan - is a 
>zipimporter object.). To be honest, this is a major can of worms. 
>But if PEP 376 is not going to support PEP 302, then it must state 
>that fact explicitly, to avoid giving people false expectations - 
>particularly with Brett's importlib in Python 3.1, which will make 
>it far easier for people to experiment with new packaging formats 
>such as the ones Lukasz mentions above. And it MUST fail gracefully 
>in the face of unsupported importer types.

Well, we could always resurrect PEP 365, since pkg_resources already 
has documented extensible support for arbitrary importers.  That 
solves backward *and* forward compatibility.  Then PEP 376's 
uninstall facilities could be implemented using pkg_resources' 
existing metadata query features.

The primary downside to that, of course, is that it brings in the 
matter of version specifications and dependencies...  which appear to 
be a contentious topic.  (Note that Tarek is proposing to drop the 
PEP 386 proposal to standardize a much more restrictive scheme than 
seutptools' version parser, precisely because of the controversy.)


From p.f.moore at gmail.com  Fri Jul  3 19:01:05 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 3 Jul 2009 18:01:05 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4E1572.3000801@gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<4A4E1572.3000801@gmail.com>
Message-ID: <79990c6b0907031001h39caea63j98f500afdcc05fbf@mail.gmail.com>

2009/7/3 Nick Coghlan <ncoghlan at gmail.com>:
> I agree that even if the reference implementation of PEP 376 only
> handles normal directories and zipfiles, the PEP itself needs to explain
> how the developer of a custom PEP 302 importer or loader can hook into
> the mechanism in order to provide metadata that distutils can
> understand. Or, as you suggest, state explicitly that the proposal at
> this stage is specifically limited to filesystem and zipfile packages
> and defer extension to arbitrary PEP 302 importers and loaders to a
> later PEP.
>
> To be honest, I'd personally be OK with the latter strategy - while
> other PEP 302 loaders and importers do exist (as Lukasz'z post shows),
> it would be unfortunate to unduly hold up improved installation metadata
> for the vast majority of typical use cases while we try to figure out
> ways to support the more esoteric use cases. Supporting both would
> obviously be better, but given the choice between the status quo and
> partial support, I believe this is a case where the partial support
> would still be worthwhile.

You put it more clearly than I did. That's basically what I think,
with the one proviso that we should make sure that PEP 376 doesn't
specify something that out and out breaks the more esoteric PEP 302
cases.

When Just and I were developing PEP 302, we found that the best way to
do that was to leave anything that didn't need to be specified, as
unspecified (hence the fact that there's so little defined in PEP
302!). It's easier to add things later than to remove or change them.
That's why I was recommending to Tarek that he take out of the PEP any
details about classes or APIs that couldn't be directly obtained from
the core API. The same applies here (just talking in terms of a
duck-typed notional Distribution class, allows people to implement
their own URLDistribution or SqliteDistribution, or whatever, and not
have to emulate any more of a filesystem than necessary).

> I will note (and I believe this is also the main point that Lukasz was
> making) that having the distribution metadata outside the distribution
> as currently proposed in PEP 376 is going to make any eventual PEP 302
> integration much harder - 302 importers only provide a mechanism for
> accessing files inside the distribution, not "adjacent" to them, so the
> mechanism in the PEP doesn't generalise properly.
>
> I suspect this limitation of the PEP 302 APIs is the origin of the
> setuptools format that embeds the metadata inside the distribution - it
> lets you get at the metadata without having to assume that it exists
> directly on the filesystem anywhere.

Again, agreed. But remember that PEP 302 is driven by looking up a
module or package name. PEP 376 is looking up a *distribution* name.
The docutils example in the PEP shows this:

- docutils/
- roman.py
- docutils-0.5-py2.6.egg-info/

There are 3 things here: a package (docutils), a module (roman) and a
distribution (also named docutils, but could be called George for all
it matters). So none of the PEP 302 lookup mechanisms (which work on
package/module name) apply.

The need for a separate concept (a "distribution") is unfortunate, as
it adds complexity, but there are enough real life cases that make it
clear that it's necessary.

Hmm, I suspect that the implication here is that PEP 302 could do with
an overhaul, to extend it to encompass the concept of a distribution.
I'd be willing to have a look at that.

Paul.

From p.f.moore at gmail.com  Fri Jul  3 19:06:55 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 3 Jul 2009 18:06:55 +0100
Subject: [Python-Dev] PEP 376
In-Reply-To: <20090703165942.38CFA3A4109@sparrow.telecommunity.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<20090703165942.38CFA3A4109@sparrow.telecommunity.com>
Message-ID: <79990c6b0907031006rd0c6265hf27a3f1488ba3af2@mail.gmail.com>

2009/7/3 P.J. Eby <pje at telecommunity.com>:
> Well, we could always resurrect PEP 365, since pkg_resources already has
> documented extensible support for arbitrary importers. ?That solves backward
> *and* forward compatibility. ?Then PEP 376's uninstall facilities could be
> implemented using pkg_resources' existing metadata query features.

It would be easier to comment on that suggestion if the PEP documented
the reasons why it was rejected. Sadly, that never got added to the
PEP at the time. Do you recall the reasons, and if so would you be
willing to update the PEP with them?

Paul.

From pje at telecommunity.com  Fri Jul  3 19:10:48 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 03 Jul 2009 13:10:48 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4E1572.3000801@gmail.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<4A4E1572.3000801@gmail.com>
Message-ID: <20090703170748.818B73A4109@sparrow.telecommunity.com>

At 12:28 AM 7/4/2009 +1000, Nick Coghlan wrote:
>I suspect this limitation of the PEP 302 APIs is the origin of the
>setuptools format that embeds the metadata inside the distribution - it
>lets you get at the metadata without having to assume that it exists
>directly on the filesystem anywhere.

I think you have this backwards; it's setuptools that doesn't care 
where (or whether) the metadata exists on the file system; it 
delegates metadata operations to a "metadata provider" that's usually 
an adapter over a PEP 302 "loader".

See 
http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-importers 
for the API details of how to register support for arbitrary PEP 302 
importers and loaders.  (Which presumably, Lukasz is using.  I didn't 
know that anybody was actually using it, but it's nice to know that 
the documentation is apparently sufficient for *some* people.  ;-) )


From brett at python.org  Fri Jul  3 20:04:16 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Jul 2009 11:04:16 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>

On Thu, Jul 2, 2009 at 13:42, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:

> In response to some rumblings on python-committers and just to request
> more feedback, a progress report. I know it's long, I've tried to put
> to keep it concise and chunked, though.
>
> - First of all, I've got the basic conversion down, I've done it a few
> times now, with progressively better results. You can view some
> results at http://hg.python.org/, which has a preliminary cpython
> repository. *** The changeset hashes for that repo will change, so you
> won't be able to commit or pull from it in the future.***
>
> - Second of all, some planning. I've thought about it a bit, and I
> think we should aim for going live with hg on August 1. Given that I'm
> on vacation from 8-18 July (and I'm not sure whether I'll be able to
> actually work on it during that time, though I imagine I'll be able to
> spend some time on it at least), that's quite ambitious, so I'm going
> to say it's okay if it slips by a few days. Putting a deadline out
> there is a good thing, anyway.
>

Fine by me as long as people realize that if anything is questionable then
the switch will not happen. Getting this right takes precedence over any
deadline. And obviously we will need to do at least one live conversion on
python.org hardware to make sure everything will work smoothly.


>
> - Third of all, to make this possible, it would be helpful if I got
> more feedback on the PEP. Last time I raised it, there was virtually
> nothing. This time, I'll include it inline so there's hopefully less
> of a barrier to reviewing it.
>
> - Fourth, Mercurial 1.3 was just released! Bet you didn't see that
> coming. It's looking like a pretty good release, with an experimental
> version of the much-coveted subrepository support (like
> svn:externals). This also means that the latest version of
> hgsubversion, the tool I used for the conversion, will be more
> accessible for converting other projects. You know you want to!
>

And will make the idea of splitting out the standard library and tests a
reasonable thing to do.


>
> - Fifth, here's a list of things, off the top of my head, that still need
> doing:
>
> * Get agreement on branch strategy and branch processing (list of
> branches + proposed handling at
> http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
> REVIEW
> * Get agreement on tag processing (first come up with a list)
> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs
>
> - Sixth (this is the good part), less obvious things that have been
> done or don't need doing:
>
> * .hgignore generation (I've been convinced it's too hard, the current
> version will do)


Yeah, we can do this manually.


>
> * revlog reordering (it's painless and a big win)
>
> I'll get through all of these myself, but obviously any help would be
> welcome. For any hg users, writing docs should be an easy start. For
> others, please review the PEP (below), the branch map in
> http://hg.python.org/pymigr/file/tip/all-branches.txt and the author
> map at http://hg.python.org/pymigr/file/tip/author-map (not that much
> has changed since the start, so if you've looked at it already, feel
> free to skip this part). Right now I'm a little stuck on branch
> processing, because it's a long running script that needs a bunch of
> debugging, but I'll get going on that again.
>
> I think that's all I can think of for now, I'll update the PEP with
> new bits soon. Here it is, ready for your review:
>
> ==============================================================
>
> Motivation
>
> After having decided to switch to the Mercurial DVCS, the actual
> migration still has to be performed. In the case of an important piece
> of infrastructure like the version control system for a large,
> distributed project like Python, this is a significant effort. This
> PEP is an attempt to describe the steps that must be taken for further
> discussion. It's somewhat similar to PEP 347 [1], which discussed the
> migration to SVN.
>
> To make the most of hg, I (Dirkjan) would like to make a high-fidelity
> conversion, such that (a) as much of the svn metadata as possible is
> retained, and (b) all metadata is converted to formats that are common
> in Mercurial. This way, tools written for Mercurial can be optimally
> used. In order to do this, I want to use the hgsubversion [2] software
> to do an initial conversion. This hg extension is focused on providing
> high-quality conversion from Subversion to Mercurial for use in
> two-way correspondence, meaning it doesn't throw away as much
> available metadata as other solutions.
>
> Such a conversion also seems like a good time to reconsider the
> contents of the repository and determine if some things are still
> valuable. In this spirit, the following sections also propose
> discarding some of the older metadata.
> Timeline
>
> TBD; needs fully working hgsubversion and consensus on this document.
> Transition plan
> Branch strategy
>
> Mercurial has two basic ways of using branches: cloned branches, where
> each branch is kept in a separate repository, and named branches,
> where each revision keeps metadata to note on which branch it belongs.
> The former makes it easier to distinguish branches, at the expense of
> requiring more disk space on the client. The latter makes it a little
> easier to switch between branches, but often has somewhat unintuitive
> results for people (though this has been getting better in recent
> versions of Mercurial).
>
> I'm still a bit on the fence about whether Python should adopt cloned
> branches and named branches. Since it usually makes more sense to tag
> releases on the maintenance branch, for example, mainline history
> would not contain release tags if we used cloned branches. Also,
> Mercurial 1.2 and 1.3 have the necessary tools to make named branches
> less painful (because they can be properly closed and closed heads are
> no longer considered in relevant cases).
>
> A disadvantage might be that the used clones will be a good bit larger
> (since they essentially contain all other branches as well). This can
> me mitigated by keeping non-release (feature) branches in separate
> clones. Also note that it's still possible to clone a single named
> branch from a combined clone, by specifying the branch as in hg clone
> http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a
> separate clone problably also makes sense.
>

While I really like the idea of using named branches for each release so
that there is a single py3k branch that contains all relevant history for
every release, I think we should start simple and go with cloned branches.
That way the workflow does not radically shift from what we do now for svn
to start. Once the conversion is done and people are comfortable with hg we
can then discuss moving towards a named branch approach.


>
> XXX To do: size comparison for selected separation scenarios.
> Converting branches
>
> There are quite a lot of branches in SVN's branches directory. I
> propose to clean this up a bit, by employing the following the
> strategy:
>
>    * Keep all release (maintenance) branches
>    * Discard branches that haven't been touched in 18 months, unless
> somone indicates there's still interest in such a branch
>    * Keep branches that have been touched in the last 18 months,
> unless someone indicates the branch can be deprecated
>

Sounds reasonable to me. We can just make a list and send it to
python-committers to make final decisions of what should stick around.


>
> Converting tags
>
> The SVN tags directory contains a lot of old stuff. Some of these are
> not, in fact, full tags, but contain only a smaller subset of the
> repository. I think we should keep all release tags, and consider
> other tags for inclusion based on requests from the developer
> community. I'd like to consider unifying the release tag naming scheme
> to make some things more consistent, if people feel that won't create
> too many problems. For example, Mercurial itself just uses '1.2.1' as
> a tag, where CPython would currently use r121.


I don't use tags so I don't really care, but in the name of easy transition
I say we don't change the naming scheme (although I have no issue dropping
obscure tags).


>
> Author map
>
> In order to provide user names the way they are common in hg (in the
> 'First Last <user at example.org>' format), we need an author map to map
> cvs and svn user names to real names and their email addresses. I have
> a complete version of such a map in my migration tools repository [3].
> The email addresses in it might be out of date; that's bound to
> happen, although it would be nice to try and have as many people as
> possible review it for addresses that are out of date. The current
> version also still seems to contain some encoding problems.


Something else that can go out to python-committers before the switch.


>
> Generating .hgignore
>
> The .hgignore file can be used in Mercurial repositories to help
> ignore files that are not eligible for version control. It does this
> by employing several possible forms of pattern matching. The current
> Python repository already includes a rudimentary .hgignore file to
> help with using the hg mirrors.
>
> It might be useful to have the .hgignore be generated automatically
> from svn:ignore properties. This would make sure all historic
> revisions also have useful ignore information (though one could argue
> ignoring isn't really relevant to just checking out an old revision).


Don't bother with anything automatic. We can change the .hgignore file by
hand. We all know glob and regex syntax. =)


>
> Revlog reordering
>
> As an optional optimization technique, we should consider trying a
> reordering pass on the revlogs (internal Mercurial files) resulting
> from the conversion. In some cases this results in dramatic decreases
> in on-disk repository size.


Fine by me.


>
> Other repositories
>
> Richard Tew has indicated that he'd like the Stackless repository to
> also be converted. What other projects in the svn.python.org
> repository should be converted? Do we want to convert the peps
> repository? distutils? others?


I don't think there is a single project we host -- all two of them -- that
have not said they want to convert. So I say convert everything and let's
turn off the svn server by the end of the year.


>
> Infrastructure
> hg-ssh
>
> Developers should access the repositories through ssh, similar to the
> current setup. Public keys can be used to grant people access to a
> shared hg@ account. A hgwebdir instance should also be set up for easy
> browsing and read-only access. If we're using ssh, developers should
> trivially be able to start new clones (for longer-term features that
> profit from a separate branch).
> Hooks
>
> A number of hooks is currently in use. The hg equivalents for these
> should be developed and deployed. The following hooks are being used:
>
>    * check whitespace: a hook to reject commits in case the
> whitespace doesn't match the rules for the Python codebase. Should be
> straightforward to re-implement from the current version. We can also
> offer a whitespace hook for use with client-side repositories that
> people can use; it could either warn about whitespace issues and/or
> truncate trailing whitespace from changed lines. Open issue: do we
> check only the tip after each push, or do we check every commit in a
> changegroup?
>    * commit mails: we can leverage the notify extension for this
>    * buildbots: both the regular and the community build masters must
> be notified. Fortunately buildbot includes support for hg. I've also
> implemented this for Mercurial itself, so I don't expect problems
> here.
>    * check contributors: in the current setup, all changesets bear
> the username of committers, who must have signed the contributor
> agreement. In a DVCS, the committers are not necessarily the same
> people who push, and so we can't check if the committer is a
> contributor. We could use a hook to check if the committer is a
> contributor if we keep a list of registered contributors.


Can we check these scripts into the repository itself? That way there is a
chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for
``make patchcheck``.


>
>
> hgwebdir
>
> A more or less stock hgwebdir installation should be set up. We might
> want to come up with a style to match the Python website. It may also
> be useful to build a quick extension to augment the URL rev parser so
> that it can also take r[0-9]+ args and come up with the matching hg
> revision.
> After migration
> Where to get code
>
> It needs to be decided where the hg repositories will live. I'd like
> to propose to keep the hgwebdir instance at hg.python.org. This is an
> accepted standard for many organizations, and an easy parallel to
> svn.python.org. The 2.7 (trunk) repo might live at
> http://hg.python.org/main/, for example, with py3k at
> http://hg.python.org/py3k/. For write access, developers will have to
> use ssh, which could be ssh://hg at hg.python.org/main/. A demo
> installation will be set up with a preliminary conversion so people
> can experiment and review; it can live at
> http://hg.python.org/example/.
>
> code.python.org was also proposed as the hostname. Personally, I think
> that using the VCS name in the hostname is good because it prevents
> confusion: it should be clear that you can't use svn or bzr for
> hg.python.org.
>

How about hg.python.org for the official branches and we keep
code.python.org for personal branches of the developers like we have done
with the bzr experiments?


>
> hgwebdir can already provide tarballs for every changeset. I think
> this obviates the need for daily snapshots; we can just point users to
> tip.tar.gz instead, meaning they will get the latest. If desired, we
> could even use buildbot results to point to the last good changeset.


I like the stable buildbot tarball idea.


>
> Python-specific documentation
>
> hg comes with good built-in documentation (available through hg help)
> and a wiki [4] that's full of useful information and recipes. In
> addition to that, the parts of the developer FAQ [5] concerning
> version control will gain a section on using hg for Python
> development. Some of the text will be dependent on the outcome of
> debate about this PEP (for example, the branching strategy).
>
> Think first, commit later?
>
> In recent history, old versions of Python have been maintained by a
> select group of people backporting patches from trunk to release
> branches. While this may not scale so well as the development pace
> grows, it also runs into some problems with the current crop of
> distributed versioning tools. These tools (I believe similar problems
> would exist for either git, bzr, or hg, though some may cope better
> than others) are based on the idea of a Directed Acyclic Graph (or
> DAG), meaning they keep track of relations of changesets.
>
> Mercurial itself has a stable branch which is a ''strict'' subset of
> the unstable branch. This means that generally all fixes for the
> stable branch get committed against the tip of the stable branch, then
> they get merged into the unstable branch (which already contains the
> parent of the new cset). This provides a largely frictionless
> environment for moving changes from stable to unstable branches.
> Mistakes, where a change that should go on stable goes on unstable
> first, do happen, but they're usually easy to fix. That can be done by
> copying the change over to the stable branch, then trivial-merging
> with unstable -- meaning the merge in fact ignores the parent from the
> stable branch).
>
> This strategy means a little more work for regular committers, because
> they have to think about whether their change should go on stable or
> unstable; they may even have to ask someone else (the RM) before
> committing. But it also relieves a dedicated group of committers of
> regular backporting duty, in addition to making it easier to work with
> the tool.
>
> Now would be a good time to consider changing strategies in this
> regard, although it would be relatively easy to switch to such a model
> later on.


As I have said, we should change our workflow habits after the switch and
people are comfortable with hg.


>
> The future of Subversion
>
> What happens to the Subversion repositories after the migration? Since
> the svn server contains a bunch of repositories, not just the CPython
> one, it will probably live on for a bit as not every project may want
> to migrate or it takes longer for other projects to migrate. To
> prevent people from staying behind, we may want to remove migrated
> projects from the repository.
> Build identification
>
> Python currently provides the sys.subversion tuple to allow Python
> code to find out exactly what version of Python it's running against.
> The current version looks something like this:
>
>    * ('CPython', 'tags/r262', '71600')
>    * ('CPython', 'trunk', '73128M')
>
> Another value is returned from Py_GetBuildInfo() in the C API, and
> available to Python code as part of sys.version:
>
>    * 'r262:71600, Jun 2 2009, 09:58:33'
>    * 'trunk:73128M, Jun 2 2009, 01:24:14'
>
> I propose that the revision identifier will be the short version of
> hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
> (instead of 'M') if the working directory from which it was built was
> modified. This mirrors the output of the hg id command, which is
> intended for this kind of usage.
>
> For the tag/branch identifier, I propose that hg will check for tags
> on the currently checked out revision, use the tag if there is one
> ('tip' doesn't count), and uses the branch name otherwise.
> sys.subversion becomes
>
>    * ('CPython', '2.6.2', 'dd3ebf81af43')
>    * ('CPython', 'default', 'af694c6a888c+')
>
> and the build info string becomes
>
>    * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33'
>    * 'default:af694c6a888c+, Jun 2 2009, 01:24:14'
>
> This reflects that the default branch in hg is called 'default'
> instead of Subversion's 'trunk', and reflects the proposed new tag
> format.


Should we consider adding a sys.revision attribute and begin the deprecation
of sys.subversion?


>
> References
> [1]     http://www.python.org/dev/peps/pep-0347/
> [2]     http://bitbucket.org/durin42/hgsubversion/
> [3]     http://hg.xavamedia.nl/cpython/pymigr/
> [4]     http://www.selenic.com/mercurial/wiki/
> [5]     http://www.python.org/dev/faq/#version-control
>
> =====================================================
>
> Cheers,
>
> Dirkjan
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/90bca0ab/attachment-0001.htm>

From tjreedy at udel.edu  Fri Jul  3 20:22:57 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 03 Jul 2009 14:22:57 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <h2lia7$v1j$1@ger.gmane.org>

Dirkjan Ochtman wrote:

> It needs to be decided where the hg repositories will live. I'd like
> to propose to keep the hgwebdir instance at hg.python.org. This is an
> accepted standard for many organizations, and an easy parallel to
> svn.python.org. The 2.7 (trunk) repo might live at
> http://hg.python.org/main/, for example, with py3k at
> http://hg.python.org/py3k/.

I would very much like the 'k' dropped from the py3 name. It was a funny 
joke when py3 was vaporware, now it is excess baggage which only puzzles 
non-insiders and newcomers.

I think the two repos should be either symmetrically named

hg.python.org/py2
hg.python.org/py3

If one must be designated 'main', it should be py3.

Continuing to call py2 'main' will continue to discourage use of py3.

Terry Jan Reedy


From python at mrabarnett.plus.com  Fri Jul  3 20:41:00 2009
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 03 Jul 2009 19:41:00 +0100
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2lia7$v1j$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2lia7$v1j$1@ger.gmane.org>
Message-ID: <4A4E50BC.8060201@mrabarnett.plus.com>

Terry Reedy wrote:
> Dirkjan Ochtman wrote:
> 
>> It needs to be decided where the hg repositories will live. I'd like
>> to propose to keep the hgwebdir instance at hg.python.org. This is an
>> accepted standard for many organizations, and an easy parallel to
>> svn.python.org. The 2.7 (trunk) repo might live at
>> http://hg.python.org/main/, for example, with py3k at
>> http://hg.python.org/py3k/.
> 
> I would very much like the 'k' dropped from the py3 name. It was a funny 
> joke when py3 was vaporware, now it is excess baggage which only puzzles 
> non-insiders and newcomers.
> 
> I think the two repos should be either symmetrically named
> 
> hg.python.org/py2
> hg.python.org/py3
> 
> If one must be designated 'main', it should be py3.
> 
> Continuing to call py2 'main' will continue to discourage use of py3.
> 
We could regard py3k as the phase from the original concept of Python 3
to its 'prototype', Python 3.0. Python 3.1 would be the first
'real/usable' version.

From aahz at pythoncraft.com  Fri Jul  3 21:06:53 2009
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 3 Jul 2009 12:06:53 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
Message-ID: <20090703190653.GA7213@panix.com>

On Fri, Jul 03, 2009, Brett Cannon wrote:
>
> Should we consider adding a sys.revision attribute and begin the deprecation
> of sys.subversion?

+1
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha

From p.f.moore at gmail.com  Fri Jul  3 21:29:57 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 3 Jul 2009 20:29:57 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide
	distribution metadata
Message-ID: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>

This post basically follows on from the previous thread about PEP 376
(Changing the .egg-info structure) where it was pointed out that the
PEP doesn't cover PEP 302 import hooks (except in the explicit special
case of zip files). This is likely to be a fairly long posting, but
I'd like to try and cover the subtleties a little, so that people can
comment without having to refer back to too many documents.

Comments are most definitely welcome! While I know PEP 302 reasonably
well, and the zip importer implementation, I'm not an expert on the
egg-info structure, or the practical drivers behind it, so if I miss
key issues because of a too-theoretical approach, I'd be grateful for
corrections.

The basic structure of PEP 302 imports is as follows:

  - scan sys.meta_path checking each finder in turn
  - if nothing found, scan sys.path and for each entry pass it to each
element of sys.path_hooks, to get a finder to check
  - do the default filesystem processing on path items for which no
path hook applies

The first finder (if any) to recognise the module name wins (and
returns a loader responsible for creating the actual module object).

So it's the finders that are responsible for scanning the "filesystem"
(more accurately, the namespace that finder manages).

PEP 302 is entirely couched in terms of modules, packages and module
names. There is no concept of a "distribution" in PEP 376 terms. This
is entirely understandable, as imports don't know about distributions
(see the docutils example in PEP 376 - the docutils *distribution*
contains the *module* roman, but there's nothing about importing roman
that requires you to know that it came from the docutils
distribution).

So we need to introduce a new concept, that of a distribution, into
PEP 302 loaders. And the place it should be located is in the finder
(which handles the "filesystem" aspects of the protocol. So we have
the methods:

Finder.list_distributions()
Returns a list of all distribution names in the "filesystem" managed
by the finder (usually one zip file, path directory, salite database,
or whatever)

Finder.get_metadata(distribution_name)
Returns a metadata object for the given distribution (this is the PEP
376 Distribution object). One note here - get_egginfo_file should be
specified as returning a *file-like object* rather than a file
instance. Another note - get_egginfo_file and get_egginfo_files could
just as easily be named get_metadata_file and get_metadata_files -
just as meaningful, and less tied to the egg format (as well as
avoiding the "egg" name, which I personally dislike :-))

Finder.uninstall(distribution_name, filter=callable, installer=name)
Uninstall the given distribution. It's likely that many finders will
be read-only. In that case, this function should return None.
Otherwise, return a list of the "files" removed. (This may need some
clarification, as many finders won't have a concept of a "file name").

I don't think anything else is needed to support PEP 376.

The prototype implementation of PEP 376 could be extended to work with
finders (doing the relevant sys.meta_path and sys.path_hooks
searches). For the final implementation, the special casing of zip
files would be replaced by an implementation of the extended finder
protocol in the zipimporter object.

There's almost certainly aspects missing from the above proposal. But
it does have some definite advantages, above and beyond simply
allowing PEP 302 importers to participate in the PEP 376 protocol.
Setuptools-style eggs could be handled simply by creating a
specialised finder (IIUC, they currently just use the standard
zipimporter - the specialised version could subclass this) to override
the metadata methods so as to cater for their specialised egg-info
format. Other formats could be handled similarly.

Does this sound sensible? Tarek, would you be OK with me attempting to
modify your prototype to support this protocol? Are there any tests
for PEP 376, so that I can confirm I haven't completely broken
something? If I can, I'll knock up some simple prototype importers for
non-standard examples, and see how they work with all this.

Paul.

From ziade.tarek at gmail.com  Fri Jul  3 21:46:24 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 3 Jul 2009 21:46:24 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
Message-ID: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>

2009/7/3 Paul Moore <p.f.moore at gmail.com>:
> Does this sound sensible? Tarek, would you be OK with me attempting to
> modify your prototype to support this protocol? Are there any tests
> for PEP 376, so that I can confirm I haven't completely broken
> something? If I can, I'll knock up some simple prototype importers for
> non-standard examples, and see how they work with all this.

Yes that's exactly what I was thinking about after the discussion we
had in the other
thread. This change would allow much more flexibility.

Pleas go ahead, it's located here :

http://bitbucket.org/tarek/pep376/

You can give me a bitbucket account so I can give you write access to the repo,
There are tests as long as you install Nose.

Tarek

From ziade.tarek at gmail.com  Fri Jul  3 22:14:27 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 3 Jul 2009 22:14:27 +0200
Subject: [Python-Dev] PEP 376
In-Reply-To: <20090703165942.38CFA3A4109@sparrow.telecommunity.com>
References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com>
	<A1CA249E-D1AF-4963-9BF3-60DC1BA7A9BE@stxnext.pl>
	<79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com>
	<20090703165942.38CFA3A4109@sparrow.telecommunity.com>
Message-ID: <94bdd2610907031314p64df2581wae746bc595ef86f3@mail.gmail.com>

On Fri, Jul 3, 2009 at 7:02 PM, P.J. Eby<pje at telecommunity.com> wrote:
> At 02:54 PM 7/3/2009 +0100, Paul Moore wrote:
>>
>> Eggs are fundamentally a PEP 302 zip file format. There are some extra
>> bits of metadata for setuptools/easy_install in there (as I understand
>> things) but essentially they are zip files. When you say "decoupling the egg
>> format", I assume you mean "decoupling the egg metadata" - which is fine,
>> but to properly decouple, you need API level access to the metadata. PEP 376
>> offers read-only access, but as you rightly point out, it is only for
>> filesystem data (and some form of zip file, which appears to be limited in
>> some way, as it isn't PEP 302 based, and the actual format isn't defined
>> anywhere). The basic point here is that PEP 376 needs to define precisely
>> how pkgutil.get_distributions() scans sys.path looking for ".egg-info
>> directories". What does it do for sys.path entries that don't correspond to
>> filesystem directories? (Note - these may or may not be zip files. Even if
>> they are zip files, an earlier entry on sys.path_hooks could have taken
>> precedence. At the very least, you should only process path entries as zip
>> files if their importer - in sys.path_importer_cache or via an explicit path
>> hook scan - is a zipimporter object.). To be honest, this is a major can of
>> worms. But if PEP 376 is not going to support PEP 302, then it must state
>> that fact explicitly, to avoid giving people false expectations -
>> particularly with Brett's importlib in Python 3.1, which will make it far
>> easier for people to experiment with new packaging formats such as the ones
>> Lukasz mentions above. And it MUST fail gracefully in the face of
>> unsupported importer types.
>
> Well, we could always resurrect PEP 365, since pkg_resources already has
> documented extensible support for arbitrary importers. ?That solves backward
> *and* forward compatibility. ?Then PEP 376's uninstall facilities could be
> implemented using pkg_resources' existing metadata query features.
>
> The primary downside to that, of course, is that it brings in the matter of
> version specifications and dependencies... ?which appear to be a contentious
> topic. ?(Note that Tarek is proposing to drop the PEP 386 proposal to
> standardize a much more restrictive scheme than seutptools' version parser,
> precisely because of the controversy.)

I don't think we can add pkg_resources as-is because it does much more than
providing the query features, which is beyond distutils scope.

I think that part of the PEP 376 code at the end will be very similar
to the subpart of pkg_resources
that has the query features (as it is now with the current prototype)

And what is nice with Paul's proposal on changing PEP 302 is that
setuptools will
be able to propose its EggFormats on the top of the pkgutils/distutils
changes and make its
pkg_resource code lighter.

From martin at v.loewis.de  Fri Jul  3 22:36:26 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 22:36:26 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E0825.5020407@skippinet.com.au>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au>
Message-ID: <4A4E6BCA.2090103@v.loewis.de>

> So we must work without effective EOL support?

If that's the case (i.e. no effective EOL support, the way svn
supported it), then I think the PEP should make that clear (e.g.
in a discussion section).

For the server-side hooks: it would be good to know exactly
what they check, wrt. line endings.

To find out what files should not be stored with LF line endings,
do "svn pg -R svn:eol-style .|grep CRLF".

For win32text, it would probably be good if the FAQ would provide
the relevant configuration instructions; it would be really helpful
if somebody familiar with Windows and hg could provide detailed
instructions well in advance of August 1.

If we don't have anybody familiar with Windows and hg, we have a
really serious problem.

Regards,
Martin

From martin at v.loewis.de  Fri Jul  3 22:38:32 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 22:38:32 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>	<4A4E0825.5020407@skippinet.com.au>
	<ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
Message-ID: <4A4E6C48.7070408@v.loewis.de>

>> This is exactly why I was asking for your advice - I can't work out how to
>> work effectively with win32text as it stands myself, so remain stuck
>> accidently checking in files with inappropriate line endings and stuck
>> working out how to move pywin32's CVS repo with abandoning the very
>> primitive EOL safety it offers...
> 
> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

If "both" means "both the server side test, and win32text", then Mark
already gave the answer: he cannot use win32test because he does not
know how to operate it (and doesn't bother studying its source code
to understand it in detail).

Regards,
Martin

From tjreedy at udel.edu  Fri Jul  3 22:45:42 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 03 Jul 2009 16:45:42 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E50BC.8060201@mrabarnett.plus.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<h2lia7$v1j$1@ger.gmane.org>
	<4A4E50BC.8060201@mrabarnett.plus.com>
Message-ID: <h2lqll$mcq$1@ger.gmane.org>

MRAB wrote:
> Terry Reedy wrote:
>> Dirkjan Ochtman wrote:
>>
>>> It needs to be decided where the hg repositories will live. I'd like
>>> to propose to keep the hgwebdir instance at hg.python.org. This is an
>>> accepted standard for many organizations, and an easy parallel to
>>> svn.python.org. The 2.7 (trunk) repo might live at
>>> http://hg.python.org/main/, for example, with py3k at
>>> http://hg.python.org/py3k/.
>>
>> I would very much like the 'k' dropped from the py3 name. It was a 
>> funny joke when py3 was vaporware, now it is excess baggage which only 
>> puzzles non-insiders and newcomers.
>>
>> I think the two repos should be either symmetrically named
>>
>> hg.python.org/py2
>> hg.python.org/py3
>>
>> If one must be designated 'main', it should be py3.
>>
>> Continuing to call py2 'main' will continue to discourage use of py3.
>>
> We could regard py3k as the phase from the original concept of Python 3
> to its 'prototype', Python 3.0. 

Right. And that phase is over, especially with Barry posting today on 
python-list that there will be no more 3.0.x releases ever.

> Python 3.1 would be the first 'real/usable' version.

Right. 'is'.  as Barry also posted.

tjr


From martin at v.loewis.de  Fri Jul  3 22:49:57 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 22:49:57 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
Message-ID: <4A4E6EF5.4000901@v.loewis.de>

> It will handle it, for sure, but I think it would all go easier if we
> could work with stricter subset branches (and leave the effective
> cherrypicking for the occasional problem).

So I think the PEP should propose a workflow (or: merge flow) if you
think we would be better off with a different one.

In proposing such a workflow, consider these requirements:
- we current have four active "maintenance" branches (i.e. where
  the entire code basis evolves): trunk, 3k, 2.6, and 3.1 (3.0
  also until this morning).
- in addition, we have two security branches currently: 2.4 and
  2.5, although 2.4 will be closed soon.
- our committers consistently refuse to merge changes across
  branches themselves, and likely continue to do so unless there
  is some feature of hg that I missed (e.g. one were merging
  would happen without any user specifically asking for it)

Regards,
Martin

From martin at v.loewis.de  Fri Jul  3 23:00:35 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 23:00:35 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
Message-ID: <4A4E7173.1040507@v.loewis.de>

> Should we consider adding a sys.revision attribute and begin the
> deprecation of sys.subversion?

I wouldn't mind killing sys.subversion "right away" (i.e. in trunk
and 3k - obviously it has to stay in 2.6 and 3.1, and all the older
branches).

I'm -1 on calling it "sys.revision", as this makes it difficult to
tell what the actual versioning system was, and hence how the
data should be interpreted. It will already be a problem for 2.6,
when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
which will surely crash existing applications.

I'm not sure what the motivation for a sys.revision is; it's
probably similar to the desire of calling the machine code.python.org
(instead of hg.python.org). It gives the illusion of being agnostic
of the actual RCS being used. However, this is a complete illusion:
anybody using it (either code.python.org, or sys.revision), *cannot*
be agnostic of the specific technology.

Regards,
Martin

From martin at v.loewis.de  Fri Jul  3 23:25:15 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 03 Jul 2009 23:25:15 +0200
Subject: [Python-Dev] 3.1 buildbots
Message-ID: <4A4E773B.10901@v.loewis.de>

I have now turned the 3.0 buildbots off, and created new
builders for 3.1.

Regards,
Martin

From brett at python.org  Fri Jul  3 23:39:14 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Jul 2009 14:39:14 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E7173.1040507@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com> 
	<4A4E7173.1040507@v.loewis.de>
Message-ID: <bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com>

On Fri, Jul 3, 2009 at 14:00, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> > Should we consider adding a sys.revision attribute and begin the
> > deprecation of sys.subversion?
>
> I wouldn't mind killing sys.subversion "right away" (i.e. in trunk
> and 3k - obviously it has to stay in 2.6 and 3.1, and all the older
> branches).
>
> I'm -1 on calling it "sys.revision", as this makes it difficult to
> tell what the actual versioning system was, and hence how the
> data should be interpreted. It will already be a problem for 2.6,
> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
> which will surely crash existing applications.
>
> I'm not sure what the motivation for a sys.revision is; it's
> probably similar to the desire of calling the machine code.python.org
> (instead of hg.python.org). It gives the illusion of being agnostic
> of the actual RCS being used. However, this is a complete illusion:
> anybody using it (either code.python.org, or sys.revision), *cannot*
> be agnostic of the specific technology.


We could add another value in the tuple that specifies the VCS: ('CPython',
'branches/release25-maint', '61464', 'svn'). I agree that VCSs are not
universally the same, but the concept of a revision is universal.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/7ca83a16/attachment-0001.htm>

From brett at python.org  Fri Jul  3 23:41:56 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Jul 2009 14:41:56 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2lia7$v1j$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<h2lia7$v1j$1@ger.gmane.org>
Message-ID: <bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>

On Fri, Jul 3, 2009 at 11:22, Terry Reedy <tjreedy at udel.edu> wrote:

> Dirkjan Ochtman wrote:
>
>  It needs to be decided where the hg repositories will live. I'd like
>> to propose to keep the hgwebdir instance at hg.python.org. This is an
>> accepted standard for many organizations, and an easy parallel to
>> svn.python.org. The 2.7 (trunk) repo might live at
>> http://hg.python.org/main/, for example, with py3k at
>> http://hg.python.org/py3k/.
>>
>
> I would very much like the 'k' dropped from the py3 name. It was a funny
> joke when py3 was vaporware, now it is excess baggage which only puzzles
> non-insiders and newcomers.
>

Is it really that confusing? I have never heard of anyone asking "what is
py3k?" Plus I like keeping that bit of Python history around. I know I still
use py3k as shorthand for Python 3.x. And we are not that serious of a
bunch. =)


>
> I think the two repos should be either symmetrically named
>
> hg.python.org/py2
> hg.python.org/py3
>

If we make it universal I say it should be '2.x' and '3.x'. The whole 'py'
prefix is redundant.


>
> If one must be designated 'main', it should be py3.
>
> Continuing to call py2 'main' will continue to discourage use of py3.


Yeah, 2.x shouldn't be special anymore.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/29e4f1ba/attachment.htm>

From martin at v.loewis.de  Fri Jul  3 23:52:39 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 23:52:39 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
	<4A4E7173.1040507@v.loewis.de>
	<bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com>
Message-ID: <4A4E7DA7.2030402@v.loewis.de>

> We could add another value in the tuple that specifies the VCS:
> ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that
> VCSs are not universally the same, but the concept of a revision is
> universal.

Actually, I think that's not the case. For bzr, the usual way of
identifying a revision is by revision number, which, however, is not
unique within a project, as each branch will use contiguous integers
for numbers. There are also unique identifications - so a bzr revision
has actually two numbers.

More general, in a DVCS, it is not possible to access the revision being
referred to by such a tuple. For sys.subversion, if [0]=='CPython', then
you could go to svn.python.org. For a DVCS, the revision being
identified may not be publically available, or may not live on a host
that you can infer from your proposed sys.revision.

For cloned branches, I wonder how sys.revision[1] would be computed.

Regards,
Martin

From brett at python.org  Fri Jul  3 23:53:52 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Jul 2009 14:53:52 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E7DA7.2030402@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com> 
	<4A4E7173.1040507@v.loewis.de>
	<bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com> 
	<4A4E7DA7.2030402@v.loewis.de>
Message-ID: <bbaeab100907031453k2b74afadm3b32dfc48a1742b0@mail.gmail.com>

On Fri, Jul 3, 2009 at 14:52, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> > We could add another value in the tuple that specifies the VCS:
> > ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that
> > VCSs are not universally the same, but the concept of a revision is
> > universal.
>
> Actually, I think that's not the case. For bzr, the usual way of
> identifying a revision is by revision number, which, however, is not
> unique within a project, as each branch will use contiguous integers
> for numbers. There are also unique identifications - so a bzr revision
> has actually two numbers.
>
> More general, in a DVCS, it is not possible to access the revision being
> referred to by such a tuple. For sys.subversion, if [0]=='CPython', then
> you could go to svn.python.org. For a DVCS, the revision being
> identified may not be publically available, or may not live on a host
> that you can infer from your proposed sys.revision.
>
> For cloned branches, I wonder how sys.revision[1] would be computed.


So are you saying we should drop the idea of a revision value altogether, or
just embrace the differences and add a sys.mercurial attribute?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/fa394dbe/attachment.htm>

From martin at v.loewis.de  Fri Jul  3 23:59:14 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 03 Jul 2009 23:59:14 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031453k2b74afadm3b32dfc48a1742b0@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
	<4A4E7173.1040507@v.loewis.de>	<bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com>
	<4A4E7DA7.2030402@v.loewis.de>
	<bbaeab100907031453k2b74afadm3b32dfc48a1742b0@mail.gmail.com>
Message-ID: <4A4E7F32.7080309@v.loewis.de>

> So are you saying we should drop the idea of a revision value
> altogether, or just embrace the differences and add a sys.mercurial
> attribute?

That's what I would propose. It should be a best-effort(*) approach at
providing all information that is needed to really find the source
used for the specific version.

Regards,
Martin

(*) even for svn it was best-effort only in case there were local
modifications.

From dirkjan at ochtman.nl  Sat Jul  4 00:00:28 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 00:00:28 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2lia7$v1j$1@ger.gmane.org>
	<bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>
Message-ID: <ea2499da0907031500j4ada45fwda911c2de9ea827@mail.gmail.com>

On Fri, Jul 3, 2009 at 23:41, Brett Cannon<brett at python.org> wrote:
> If we make it universal I say it should be '2.x' and '3.x'. The whole 'py'
> prefix is redundant.

Right, I was aiming for /python/2.x and /python/3.x as well.

Actually, I currently have /cpython to also make CPython less special
among it's peers, but that idea was met with some resistance on
#python-dev.

Cheers,

Dirkjan

From brett at python.org  Sat Jul  4 00:02:08 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Jul 2009 15:02:08 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907031500j4ada45fwda911c2de9ea827@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<h2lia7$v1j$1@ger.gmane.org>
	<bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com> 
	<ea2499da0907031500j4ada45fwda911c2de9ea827@mail.gmail.com>
Message-ID: <bbaeab100907031502u2c2645ci6c75a33ede9a9e21@mail.gmail.com>

On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:

> On Fri, Jul 3, 2009 at 23:41, Brett Cannon<brett at python.org> wrote:
> > If we make it universal I say it should be '2.x' and '3.x'. The whole
> 'py'
> > prefix is redundant.
>
> Right, I was aiming for /python/2.x and /python/3.x as well.
>
> Actually, I currently have /cpython to also make CPython less special
> among it's peers, but that idea was met with some resistance on
> #python-dev.


Don't worry about doing that right now. When the stdlib gets separated out
we can revisit this. But for now just leave it as python.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/4252740d/attachment-0001.htm>

From barry at python.org  Sat Jul  4 00:03:09 2009
From: barry at python.org (Barry Warsaw)
Date: Fri, 3 Jul 2009 18:03:09 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E7173.1040507@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
	<4A4E7173.1040507@v.loewis.de>
Message-ID: <EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>

On Jul 3, 2009, at 5:00 PM, Martin v. L?wis wrote:

> I'm -1 on calling it "sys.revision", as this makes it difficult to
> tell what the actual versioning system was, and hence how the
> data should be interpreted. It will already be a problem for 2.6,
> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
> which will surely crash existing applications.

I can release a 2.6.3 right before the cut-over (well, just about any  
time between now and August 1st).  Should we just plan now for a 2.6.3  
on say July 24th, with a release candidate on July 20th?

> I'm not sure what the motivation for a sys.revision is; it's
> probably similar to the desire of calling the machine code.python.org
> (instead of hg.python.org). It gives the illusion of being agnostic
> of the actual RCS being used. However, this is a complete illusion:
> anybody using it (either code.python.org, or sys.revision), *cannot*
> be agnostic of the specific technology.

Agreed.  I originally chose code.python.org because I didn't want to  
be biased (maybe I should have been :).   +1 for hg.python.org.  I'd  
prefer to spell out sys.mercurial_revision.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 832 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/c88bc27b/attachment.pgp>

From martin at v.loewis.de  Sat Jul  4 00:09:52 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Jul 2009 00:09:52 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <4A4E81B0.6010700@v.loewis.de>

> - First of all, I've got the basic conversion down, I've done it a few
> times now, with progressively better results. You can view some
> results at http://hg.python.org/, which has a preliminary cpython
> repository. *** The changeset hashes for that repo will change, so you
> won't be able to commit or pull from it in the future.***

I'd like to see as much infrastructure going "life" well in advance
of the switch over. Ideally, when we do switch, there should be no
surprises (i.e. all problems should be known, and, ideally, be
resolved).

> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs

These are the ones that I was thinking of (and mentioned before, so
they got onto your list :-)

I would drop the roundup integration from the things that need to
be done pre-migration - there currently is no svn integration, so
not having it for hg is not a step backwards.

>     * buildbots: both the regular and the community build masters must
> be notified. Fortunately buildbot includes support for hg. I've also
> implemented this for Mercurial itself, so I don't expect problems
> here.

This should also be setup before the switchover, with a few test
buildslaves (say, a Unix one and a Windows one).

> In a DVCS, the committers are not necessarily the same
> people who push, and so we can't check if the committer is a
> contributor. We could use a hook to check if the committer is a
> contributor if we keep a list of registered contributors.

In the first sentence, you say that it can't actually work - so I
think we should drop the test.

> I propose that the revision identifier will be the short version of
> hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
> (instead of 'M') if the working directory from which it was built was
> modified. This mirrors the output of the hg id command, which is
> intended for this kind of usage.

I would like to see this well before the switch also. It could be
a patch (unified diff) stored in the tracker, or it could be an actual
branch that then needs to get merged with all active branches (IIUC).

Regards,
Martin

From martin at v.loewis.de  Sat Jul  4 00:15:54 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Jul 2009 00:15:54 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>	<4A4E7173.1040507@v.loewis.de>
	<EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>
Message-ID: <4A4E831A.6090404@v.loewis.de>

Barry Warsaw wrote:
> On Jul 3, 2009, at 5:00 PM, Martin v. L?wis wrote:
> 
>> I'm -1 on calling it "sys.revision", as this makes it difficult to
>> tell what the actual versioning system was, and hence how the
>> data should be interpreted. It will already be a problem for 2.6,
>> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
>> which will surely crash existing applications.
> 
> I can release a 2.6.3 right before the cut-over (well, just about any
> time between now and August 1st).  Should we just plan now for a 2.6.3
> on say July 24th, with a release candidate on July 20th?

I'm fine with that plan - but the original problem remains. We will
surely release 2.6.4 at some point, and it will have a different version
identification (based on hg rev ids).

So those existing applications (which are probably few) will then crash
for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
arrange to fake sys.subversion somehow (e.g. freezing it on the last
subversion revision - which might still break applications that insist
on accessing the revision mentioned - not sure whether such applications
actually exist).

Regards,
Martin

From barry at python.org  Sat Jul  4 00:37:19 2009
From: barry at python.org (Barry Warsaw)
Date: Fri, 3 Jul 2009 18:37:19 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E831A.6090404@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>	<4A4E7173.1040507@v.loewis.de>
	<EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>
	<4A4E831A.6090404@v.loewis.de>
Message-ID: <E233B5DE-56E8-4CC1-A0A5-2D65BA593868@python.org>

On Jul 3, 2009, at 6:15 PM, Martin v. L?wis wrote:

> I'm fine with that plan - but the original problem remains. We will
> surely release 2.6.4 at some point, and it will have a different  
> version
> identification (based on hg rev ids).
>
> So those existing applications (which are probably few) will then  
> crash
> for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
> arrange to fake sys.subversion somehow (e.g. freezing it on the last
> subversion revision - which might still break applications that insist
> on accessing the revision mentioned - not sure whether such  
> applications
> actually exist).

Doesn't Mercurial support an Subversion bridge?  Would it be possible  
to provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5,  
2.6, and 3.1?  If so, then the release managers would simply have to  
cut their releases from the svn copy instead of the hg master.  /All/  
other work would be done from the hg master.  This shouldn't be too  
much of a burden since it's done so rarely and would end with the EOL  
of each of those branches.

It would mean maintaining the bridge until all currently released  
versions are EOL.

If that's not possible or feasible, then given the documented  
sys.subversion semantics, I think we should just freeze the tuple at  
e.g. ('CPython', 'branches/release26-maint', None).

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 832 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090703/ba99b122/attachment.pgp>

From mhammond at skippinet.com.au  Sat Jul  4 01:30:28 2009
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Sat, 04 Jul 2009 09:30:28 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E1009.4050901@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com>
Message-ID: <4A4E9494.2040406@skippinet.com.au>

On 4/07/2009 12:04 AM, Nick Coghlan wrote:

> However, I expect that would still be painful to work with for Windows
> developers, even if it prevented any line ending problems from making
> their way into the main repository. I believe that is where the
> win32text extensions can help. Looking at the Wiki page for win32text
> [1], I believe it would be a matter of configuring the extension to
> encode and decode all files with the extensions:
>
> *.py
> *.pyw
> *.h
> *.c
> *.in
> *.rst
> *.asdl
>
> That said, I don't see a way to tell win32text to also translate files
> which don't have an extension at all (e.g. NEWS or ACKS), and there
> doesn't seem to be a way to tell it to *skip* files matching certain
> patterns (if there was, we could just tell it to ignore extensions like
> .bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert
> everything else)

The big problem I have with win32text is that the rules are not 
versioned, meaning we will rely on every (windows only?) developer 
manually enabling the win32text extension, then manually adding these 
rules to an unversioned hgrc file.  Should the rules change over time, 
we will need to communicate that to every developer.  Contrast this 
with, say, SVN, where the rule is a versioned property on the file.

Further, if a repo *already* has mixed line endings and the file doesn't 
appear in the rules, win32text makes *lots* of noise regarding existing 
line endings and to "review your rules before you commit."  In effect, 
it seems to me that win32text will only work effectively if *all* 
developers using a repo are using it, or none are, or other steps have 
been put in place to ensure \r\n line endings in the repo are otherwise 
impossible.  I've completely failed to use win32text on the mozilla hg 
repo for exactly this reason.

Cheers,

Mark

From scott+python-dev at scottdial.com  Sat Jul  4 01:11:03 2009
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Fri, 03 Jul 2009 19:11:03 -0400
Subject: [Python-Dev] PEP 376
In-Reply-To: <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>
	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>	<877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp>
	<94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com>
Message-ID: <4A4E9007.1050908@scottdial.com>

Tarek Ziad? wrote:
> On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull<stephen at xemacs.org> wrote:
>> Another general principle: even in the draft PEP, say "is", not "will
>> be".
> 
> Ok I'll fix that. That's a French stuff : in french, "will be" isn't
> speculative at all.
> 

I don't think "will be" is necessarily speculative in English either. I
think the issue is that after the PEP is implemented, the document lives
on. And when one reads, "X will be done." If "X" is not done in the
current implementation, it is unclear whether that is an error or a
promise that at some point in the future the implementation will be
changed to do "X". In other words, the PEP will live on long after you
have completed the implementation and it's at that point that
occurrences of "will be" in the PEP become speculative.

Someone feel free to correct me if I am incorrect about the desired tone
and use of the document..

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From mhammond at skippinet.com.au  Sat Jul  4 01:37:29 2009
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Sat, 04 Jul 2009 09:37:29 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A4D3C44.60504@gmail.com>	
	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>	
	<4A4E0825.5020407@skippinet.com.au>
	<ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
Message-ID: <4A4E9639.3010608@skippinet.com.au>

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:
> On Fri, Jul 3, 2009 at 15:31, Mark Hammond<mhammond at skippinet.com.au>  wrote:
>> So we must work without effective EOL support?  I fear we will end up like
>> the mozilla hg repo with some files in windows line endings and some with
>> linux.  While my editing tools are good enough to preserve existing EOL
>> styles, I've found myself accidentally checking in new \r\n terminated files
>> in a repo which otherwise uses \n line endings.  IMO, SVN's EOL support was
>> better than no EOL support.
>
> This is why we'll have hooks -- to prevent you from pushing changesets
> with inappropriate, to say the least, and, if you're willing to do a
> little bit of extra work, to prevent you from committing them.
>
>> This is exactly why I was asking for your advice - I can't work out how to
>> work effectively with win32text as it stands myself, so remain stuck
>> accidently checking in files with inappropriate line endings and stuck
>> working out how to move pywin32's CVS repo with abandoning the very
>> primitive EOL safety it offers...
>
> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

I'm surprised it isn't obvious.  Think about it this way: CVS had basic 
EOL support and SVN gave slightly better support, and we leveraged this 
support in both those tools.  The end result is a very clean repo with 
consistent line endings and a distinct lack of developer friction 
between the platforms.

Your position seems to be that we *never* needed EOL conversion, 
especially in SVN which offered decent hooks, and that having a source 
repo with a mismatch and unpredictable line endings isn't really a 
problem at all.

Even just at face value, it seems clear we have taken full advantage of 
that feature for around 1.5 decades, so to suggest we never actually 
needed it, or at least don't need it now for some reason, seems slightly 
disingenuous...

Cheers,

Mark

From martin at v.loewis.de  Sat Jul  4 01:47:39 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Jul 2009 01:47:39 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <E233B5DE-56E8-4CC1-A0A5-2D65BA593868@python.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>	<4A4E7173.1040507@v.loewis.de>
	<EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>
	<4A4E831A.6090404@v.loewis.de>
	<E233B5DE-56E8-4CC1-A0A5-2D65BA593868@python.org>
Message-ID: <4A4E989B.9010806@v.loewis.de>

> Doesn't Mercurial support an Subversion bridge?  Would it be possible to
> provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6,
> and 3.1?  If so, then the release managers would simply have to cut
> their releases from the svn copy instead of the hg master.  /All/ other
> work would be done from the hg master.  This shouldn't be too much of a
> burden since it's done so rarely and would end with the EOL of each of
> those branches.
> 
> It would mean maintaining the bridge until all currently released
> versions are EOL.

That sounds very complicated (but Dirkjan should correct me if I'm
wrong).

I have already mentally prepared to port welease to hg, to create
the missing 2.4 and 2.5 releases.

> If that's not possible or feasible, then given the documented
> sys.subversion semantics, I think we should just freeze the tuple at
> e.g. ('CPython', 'branches/release26-maint', None).

That would work for me (I had to re-read the documentation to see
that None is a valid, documented value for version, and allowed
if "the tree was exported").

Interesting to notice that sys.subversion didn't even last the
life time of one feature release :-)

Regards,
Martin

From ncoghlan at gmail.com  Sat Jul  4 02:47:05 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 10:47:05 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks
 to	provide distribution metadata
In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
Message-ID: <4A4EA689.1010704@gmail.com>

Tarek Ziad? wrote:
> 2009/7/3 Paul Moore <p.f.moore at gmail.com>:
>> Does this sound sensible? Tarek, would you be OK with me attempting to
>> modify your prototype to support this protocol? Are there any tests
>> for PEP 376, so that I can confirm I haven't completely broken
>> something? If I can, I'll knock up some simple prototype importers for
>> non-standard examples, and see how they work with all this.
> 
> Yes that's exactly what I was thinking about after the discussion we
> had in the other
> thread. This change would allow much more flexibility.

It's also very similar to the ideas that started to tick over in the
back of my head after the previous discussion (only far more fleshed out
that anything I got to - fortunately I read Paul's message before
spending too much time on it!)

Specifically regarding this comment from Paul:

> Finder.uninstall(distribution_name, filter=callable, installer=name)
> Uninstall the given distribution. It's likely that many finders will
> be read-only. In that case, this function should return None.
> Otherwise, return a list of the "files" removed. (This may need some
> clarification, as many finders won't have a concept of a "file name").

PEP 302 finders and loaders actually *are* expected to have a concept of
a "file name". For Python modules they they are expected to have it so
they can populate __file__ correctly (runpy is also a lot happier with
them when they expose get_filename(), a relatively recent addition to
the PEP 302 API). For other resources they're expected to have it so the
get_data() API can work correctly.

One other thing to consider: it may be desirable to have hooks for
meta_path and path_hooks in distutils that are checked *before* the full
import hooks in the sys module. The reason I say this is that it would
allow someone to override just the metadata retrieval (e.g. to pick up
the extra dependency information saved by setuptools and friends)
without having to worry about breaking the actual imports from those
locations.

Cheers,
Nick.

P.S. +lots on using 'metadata' in the PEP 376 method names rather than
the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
gratuitous in this case.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat Jul  4 03:20:58 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 11:20:58 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E9494.2040406@skippinet.com.au>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com>
	<4A4E9494.2040406@skippinet.com.au>
Message-ID: <4A4EAE7A.5090406@gmail.com>

Mark Hammond wrote:
> On 4/07/2009 12:04 AM, Nick Coghlan wrote:
> The big problem I have with win32text is that the rules are not
> versioned, meaning we will rely on every (windows only?) developer
> manually enabling the win32text extension, then manually adding these
> rules to an unversioned hgrc file.  Should the rules change over time,
> we will need to communicate that to every developer.  Contrast this
> with, say, SVN, where the rule is a versioned property on the file.
> 
> Further, if a repo *already* has mixed line endings and the file doesn't
> appear in the rules, win32text makes *lots* of noise regarding existing
> line endings and to "review your rules before you commit."  In effect,
> it seems to me that win32text will only work effectively if *all*
> developers using a repo are using it, or none are, or other steps have
> been put in place to ensure \r\n line endings in the repo are otherwise
> impossible.  I've completely failed to use win32text on the mozilla hg
> repo for exactly this reason.

I didn't realise Mercurial didn't have a way for a repository to provide
versioned extension settings, but in this case I think using our own
server side hook can deal with the problem (either adding it to the
existing whitespace hook that checks for tab/space sanity in Python
files, or else as a separate hook that just checks line endings).

Such a hook should be able to base the determination of which files it
checks for \r\n line endings on a normal versioned file in the
repository. For everything that *isn't* mentioned in that file, the hook
would check that it had \n line endings.

I believe that would address both of your concerns:
  1. A server side hook means that the rules are enforced on everyone
regardless of whether they use win32text or not (e.g no Linux developers
corrupting a .dsp file by opening it and saving it with \n line endings)
  2. The server side hooks rules will be versioned, allowing people to
copy them to their hgrc file so that win32text will correctly handle the
line endings for them.

Also, on another read of the win32text instructions, I believe the
configuration actually *does* support negative filters, it just doesn't
explain them very well (it only mentions them in the context of
overriding default settings in the config file for a specific
repository). I think a .hgrc file along the following lines might do
what we want (I'm not developing on Windows though, so I haven't
actually tested it):

=============
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !
=============

Assuming we can find settings that do what we want then a sample HGRC
file could easily be provided in the Misc directory alongside things
like python-mode.el.

An alternative to the above would be to store *everything* (even the
Windows specific files) in the repository with \n line endings (still
enforced by a server side hook), and then *require* that Windows users
run win32text with settings like the following:

=============
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave known binary files alone
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave known binary files alone
**.ico = !
**.bmp = !
=============

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From mhammond at skippinet.com.au  Sat Jul  4 03:20:29 2009
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Sat, 04 Jul 2009 11:20:29 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A4D3C44.60504@gmail.com>	
	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>	
	<4A4E0825.5020407@skippinet.com.au>
	<ea2499da0907030643x78d83467nc469e8c3c98008e2@mail.gmail.com>
Message-ID: <4A4EAE5D.6000908@skippinet.com.au>

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:

> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

In that case, and given that I expect more Windows users to be clueless 
about EOL issues than users of other operating systems, I propose we 
check in all files initially with Windows line endings <wink>

Mark

From ncoghlan at gmail.com  Sat Jul  4 03:33:18 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 11:33:18 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031502u2c2645ci6c75a33ede9a9e21@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2lia7$v1j$1@ger.gmane.org>	<bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>
	<ea2499da0907031500j4ada45fwda911c2de9ea827@mail.gmail.com>
	<bbaeab100907031502u2c2645ci6c75a33ede9a9e21@mail.gmail.com>
Message-ID: <4A4EB15E.3080808@gmail.com>

Brett Cannon wrote:
> On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman <dirkjan at ochtman.nl
> <mailto:dirkjan at ochtman.nl>> wrote:
>     Actually, I currently have /cpython to also make CPython less special
>     among it's peers, but that idea was met with some resistance on
>     #python-dev.
> 
> Don't worry about doing that right now. When the stdlib gets separated
> out we can revisit this. But for now just leave it as python.

Yes, if we ever do that split we can leave the common code (stdlib, test
suite) under /python and move the CPython specific stuff to /cpython.

+1 on 2.x and 3.x for the main development branches (although that does
lead me to wonder if we should just make the maintenance branches 2.6,
3.1, etc).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From mhammond at skippinet.com.au  Sat Jul  4 03:33:06 2009
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Sat, 04 Jul 2009 11:33:06 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EAE7A.5090406@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com>
	<4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com>
Message-ID: <4A4EB152.6040406@skippinet.com.au>

On 4/07/2009 11:20 AM, Nick Coghlan wrote:
>
> I didn't realise Mercurial didn't have a way for a repository to provide
> versioned extension settings, but in this case I think using our own
> server side hook can deal with the problem (either adding it to the
> existing whitespace hook that checks for tab/space sanity in Python
> files, or else as a separate hook that just checks line endings).
>
> Such a hook should be able to base the determination of which files it
> checks for \r\n line endings on a normal versioned file in the
> repository. For everything that *isn't* mentioned in that file, the hook
> would check that it had \n line endings.
>
> I believe that would address both of your concerns:
>    1. A server side hook means that the rules are enforced on everyone
> regardless of whether they use win32text or not (e.g no Linux developers
> corrupting a .dsp file by opening it and saving it with \n line endings)
>    2. The server side hooks rules will be versioned, allowing people to
> copy them to their hgrc file so that win32text will correctly handle the
> line endings for them.

That sounds like it should work - presumably mention of this should go 
into the PEP.

(While that is great for Python, it is a shame I probably can't use the 
same strategy for other hg repos I care about as I don't have access to 
server side hooks - but that if off-topic for this list.)

> Also, on another read of the win32text instructions, I believe the
> configuration actually *does* support negative filters, it just doesn't
> explain them very well (it only mentions them in the context of
> overriding default settings in the config file for a specific
> repository). I think a .hgrc file along the following lines might do
> what we want (I'm not developing on Windows though, so I haven't
> actually tested it):

This *appears* to be correct at first glance, but in practice it doesn't 
interact well with wildcards specifications - particularly 
'**=cleverencode'.  I started a thread on the ML and submitted a patch a 
few months ago to fix this and it was accepted, but sadly it seemed to 
have dropped off the queue somewhere.  The patch now conflicts, and I've 
promised to resubmit it when I find time.  But even with that in place 
it doesn't address the more general problem that when only *some* 
developers use the extension - mixed mode files are still very possible, 
at which point win32text starts reporting enough spurious conflicts to 
become unusable for me.  Eg, doing a clean checkout of mozilla with 
win32text enabled results in a working tree with hundreds of files 
reporting every line in the file has changed.

It is worth noting that win32text also works on Linux, and that in 
theory, win32text should be able to allow Linux users to work seamlessly 
with a \r\n repo, in the same way it promises to allow Windows users to 
work with \n repos.  But in practice, it is simply never used by Linux 
developers, even in the hg core dev team.  Patches or comments regarding 
win32text are always met with "but no one here ever uses it".  If they 
did, and some of the core hg team tried to experiment with win32text and 
mixed line ending repos the problems would, I believe, become 
self-evident...

Cheers,

Mark

From ncoghlan at gmail.com  Sat Jul  4 03:37:50 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 11:37:50 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E81B0.6010700@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4E81B0.6010700@v.loewis.de>
Message-ID: <4A4EB26E.3050706@gmail.com>

Martin v. L?wis wrote:
> I would drop the roundup integration from the things that need to
> be done pre-migration - there currently is no svn integration, so
> not having it for hg is not a step backwards.

That's not quite true - the tracker turns text like "r64537" into a link
to the appropriate changeset on svn.python.org. It would be unfortunate
to lose that easy ability to indicate when a bug was fixed (or introduced).

This is more something on the roundup side though: recognising the
presence of a hg id in a comment and turning it into a link to
hg.python.org.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat Jul  4 03:44:17 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 11:44:17 +1000
Subject: [Python-Dev] PEP 376
In-Reply-To: <4A4E9007.1050908@scottdial.com>
References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com>	<4A4927D5.4040001@v.loewis.de>	<4A4A05DE.2040804@gmail.com>	<ca471dc20906301021m2f5386f1r1e144ba9f1970d60@mail.gmail.com>	<79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com>	<94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com>	<79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com>	<79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com>	<94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com>	<877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp>	<94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com>
	<4A4E9007.1050908@scottdial.com>
Message-ID: <4A4EB3F1.6090705@gmail.com>

Scott Dial wrote:
> Someone feel free to correct me if I am incorrect about the desired tone
> and use of the document..

I don't think we're particularly consistent. For example, I actually
ending up adding a note to PEP 343 indicating that people shouldn't read
too much into the verb tense in that document because we were working on
it for over a year and things that were in the future when we started
were well in the past by the time we finished.

In general, I agree that using present tense ("is") in specifications
and documentation such as PEPs ends up standing the test of time better
than using future tense ("will be").

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat Jul  4 04:30:22 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 12:30:22 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EB152.6040406@skippinet.com.au>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com>
	<4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com>
	<4A4EB152.6040406@skippinet.com.au>
Message-ID: <4A4EBEBE.7040604@gmail.com>

Mark Hammond wrote:
> This *appears* to be correct at first glance, but in practice it doesn't
> interact well with wildcards specifications - particularly
> '**=cleverencode'.  I started a thread on the ML and submitted a patch a
> few months ago to fix this and it was accepted, but sadly it seemed to
> have dropped off the queue somewhere.  The patch now conflicts, and I've
> promised to resubmit it when I find time.  But even with that in place
> it doesn't address the more general problem that when only *some*
> developers use the extension - mixed mode files are still very possible,
> at which point win32text starts reporting enough spurious conflicts to
> become unusable for me.  Eg, doing a clean checkout of mozilla with
> win32text enabled results in a working tree with hundreds of files
> reporting every line in the file has changed.

Ah, the scope of the issues begin to become clear...

While a server side hook should be able to deal with the mixed mode file
problem, I'm not sure what can be done about the problems with properly
configuring win32text.

If wildcard specifications don't interact properly with filtered
negative specifications then that would appear to rule out that
approach. The presence of text files without extensions like NEWS and
ACKS in the Python repository appears to rule out the use of positive
filters to select only the files that are stored in the repository with
\n line endings.

I spent some more time exploring the approach recommended on the
win32text documentation page as a possible way of handling the situation:
1. Store all text files (even Windows specific ones) in \n format in the
repository
2. Apply the win32text precommit hook to forbid the introduction of \r\n
line endings
3. Use the recommended settings from the win32text documentation:
===============================
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
===============================

This would be the equivalent of setting "svn:eol-style native" on every
file in the repository.

However, in running Martin's query (svn pg -R svn:eol-style .|grep CRLF)
over 2.x and 3.x checkouts, I found that \r\n line endings are currently
enforced for:
 - .bat files under Tools/buildbot
 - .dsp and .dsw files under PC/VC6
 - Lib/email/test/data/msg_26.txt

I believe the rationale for the first two is to allow a source tarball
to be prepared on Linux but still be usable on Windows (e.g. see [1]).

I'm not clear on the rationale for the explicit CRLF line ending on the
email test message, but I would guess it is to ensure that the email
module can handle CRLF line endings correctly regardless of platform.

Only VC6 files appear on the list because later versions of Visual
Studio actually tolerate \n line endings in their project files.

Mercurial's heuristic handling of text vs binary and expected line
endings fails completely for these use cases.

I think there needs to be a solid answer in place for these use cases
before the actual migration to Mercurial takes place. A hand-waving "use
win32text" isn't enough - it needs to be "use win32text with these exact
settings" (with server side hook support to enforce the rules).

And since Mercurial doesn't even allow us to say "this is a binary file"
 the way CVS used to I'm currently not seeing any way for that to happen
except for win32text to be updated to correctly handle wild cards in
combination with negative filters.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Sat Jul  4 05:20:04 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 04 Jul 2009 12:20:04 +0900
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
Message-ID: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>

Dirkjan Ochtman writes:
 > On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull<stephen at xemacs.org> wrote:
 > > I'll have to try them again now that 1.3 is out, but I found Mercurial
 > > named branches fundamentally unsuited to release management.
 > 
 > Can you explain why, please? It's not clear from what you say
 > below.

Well, the main problem I had was one that you say has improved: the
various issues of confusion due to the presence of multiple active
heads in a single repository.

IME, Mercurial strongly encourages a non-branching style.  Although I
can't fully explain in concrete terms what makes me feel that way,
it's certainly consistent with your own inclination to advise "subset
branches".  Part of it comes from the fact that you can't have a
single revision on two branches.  I would really like the node of a
release branch to be on both the branch and the mainline so that the
tag appears in the history of both, but that's not possible.

Another issue (which you say has improved) is the handling of multiple
heads.  With unnamed heads, it's just too painful to leave them
hanging around, so you merge them immediately (in fact, all XEmacs
committers currently active use either "pull -u" or the fetch
extension).

With named branches, the additional heads do tend to hang around.  I
found that named branches tended to get inadvertantly pushed, and
worse, they'd end up being the tip, which Mercurial treats specially.
In one case a completely private branch inadvertantly got pushed into
my "pristine" clone of the public repo, and from there into the public
repo, where (since it had been the tip in my private workspace) it
ended up as the tip in public.  Embarrassing, to say the least.
Fortuntely, the branch was more or less ready to be pushed anyway, but
one of my colleagues ended up working on that branch without realizing
that he wasn't on (his) mainline any more, and wondered why some
previously done work suddenly disappeared.  A good time was had by
nobody involved.

I don't know if that has been fixed in hg; the experience was painful
enough that my workflow adjusted immediately.

 > > Ditto named branches. ?The problem is that (unless the internal
 > > implementation has changed very recently) a Mercurial revision can be
 > > on exactly one named branch (or on the trunk).
 > 
 > That's still true.
 > 
 > > Which defeats the purpose of having named branches, really. ?(I mean
 > > the version control purpose; obviously it still can save disk space.)
 > 
 > Why does it defeat the purpose? What, in your opinion, is the
 > purpose?

I use named branches to collect a sequence of revisions as a named
object, for viewing and manipulation, as differentiated from some
other sequence, for *various* values of "some other sequence".  The
problem is that suppose you have a branch A off the trunk, and you
then (several revisions down the line of development) branch B off A.
Now A meanders off and runs into "not ready for prime time" problems,
while B just swims along.  Problem is, you can't easily find the
history of B relative to the trunk because much of its history (since
forking from the *trunk*) is labeled A.

 > > Unless you're really short on space, though, that's not a big deal.
 > > What would be more important to me (not that I matter for the purpose
 > > of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
 > > the other way around: pulling an external branch into a named branch.
 > > I have a feeling that working with such a repository with others would
 > > be a little difficult.
 > 
 > Can you give an example?

No, I haven't tried it.  What concerns me is that I suspect that the
branch name will end up as part of the revision's internal identifier,
and that means that if you and I separately create the same named
branch we have to choose exactly the same name or the branches won't
be recognized as the same, resulting in the mother of all spurious
merge conflicts.

 > > As others (MvL, I think) have commented, this isn't really relevant to
 > > Python which generally has four mainlines going at once. ?I don't see
 > > why the requirements are going to change with the shift to hg, and I
 > > see no reason why hg won't handle the existing workflow just fine.
 > 
 > It will handle it, for sure, but I think it would all go easier if we
 > could work with stricter subset branches (and leave the effective
 > cherrypicking for the occasional problem).

Sure, but what do you propose?  That we nuke Python 3.1, 3.2, and 2.6?
They're all pretty divergent from 2.7 by now, as well as from each
other.  Not to mention the "security branches" that are still around.
And individual developers are for sure going to do interesting things
in their private workspaces.

I see that George Brandl and Martin van Loewis seem to be accomodating
your viewpoint, but I don't get the impression that either you (as the
hg migration proponent) nor they (as core committers) realize how far
apart your assumptions are.  You are talking about the Mercurial
project, which has *one* line of development.  There are many such
projects; the ones I'm most familiar with are XEmacs and Scheme48,
which have adopted a subset branch approach different the Mercurial
project's (XEmacs is modeled on Scheme48).  Works well enough (when
I'm wearing my release manager hat; it's a little constraining as a
developer used to the insane flexibility of git).

But Python is *not* such a project.  The problem is not coordinating
concurrent development of a closeknit group of committers all working
on the same mainline.  In Python, there are *four* mainlines with
rather different purposes, and a diverse group of developers, some who
work on only one line of development, others who work on several, and
a few who accept the role of coordinating across them.




From martin at v.loewis.de  Sat Jul  4 05:21:46 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Jul 2009 05:21:46 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EB26E.3050706@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com>
Message-ID: <4A4ECACA.5080508@v.loewis.de>

Nick Coghlan wrote:
> Martin v. L?wis wrote:
>> I would drop the roundup integration from the things that need to
>> be done pre-migration - there currently is no svn integration, so
>> not having it for hg is not a step backwards.
> 
> That's not quite true - the tracker turns text like "r64537" into a link
> to the appropriate changeset on svn.python.org. It would be unfortunate
> to lose that easy ability to indicate when a bug was fixed (or introduced).
> 
> This is more something on the roundup side though: recognising the
> presence of a hg id in a comment and turning it into a link to
> hg.python.org.

Ah, right. That must be done, of course (although I suppose there is
little hope to have the existing references continue to work).

Regards,
Martin

From steve at pearwood.info  Sat Jul  4 05:36:49 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 4 Jul 2009 13:36:49 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2lia7$v1j$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2lia7$v1j$1@ger.gmane.org>
Message-ID: <200907041336.49868.steve@pearwood.info>

On Sat, 4 Jul 2009 04:22:57 am Terry Reedy wrote:

> I would very much like the 'k' dropped from the py3 name. It was a
> funny joke when py3 was vaporware, now it is excess baggage which
> only puzzles non-insiders and newcomers.

+1

Some day we'll be using Python3.7 and wondering what the "k" means. 
Alpha, beta, .... kappa version?



-- 
Steven D'Aprano

From skippy.hammond at gmail.com  Sat Jul  4 06:03:30 2009
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Sat, 04 Jul 2009 14:03:30 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EBEBE.7040604@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>	<4A4E0825.5020407@skippinet.com.au>
	<4A4E1009.4050901@gmail.com>	<4A4E9494.2040406@skippinet.com.au>
	<4A4EAE7A.5090406@gmail.com>	<4A4EB152.6040406@skippinet.com.au>
	<4A4EBEBE.7040604@gmail.com>
Message-ID: <4A4ED492.1050307@gmail.com>

On 4/07/2009 12:30 PM, Nick Coghlan wrote:
...
> I think there needs to be a solid answer in place for these use cases
> before the actual migration to Mercurial takes place. A hand-waving "use
> win32text" isn't enough - it needs to be "use win32text with these exact
> settings" (with server side hook support to enforce the rules).
>
> And since Mercurial doesn't even allow us to say "this is a binary file"
>   the way CVS used to I'm currently not seeing any way for that to happen
> except for win32text to be updated to correctly handle wild cards in
> combination with negative filters.

I agree with your conclusion.  My ruminating on this over the last few 
months leaves me thinking this would involve:

* my older 'accepted but then lost' hg patch to allow an explicit 'none' 
rule for a single file to override wildcards.

* win32text be enhanced to use a normal versioned file in the root of 
the repo, much like hgingore, where a project can maintain project wide 
rules.

* win32text be enhanced such that all python developers, regardless of 
platform, are willing to use this extension, even if the majority of 
files happen to use their native line ending (sauce for the goose is 
sauce for the gander, and all that...)

* commit hooks be implemented to enforce this - but this should not be 
necessary if the above was implemented and socially enforced.

Cheers,

Mark
>
> Cheers,
> Nick.
>
> [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html
>


From stephen at xemacs.org  Sat Jul  4 06:08:36 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 04 Jul 2009 13:08:36 +0900
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EBEBE.7040604@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4D3C44.60504@gmail.com>
	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>
	<4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com>
	<4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com>
	<4A4EB152.6040406@skippinet.com.au> <4A4EBEBE.7040604@gmail.com>
Message-ID: <87ocs1um2j.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > I'm not clear on the rationale for the explicit CRLF line ending on the
 > email test message, but I would guess it is to ensure that the email
 > module can handle CRLF line endings correctly regardless of platform.

IIRC, that's just the standard for text email.  Lines must be less
than 1000 characters long, should be no more than 78 characters long,
and are terminated by CRLF.  As they say, "honored more in the breach
than the observance", but that's what the RFCs say.


From pje at telecommunity.com  Sat Jul  4 06:31:54 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 04 Jul 2009 00:31:54 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
	<87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20090704042935.949263A40D7@sparrow.telecommunity.com>

At 12:20 PM 7/4/2009 +0900, Stephen J. Turnbull wrote:
>IME, Mercurial strongly encourages a non-branching style.  Although I
>can't fully explain in concrete terms what makes me feel that way,
>it's certainly consistent with your own inclination to advise "subset
>branches".  Part of it comes from the fact that you can't have a
>single revision on two branches.  I would really like the node of a
>release branch to be on both the branch and the mainline so that the
>tag appears in the history of both, but that's not possible.

I'm a total newbie to Mercurial - I've basically done little more 
than the tutorial, in fact.  So somebody correct me if what I'm about 
to say is completely stupid.

Wouldn't the simple thing to do in Mercurial, just be to use 
different repositories for long-lived branches?  I mean, if you're 
not merging them that much anyway, what's the point?  Isn't it just 
as easy to pull from another repository, as from another branch 
within that repository?


From ncoghlan at gmail.com  Sat Jul  4 07:13:52 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 15:13:52 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4ECACA.5080508@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com>
	<4A4ECACA.5080508@v.loewis.de>
Message-ID: <4A4EE510.7030409@gmail.com>

Martin v. L?wis wrote:
> Ah, right. That must be done, of course (although I suppose there is
> little hope to have the existing references continue to work).

I'd guess that the only way to keep those functional is to keep
svn.python.org around in read-only mode.

Although I'm not entirely sure about that - would it be possible to
build up a mapping from svn revision numbers to hg repository IDs as the
history information is being migrated? If that wasn't too difficult, it
might be worth doing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Sat Jul  4 07:50:33 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 04 Jul 2009 14:50:33 +0900
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <20090704042935.949263A40D7@sparrow.telecommunity.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
	<87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090704042935.949263A40D7@sparrow.telecommunity.com>
Message-ID: <87ljn5uhcm.fsf@uwakimon.sk.tsukuba.ac.jp>

P.J. Eby writes:

 > Wouldn't the simple thing to do in Mercurial, just be to use 
 > different repositories for long-lived branches?  I mean, if you're 
 > not merging them that much anyway, what's the point?

I basically agree with that, and so does Dirkjan, I think.  I'm not
sure why he brought up the idea of using named branches.

 > Isn't it just as easy to pull from another repository, as from
 > another branch within that repository?

Not if the other repository is remote and offline, and pulling from a
remote does take more time, as do things like diffs against that
branch.  The temptation is to leave the remote repos on the public
host, rather than put in the effort to make sure your local clones are
up to date.  If you do, there are minor annoyances like that.  So
people who actually work on more than one line of development learn
pretty soon to keep a local clone for each repo they're interested
in.  Problem solved, IMO, but some people worry about the extra space
taken up.

From dirkjan at ochtman.nl  Sat Jul  4 10:03:32 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 10:03:32 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
Message-ID: <ea2499da0907040103x5df7a15bh8f1dcb86c9adbaa0@mail.gmail.com>

On Fri, Jul 3, 2009 at 20:04, Brett Cannon<brett at python.org> wrote:
> Fine by me as long as people realize that if anything is questionable then
> the switch will not happen. Getting this right takes precedence over any
> deadline. And obviously we will need to do at least one live conversion on
> python.org hardware to make sure everything will work smoothly.

I'm not sure I see the need to do a (live? what does that mean in this
context) on python.org hardware. Why exactly is that better than me
doing it on one of my boxes, as long as all the necessary tools and an
idea of how to do it are publically available (from the pymigr repo,
for example)?

> And will make the idea of splitting out the standard library and tests a
> reasonable thing to do.

In due time, yes.

> While I really like the idea of using named branches for each release so
> that there is a single py3k branch that contains all relevant history for
> every release, I think we should start simple and go with cloned branches.
> That way the workflow does not radically shift from what we do now for svn
> to start. Once the conversion is done and people are comfortable with hg we
> can then discuss moving towards a named branch approach.

I don't think the cloned branches is much simpler than the named
branches approach, in several ways. For example, populating the branch
part of a sys.whatever value is significantly harder. Also, if you
follow a useful tagging approach, doing cloned branches means that
release tags aren't available on trunk/main/default. That seems like a
step backwards.

> Sounds reasonable to me. We can just make a list and send it to
> python-committers to make final decisions of what should stick around.

This list exists and has been referenced in my email a few times.

> I don't use tags so I don't really care, but in the name of easy transition
> I say we don't change the naming scheme (although I have no issue dropping
> obscure tags).

The current proposal is to clean up old tags to agree with the current
naming scheme (and dropping obscure and partial tags).

> Something else that can go out to python-committers before the switch.

This should just be done ASAP, it helps with a smooth conversion process.

> I don't think there is a single project we host -- all two of them -- that
> have not said they want to convert. So I say convert everything and let's
> turn off the svn server by the end of the year.

I say we tackle each one as we go. I say doing a good conversion job
is valuable, and we should take as much time as we need (though not
more). You advocate something similar below for the Python conversion.

> Can we check these scripts into the repository itself? That way there is a
> chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for
> ``make patchcheck``.

I'm not sure there's an easy way to make them into commands (although
I guess you could make an extension to that effect), but hooks would
be very easy.

> How about hg.python.org for the official branches and we keep
> code.python.org for personal branches of the developers like we have done
> with the bzr experiments?

I think that's just confusing. Most people seem to like hg.python.org,
and it's obvious that hg goes to hg.p.o. Dividing up the namespace
only makes it harder to find things.

> As I have said, we should change our workflow habits after the switch and
> people are comfortable with hg.

Well, not everyone agrees, and although I think it doesn't matter much
for the conversion itself, it may affect the branching strategy
discussion.

(sys.revision discussion elided because some others have already been
bikeshedding on it.)

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Sat Jul  4 10:05:26 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 10:05:26 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4EE510.7030409@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com>
	<4A4ECACA.5080508@v.loewis.de> <4A4EE510.7030409@gmail.com>
Message-ID: <ea2499da0907040105m10e30faei90a29dadb305312@mail.gmail.com>

On Sat, Jul 4, 2009 at 07:13, Nick Coghlan<ncoghlan at gmail.com> wrote:
> I'd guess that the only way to keep those functional is to keep
> svn.python.org around in read-only mode.

No, actually: the idea (I think I floated it in the PEP, as well), is
that I can write a simple extension for hgweb that knows the mapping
of SVN rev to hg rev and so can make
hg.python.org/python/2.x/rev/r32542 come out to the changeset that
resulted from converting that revision.

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Sat Jul  4 10:12:04 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 10:12:04 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2l7e7$mt$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2l7e7$mt$1@ger.gmane.org>
Message-ID: <ea2499da0907040112y50787220s1d66fdff37256c63@mail.gmail.com>

On Fri, Jul 3, 2009 at 17:17, Georg Brandl<g.brandl at gmx.net> wrote:
> Do you have a key to the second column in that file? E.g. the difference
> between "strip" and "discard" is not clear to me. "strip partial"?

strip == discard. strip = remove, merged should be obvious, keep-clone
means we'll keep the branch, in a separate clone, and keep-named means
we'll keep the branch as a named branch in one of the other clones.

> Why are there branch names starting with "../"?

Because some people in the history put their branch at
/python/branchname instead of /python/branches/branchname. That part
will get cleaned up.

[Regarding clone#branch syntax:]
> * Does it work with "hg pull" etc. too, afterwards?

Yes.

> * Does it support more than one branch?

You can only reference one branch at a time using the fragment specifier.

> I would just kill all feature branches unless someone indicates it is
> still used. ?There are very few active feature branches.
>
> (I guess in the case a branch gets killed erroneously it could still be
> re-created after the conversion?)

Re-creating it after the conversion could be done, I guess, but I'd
prefer to get it right, the first time. As the branch map says, I'm
mostly removing all the feature branches anyway.

> As long as it can send diffs...

It can.

> That gets very ugly as soon as you start pulling from repos that just
> fix a small typo or so.

Okay, so we do no automatic checking. We can always write a script
that walks the repository and finds all significant contributors.

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Sat Jul  4 10:18:42 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 10:18:42 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E81B0.6010700@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A4E81B0.6010700@v.loewis.de>
Message-ID: <ea2499da0907040118n20ed78ecr2827b8d988525f4d@mail.gmail.com>

On Sat, Jul 4, 2009 at 00:09, "Martin v. L?wis"<martin at v.loewis.de> wrote:
> I would drop the roundup integration from the things that need to
> be done pre-migration - there currently is no svn integration, so
> not having it for hg is not a step backwards.

Yeah, I mean just the linking here.

> In the first sentence, you say that it can't actually work - so I
> think we should drop the test.

Okay.

> I would like to see this well before the switch also. It could be
> a patch (unified diff) stored in the tracker, or it could be an actual
> branch that then needs to get merged with all active branches (IIUC).

Yeah. Some help here would be welcome, too, as C is not a language I use often.

Cheers,

Dirkjan

From dirkjan at ochtman.nl  Sat Jul  4 10:20:38 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 4 Jul 2009 10:20:38 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <E233B5DE-56E8-4CC1-A0A5-2D65BA593868@python.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
	<4A4E7173.1040507@v.loewis.de>
	<EAF016E7-084A-4E7A-9FB3-92A47D339357@python.org>
	<4A4E831A.6090404@v.loewis.de>
	<E233B5DE-56E8-4CC1-A0A5-2D65BA593868@python.org>
Message-ID: <ea2499da0907040120q7aadf731ne662012fca815057@mail.gmail.com>

On Sat, Jul 4, 2009 at 00:37, Barry Warsaw<barry at python.org> wrote:
> Doesn't Mercurial support an Subversion bridge? ?Would it be possible to
> provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6, and
> 3.1? ?If so, then the release managers would simply have to cut their
> releases from the svn copy instead of the hg master. ?/All/ other work would
> be done from the hg master. ?This shouldn't be too much of a burden since
> it's done so rarely and would end with the EOL of each of those branches.

There is some push support in hgsubversion, but it doesn't do tags at
this time, for example. I think this setup would be needlessly
complicated (and tools will need to learn the new revision specifier
anyway, so why not have it learn them sooner rather than later)?

Cheers,

Dirkjan

From tjreedy at udel.edu  Sat Jul  4 10:55:44 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 04 Jul 2009 04:55:44 -0400
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<h2lia7$v1j$1@ger.gmane.org>
	<bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com>
Message-ID: <h2n5ef$1qa$1@ger.gmane.org>

Brett Cannon wrote:

>     I would very much like the 'k' dropped from the py3 name. It was a
>     funny joke when py3 was vaporware, now it is excess baggage which
>     only puzzles non-insiders and newcomers.
> 
> 
> Is it really that confusing? I have never heard of anyone asking "what 
> is py3k?" 

Do you read python-list? It has been asked. Also, some people seem to 
think that py3k is different from python 3.

tjr


From martin at v.loewis.de  Sat Jul  4 11:28:09 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Jul 2009 11:28:09 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
	<87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4A4F20A9.5070200@v.loewis.de>

> I see that George Brandl and Martin van Loewis seem to be accomodating
> your viewpoint, but I don't get the impression that either you (as the
> hg migration proponent) nor they (as core committers) realize how far
> apart your assumptions are. 

Actually, I (probably) don't agree to a merge flow different from the
one that we currently have.

All I asked that *if* Dirkjan is in favor of a change, he should propose
it explicitly, and fleshed-out, in the PEP, so that people can tear it
down.

I know that several committers are unhappy with the current merge flow
(Georg in particular), so this is important to discuss.

Regards,
Martin

From ncoghlan at gmail.com  Sat Jul  4 12:36:20 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Jul 2009 20:36:20 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907040105m10e30faei90a29dadb305312@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com>	
	<4A4ECACA.5080508@v.loewis.de> <4A4EE510.7030409@gmail.com>
	<ea2499da0907040105m10e30faei90a29dadb305312@mail.gmail.com>
Message-ID: <4A4F30A4.5030405@gmail.com>

Dirkjan Ochtman wrote:
> On Sat, Jul 4, 2009 at 07:13, Nick Coghlan<ncoghlan at gmail.com> wrote:
>> I'd guess that the only way to keep those functional is to keep
>> svn.python.org around in read-only mode.
> 
> No, actually: the idea (I think I floated it in the PEP, as well), is
> that I can write a simple extension for hgweb that knows the mapping
> of SVN rev to hg rev and so can make
> hg.python.org/python/2.x/rev/r32542 come out to the changeset that
> resulted from converting that revision.

Ah, that's an excellent solution - good to hear it :)

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From p.f.moore at gmail.com  Sat Jul  4 14:48:13 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 4 Jul 2009 13:48:13 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
Message-ID: <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>

2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
> You can give me a bitbucket account so I can give you write access to the repo,
> There are tests as long as you install Nose.

How do I get the tests to work? Just running nosetests gives an error
(probably because pkgutil is being imported from the stdlib, rather
than from this directory).

If I set PYTHONPATH=. then I get errors. I suspect path normalisation
(for backslashes) in the zipfile handling.

Yes, it appears to be caused by the fact that Distribution uses
os.path.join to construct self.pkg_info_path (and self.record_path)
and yet zipfiles expect '/' separated pathnames.

Paul.

Error log:

>\Apps\Python26\Scripts\nosetests-2.6.exe -v
test_pkgutil.test_distributions ... ok
test_pkgutil.test_distribution ... FAIL
test_pkgutil.test_directory ... ok
test_pkgutil.test_get_file_users ... ok
test_pkgutil.test_egginfo_dirname ... ok
test_pkgutil.test_zipped_directory ... ERROR
test_pkgutil.test_zipped_distribution ... ERROR

======================================================================
ERROR: test_pkgutil.test_zipped_directory
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest
    self.test(*self.arg)
  File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 80, in
test_zipped_directory
    dir = ZippedDistributionDir(SITE_PKG+'.zip')
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__
    self.add(ZippedDistribution(self._zip_file, paths[0]))
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__
    super(ZippedDistribution, self).__init__(path)
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__
    pkginfo = self._open_pkginfo()
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo
    return self.zipfile.open(self.pkg_info_path)
  File "C:\Apps\Python26\lib\zipfile.py", line 859, in open
    zinfo = self.getinfo(name)
  File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named
'mercurial-1.0.1.egg-info\\\\PKG-INFO' in the archive"

======================================================================
ERROR: test_pkgutil.test_zipped_distribution
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 253, in setUp
    try_run(self.test, names)
  File "C:\Apps\Python26\Lib\site-packages\nose\util.py", line 487, in try_run
    return func()
  File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 90, in setup_zip
    pkgutil._dist_dirs.load(SITE_PKG+'.zip')
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 380, in load
    dist_dir = ZippedDistributionDir(path)
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__
    self.add(ZippedDistribution(self._zip_file, paths[0]))
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__
    super(ZippedDistribution, self).__init__(path)
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__
    pkginfo = self._open_pkginfo()
  File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo
    return self.zipfile.open(self.pkg_info_path)
  File "C:\Apps\Python26\lib\zipfile.py", line 859, in open
    zinfo = self.getinfo(name)
  File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named
'mercurial-1.0.1.egg-info\\\\PKG-INFO' in the archive"

======================================================================
FAIL: test_pkgutil.test_distribution
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest
    self.test(*self.arg)
  File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 60, in
test_distribution
    os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')])
AssertionError:
['C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\PKG_INFO',
'C:\\Users\\
Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\RECORD']
!= ['C:\\Users\\Gustav\\Data\\pep376\\site-
packages\\mercurial-1.0.1.egg-info/PKG_INFO',
'C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-
info/RECORD']

----------------------------------------------------------------------
Ran 7 tests in 0.019s

FAILED (errors=2, failures=1)

13:40 C:\Users\Gustav\Data\pep376
>unzip -l site-packages.zip
Archive:  site-packages.zip
  Length     EAs   ACLs    Date   Time    Name
 --------    ---   ----    ----   ----    ----
     2531      0      0  19/06/09 17:14   mercurial/filelog.py
     3449      0      0  19/06/09 17:14   mercurial/filelog.pyc
      234      0      0  19/06/09 17:14   mercurial-1.0.1.egg-info/PKG-INFO
      162      0      0  19/06/09 17:14   mercurial-1.0.1.egg-info/RECORD
        0      0      0  19/06/09 17:14   processing/__init__.py
     3482      0      0  19/06/09 17:14   processing-0.52.egg-info/PKG-INFO
      165      0      0  19/06/09 17:14   processing-0.52.egg-info/RECORD
 --------  -----  -----                   -------
    10023      0      0                   7 files

Paul.

From p.f.moore at gmail.com  Sat Jul  4 15:04:12 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 4 Jul 2009 14:04:12 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
Message-ID: <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com>

2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
> 2009/7/3 Paul Moore <p.f.moore at gmail.com>:
>> Does this sound sensible? Tarek, would you be OK with me attempting to
>> modify your prototype to support this protocol? Are there any tests
>> for PEP 376, so that I can confirm I haven't completely broken
>> something? If I can, I'll knock up some simple prototype importers for
>> non-standard examples, and see how they work with all this.
>
> Yes that's exactly what I was thinking about after the discussion we
> had in the other thread. This change would allow much more flexibility.

One important note - I plan on using the fact that DistributionDirMap
is not public, and hacking it about drastically, or possibly even
removing it. (After all, the likes of the load method don't make sense
when you've got sys.meta_path, sys.path_importer_cache and the like to
consider).

If the loss of any of the "internal" classes is an issue, say so now!

Paul.

From p.f.moore at gmail.com  Sat Jul  4 16:31:40 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 4 Jul 2009 15:31:40 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>
Message-ID: <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com>

2009/7/4 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
>> You can give me a bitbucket account so I can give you write access to the repo,
>> There are tests as long as you install Nose.
>
> How do I get the tests to work? Just running nosetests gives an error
> (probably because pkgutil is being imported from the stdlib, rather
> than from this directory).
>
> If I set PYTHONPATH=. then I get errors. I suspect path normalisation
> (for backslashes) in the zipfile handling.

Actually, the test

    assert_equals(list(dist.get_egginfo_files(local=True)),
                  [os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'),
                   os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')])

is broken, because the expected value uses slashes, which are *not*
the local separator on win32.

I've attached a patch.

But there's 2 comments I'd make (one minor, one major)

Minor one: The tests often seem to be exercising the internal classes,
not so much the public API, so many of them will probably not be of
much use to me :-(

Major one: I'm not entirely sure what the purpose of the "local" flag
is. With local=True, you are supposed to get a filesystem path in the
local form. But that makes no sense for non-filesystem based loaders
(e.g. zipfiles). So how is a user expected to use the value? With
local=false, you get slash-separated files (which is good, because
it's consistent) which are either absolute (see my earlier comments
about how broken this is in practice) or relative to the "directory"
containing the egg-info file (where this is of course not necessarily
a real filesystem location). But these aren't useful as filesystem
names, because they aren't local format.

So what is the point of the get_installed_files, uses, and
get_egginfo_files functions? Who will use the returned values, and
what for? Where will a user get a value to pass to uses()? Given that
get_egginfo_file takes names relative to the egginfo dir, shouldn't
get_egginfo_files *return* values relative to the same location, so
that they can be used there?

I guess the same type of argument applies with get_inetalled_files()
and uses() - the values returned by get_installed_files can be seen as
a set of opaque tokens (OK, realistically they will be "filenames" in
some sense) which can be passed to uses() - but if they aren't
absolute filesystem paths, you can't compare them. Example:

    distribution a contains file p.py
    distribution b contains file p.py

pkgutil.get_distribution(a).get_installed_files contains 'p.py'
pkgutil.get_distribution(b).uses('p.py') is True

What does this mean? If the 2 distributions are in the same sys.path
element, there's a clash. If not, is there a clash? What if they are
in 2 different directories? What if they are in the same directory,
but it's repeated under different names (e.g. symlinks) on sys.path?
What if one is in a directory and the other in a zip file?

This isn't even related to non-filesystem loaders. It's a fundamental
issue with the whole API. What you seem to want is actually a
canonical "name" for a file object - for filesystem files,
os.path.normcase(os.path.normpath(filename)) is probably good enough,
although it doesn't deal with symlinks. For other types of loader, you
have to rely on the loader itself - loader.get_fullname() is probably
OK. But even then, it's not clear how user code would actually *use*
these APIs.

I think you need some real-world use cases, with actual sample
(pseudo-)code, to validate the design here. As things stand, it's both
confusing and (I suspect) unusable in practice. Sorry, I know that
sounds negative, but if this isn't to be a source of subtle bugs for
years to come, it needs to be clarified now. PEP 302 is still hitting
this type of issue - runpy and importlib have brought out errors and
holes in the protocol quite recently - even though Just and I went to
great lengths to try to tease out hidden assumptions up front.

Concrete proposal:

get_metadata_files() - returns slash-separated names, relative to the
egginfo dir
get_metadata_file(path) - path must be slash-separated, relative to
the egginfo dir

get_installed_files - returns the contents of RECORD unaltered
uses(path) - checks if path is in RECORD

The latter 2 are not very useful in practice - you can't say anything
about entries in different RECORD files, which is likely the real use
case you want. Maybe RECORD could have an extra "Location" entry,
which determines where it exists globally (this would be the directory
to which the filenames were relative, in the case of filesystem-based
distributions) and RECORD entries are comparable if the Location
values in the 2 RECORD files match. That's a lot more complex - but
depending on what use people expect to make of these 2 APIs, it may be
justified.

Paul.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-win32.patch
Type: text/x-patch
Size: 666 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090704/b82f38ce/attachment-0001.bin>

From brett at python.org  Sat Jul  4 21:28:31 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Jul 2009 12:28:31 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907040103x5df7a15bh8f1dcb86c9adbaa0@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com> 
	<ea2499da0907040103x5df7a15bh8f1dcb86c9adbaa0@mail.gmail.com>
Message-ID: <bbaeab100907041228l7ac37389s10cbf9ab0d9267a0@mail.gmail.com>

On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:

> On Fri, Jul 3, 2009 at 20:04, Brett Cannon<brett at python.org> wrote:
> > Fine by me as long as people realize that if anything is questionable
> then
> > the switch will not happen. Getting this right takes precedence over any
> > deadline. And obviously we will need to do at least one live conversion
> on
> > python.org hardware to make sure everything will work smoothly.
>
> I'm not sure I see the need to do a (live? what does that mean in this
> context) on python.org hardware.


"Live" as in run through all the steps on python.org hardware w/o hitting
any final switch that turns off svn.


> Why exactly is that better than me
> doing it on one of my boxes, as long as all the necessary tools and an
> idea of how to do it are publically available (from the pymigr repo,
> for example)?


Because there are different OSs, installed software, etc. Basically because
you just never know. =) Plus it will make Martin sleep better.


>
>
> > And will make the idea of splitting out the standard library and tests a
> > reasonable thing to do.
>
> In due time, yes.
>
> > While I really like the idea of using named branches for each release so
> > that there is a single py3k branch that contains all relevant history for
> > every release, I think we should start simple and go with cloned
> branches.
> > That way the workflow does not radically shift from what we do now for
> svn
> > to start. Once the conversion is done and people are comfortable with hg
> we
> > can then discuss moving towards a named branch approach.
>
> I don't think the cloned branches is much simpler than the named
> branches approach, in several ways. For example, populating the branch
> part of a sys.whatever value is significantly harder. Also, if you
> follow a useful tagging approach, doing cloned branches means that
> release tags aren't available on trunk/main/default. That seems like a
> step backwards.
>

I personally prefer named branches, but that's just me and I am not about to
force my preferences on everyone. Guess we just have to see if others have
opinions against named branches.


>
> > Sounds reasonable to me. We can just make a list and send it to
> > python-committers to make final decisions of what should stick around.
>
> This list exists and has been referenced in my email a few times.
>

Sure, but as inlining the PEP in this email thread has shown, not making
people click a link helps. =) Plus a separate email makes it very obvious
that people need to check their email instead of making it a bullet point in
a much larger email.


>
> > I don't use tags so I don't really care, but in the name of easy
> transition
> > I say we don't change the naming scheme (although I have no issue
> dropping
> > obscure tags).
>
> The current proposal is to clean up old tags to agree with the current
> naming scheme (and dropping obscure and partial tags).
>

Fine by me.


>
> > Something else that can go out to python-committers before the switch.
>
> This should just be done ASAP, it helps with a smooth conversion process.
>
> > I don't think there is a single project we host -- all two of them --
> that
> > have not said they want to convert. So I say convert everything and let's
> > turn off the svn server by the end of the year.
>
> I say we tackle each one as we go. I say doing a good conversion job
> is valuable, and we should take as much time as we need (though not
> more). You advocate something similar below for the Python conversion.
>

I am not suggesting we do all conversions on the same day, just that
everything should eventually be converted.


>
> > Can we check these scripts into the repository itself? That way there is
> a
> > chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for
> > ``make patchcheck``.
>
> I'm not sure there's an easy way to make them into commands (although
> I guess you could make an extension to that effect), but hooks would
> be very easy.
>

OK. I was just hoping we could factor the code in such a way as to share the
basic steps the hooks were checking so as to reuse them in a command.


>
> > How about hg.python.org for the official branches and we keep
> > code.python.org for personal branches of the developers like we have
> done
> > with the bzr experiments?
>
> I think that's just confusing. Most people seem to like hg.python.org,
> and it's obvious that hg goes to hg.p.o. Dividing up the namespace
> only makes it harder to find things.
>

Yeah, I realize I have lost this battle.


>
> > As I have said, we should change our workflow habits after the switch and
> > people are comfortable with hg.
>
> Well, not everyone agrees, and although I think it doesn't matter much
> for the conversion itself, it may affect the branching strategy
> discussion.
>

Sure, to an extent.


>
> (sys.revision discussion elided because some others have already been
> bikeshedding on it.)


I think it has been answered; sys.subversion goes away and sys.mercurial or
sys.mercurial_revision comes into existence.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090704/9b67c5d7/attachment.htm>

From brett at python.org  Sat Jul  4 21:29:45 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Jul 2009 12:29:45 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2n5ef$1qa$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<h2lia7$v1j$1@ger.gmane.org>
	<bbaeab100907031441h3cd3b016hbdd61d64be42ae9@mail.gmail.com> 
	<h2n5ef$1qa$1@ger.gmane.org>
Message-ID: <bbaeab100907041229y7a68d1des8bf1b1c2e76a20be@mail.gmail.com>

On Sat, Jul 4, 2009 at 01:55, Terry Reedy <tjreedy at udel.edu> wrote:

> Brett Cannon wrote:
>
>     I would very much like the 'k' dropped from the py3 name. It was a
>>    funny joke when py3 was vaporware, now it is excess baggage which
>>    only puzzles non-insiders and newcomers.
>>
>>
>> Is it really that confusing? I have never heard of anyone asking "what is
>> py3k?"
>>
>
> Do you read python-list?


No as it would take up so much of my Python time I wouldn't be able to code
anymore.


> It has been asked. Also, some people seem to think that py3k is different
> from python 3.
>

Well, I still would not like to lose the py3k label. But if we do I still
say 2.x/3.x instead of py2/py3.

-Brett



>
> tjr
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090704/ed91a3e3/attachment.htm>

From brett at python.org  Sat Jul  4 21:54:58 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Jul 2009 12:54:58 -0700
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A4EA689.1010704@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> 
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> 
	<4A4EA689.1010704@gmail.com>
Message-ID: <bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>

On Fri, Jul 3, 2009 at 17:47, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Tarek Ziad? wrote:
> > 2009/7/3 Paul Moore <p.f.moore at gmail.com>:
> >> Does this sound sensible? Tarek, would you be OK with me attempting to
> >> modify your prototype to support this protocol? Are there any tests
> >> for PEP 376, so that I can confirm I haven't completely broken
> >> something? If I can, I'll knock up some simple prototype importers for
> >> non-standard examples, and see how they work with all this.
> >
> > Yes that's exactly what I was thinking about after the discussion we
> > had in the other
> > thread. This change would allow much more flexibility.
>
> It's also very similar to the ideas that started to tick over in the
> back of my head after the previous discussion (only far more fleshed out
> that anything I got to - fortunately I read Paul's message before
> spending too much time on it!)
>
> Specifically regarding this comment from Paul:
>
> > Finder.uninstall(distribution_name, filter=callable, installer=name)
> > Uninstall the given distribution. It's likely that many finders will
> > be read-only. In that case, this function should return None.
> > Otherwise, return a list of the "files" removed. (This may need some
> > clarification, as many finders won't have a concept of a "file name").
>
> PEP 302 finders and loaders actually *are* expected to have a concept of
> a "file name". For Python modules they they are expected to have it so
> they can populate __file__ correctly


Not only that, lots of code out there expects values from __path__ to
contain os.path.sep on top of __file__, so the idea of file paths is already
hard-coded into how import works, whether we like it or not.


> (runpy is also a lot happier with
> them when they expose get_filename(), a relatively recent addition to
> the PEP 302 API).


An addition I was not even aware of. Looks like importlib needs a little
updating.


> For other resources they're expected to have it so the
> get_data() API can work correctly.
>
> One other thing to consider: it may be desirable to have hooks for
> meta_path and path_hooks in distutils that are checked *before* the full
> import hooks in the sys module. The reason I say this is that it would
> allow someone to override just the metadata retrieval (e.g. to pick up
> the extra dependency information saved by setuptools and friends)
> without having to worry about breaking the actual imports from those
> locations.


So as the guy who plans to try to clean up import, I just want to say that
implicit finders will eventually go away. Do not start off making them
implicit, but you can have them installed on sys.meta_path or sys.path_hooks
by default.


>
>
> Cheers,
> Nick.
>
> P.S. +lots on using 'metadata' in the PEP 376 method names rather than
> the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
> gratuitous in this case.


Ditto from here. Plus I have an aversion to terminology that goes down the
reptile route instead of the Monty Python route.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090704/8a1d1edc/attachment-0001.htm>

From rdmurray at bitdance.com  Sat Jul  4 23:05:56 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 4 Jul 2009 17:05:56 -0400 (EDT)
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907041228l7ac37389s10cbf9ab0d9267a0@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com> 
	<ea2499da0907040103x5df7a15bh8f1dcb86c9adbaa0@mail.gmail.com>
	<bbaeab100907041228l7ac37389s10cbf9ab0d9267a0@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0907041700040.6306@kimball.webabinitio.net>

On Sat, 4 Jul 2009 at 12:28, Brett Cannon wrote:
> On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
>
>> On Fri, Jul 3, 2009 at 20:04, Brett Cannon<brett at python.org> wrote:
>>> Fine by me as long as people realize that if anything is questionable
>> then
>>> the switch will not happen. Getting this right takes precedence over any
>>> deadline. And obviously we will need to do at least one live conversion
>> on
>>> python.org hardware to make sure everything will work smoothly.
>>
>> I'm not sure I see the need to do a (live? what does that mean in this
>> context) on python.org hardware.
>
>
> "Live" as in run through all the steps on python.org hardware w/o hitting
> any final switch that turns off svn.

"Live" should also (and I think this is critical) include a period of at
least a week where we tell python-dev/python-committers "hg.python.org
looks like it is going to look at cutover, please try out the workflow".
The idea would be to have committers actually exercise the workflow on
their platform for at least one patch, including whatever replaces
the svnmerge step.

I can almost guarantee we will find some issues that need to be resolved,
and that people won't try it out thoroughly until it is "live".

--David

From p.f.moore at gmail.com  Sun Jul  5 00:13:14 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 4 Jul 2009 23:13:14 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
Message-ID: <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com>

2009/7/4 Brett Cannon <brett at python.org>:
>> > Finder.uninstall(distribution_name, filter=callable, installer=name)
>> > Uninstall the given distribution. It's likely that many finders will
>> > be read-only. In that case, this function should return None.
>> > Otherwise, return a list of the "files" removed. (This may need some
>> > clarification, as many finders won't have a concept of a "file name").
>>
>> PEP 302 finders and loaders actually *are* expected to have a concept of
>> a "file name". For Python modules they they are expected to have it so
>> they can populate __file__ correctly
>
> Not only that, lots of code out there expects values from __path__ to
> contain os.path.sep on top of __file__, so the idea of file paths is already
> hard-coded into how import works, whether we like it or not.

Yes, consider me duly reprimanded :-) They aren't *file* names, but
they still have to look similar enough to satisfy code that uses
things like filename manipulation functions on them.

>> (runpy is also a lot happier with
>> them when they expose get_filename(), a relatively recent addition to
>> the PEP 302 API).
>
> An addition I was not even aware of. Looks like importlib needs a little
> updating.

Yes, I didn't notice that sneak into PEP 302, either :-)

>> For other resources they're expected to have it so the
>> get_data() API can work correctly.
>>
>> One other thing to consider: it may be desirable to have hooks for
>> meta_path and path_hooks in distutils that are checked *before* the full
>> import hooks in the sys module. The reason I say this is that it would
>> allow someone to override just the metadata retrieval (e.g. to pick up
>> the extra dependency information saved by setuptools and friends)
>> without having to worry about breaking the actual imports from those
>> locations.
>
> So as the guy who plans to try to clean up import, I just want to say that
> implicit finders will eventually go away. Do not start off making them
> implicit, but you can have them installed on sys.meta_path or sys.path_hooks
> by default.

I'm against having distutils-specific hooks. I don't see a genuine
need, and it adds complexity so let's wait for a proper use case
before adding that.

With some suitably ugly hacking, you could probably add a finder to
sys.meta_path which never actually found a module, but which reported
one or more distributions for the distutils metadata APIs. Playing
around with that idea could likely give you any flexibility you needed
(at the hopefully minor cost of an extra dummy meta_path call on each
import).

As far as the currently implicit filesystem path handling is
concerned, I'm aware that the intention is for importlib to move these
into an explicit importer at some point. For my prototype changes to
Tarek's prototype, I'm creating a dummy filesystem finder with just
the new methods needed for the distribution metadata. It will be easy
to move these into a real filesystem finder when one emerges.

Paul.

From ncoghlan at gmail.com  Sun Jul  5 00:42:49 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 05 Jul 2009 08:42:49 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>	
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>	
	<4A4EA689.1010704@gmail.com>	
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com>
Message-ID: <4A4FDAE9.20205@gmail.com>

Paul Moore wrote:
> 2009/7/4 Brett Cannon <brett at python.org>:
>>> (runpy is also a lot happier with
>>> them when they expose get_filename(), a relatively recent addition to
>>> the PEP 302 API).
>> An addition I was not even aware of. Looks like importlib needs a little
>> updating.
> 
> Yes, I didn't notice that sneak into PEP 302, either :-)

I probably should have been noisier about that when I added it. I'm
pretty sure it did come up on this list, but it would have been
somewhere in the middle of a runpy discussion: runpy was stuck because
it only uses PEP 302 to *find* the modules it needs, but not to actually
load them. In the original version of PEP 302 the only way to get a
loader to tell you the filename was to load the module and see what it
set __file__ to, which wasn't useful in the runpy case.

If I recall correctly, at the time when PJE was rationalising the code
duplication between runpy and pkglib the comment was also made that
runpy's get_filename() optional loader extension should be mentioned in
PEP 302. runpy still works for loaders that don't provide it, it just
can't set __file__ or sys.argv[0] correctly in those cases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brett at python.org  Sun Jul  5 00:56:02 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Jul 2009 15:56:02 -0700
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A4FDAE9.20205@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com>
	<4A4FDAE9.20205@gmail.com>
Message-ID: <bbaeab100907041556ua174d9bj54e18781e4d493e6@mail.gmail.com>

I'm not complaining, just saying I didn't notice. I essentially did the same
with get_source/get_bytecode so as to be able to set __file__.

When I get around to it I will create a RunpyLoader in importlib.abc and add
the method in the proper machinery loaders.

On Jul 4, 2009 3:43 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

Paul Moore wrote: > 2009/7/4 Brett Cannon <brett at python.org>:

>>> (runpy is also a lot happier with >>> them when they expose
get_filename(), a relatively recent ...
I probably should have been noisier about that when I added it. I'm
pretty sure it did come up on this list, but it would have been
somewhere in the middle of a runpy discussion: runpy was stuck because
it only uses PEP 302 to *find* the modules it needs, but not to actually
load them. In the original version of PEP 302 the only way to get a
loader to tell you the filename was to load the module and see what it
set __file__ to, which wasn't useful in the runpy case.

If I recall correctly, at the time when PJE was rationalising the code
duplication between runpy and pkglib the comment was also made that
runpy's get_filename() optional loader extension should be mentioned in
PEP 302. runpy still works for loaders that don't provide it, it just
can't set __file__ or sys.argv[0] correctly in those cases.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090704/012ec5b8/attachment.htm>

From g.brandl at gmx.net  Sun Jul  5 13:10:14 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 05 Jul 2009 13:10:14 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E7DA7.2030402@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>	<4A4E7173.1040507@v.loewis.de>	<bbaeab100907031439h734f02c2y72cdef8df801eec6@mail.gmail.com>
	<4A4E7DA7.2030402@v.loewis.de>
Message-ID: <h2q1nd$og7$1@ger.gmane.org>

Martin v. L?wis schrieb:
>> We could add another value in the tuple that specifies the VCS:
>> ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that
>> VCSs are not universally the same, but the concept of a revision is
>> universal.
> 
> Actually, I think that's not the case. For bzr, the usual way of
> identifying a revision is by revision number, which, however, is not
> unique within a project, as each branch will use contiguous integers
> for numbers. There are also unique identifications - so a bzr revision
> has actually two numbers.
> 
> More general, in a DVCS, it is not possible to access the revision being
> referred to by such a tuple. For sys.subversion, if [0]=='CPython', then
> you could go to svn.python.org. For a DVCS, the revision being
> identified may not be publically available, or may not live on a host
> that you can infer from your proposed sys.revision.

At least you can tell that if the given hash is not present in the mainline
repo, the build contains something that doesn't come from python.org.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Sun Jul  5 13:17:31 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 05 Jul 2009 13:17:31 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4E6EF5.4000901@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
	<4A4E6EF5.4000901@v.loewis.de>
Message-ID: <h2q252$rpd$1@ger.gmane.org>

Martin v. L?wis schrieb:
>> It will handle it, for sure, but I think it would all go easier if we
>> could work with stricter subset branches (and leave the effective
>> cherrypicking for the occasional problem).
> 
> So I think the PEP should propose a workflow (or: merge flow) if you
> think we would be better off with a different one.
> 
> In proposing such a workflow, consider these requirements:
> - we current have four active "maintenance" branches (i.e. where
>   the entire code basis evolves): trunk, 3k, 2.6, and 3.1 (3.0
>   also until this morning).

It seems that there is a consensus to separate the 2.x and 3.x repos,
and that also makes sense to me.

Using named branches for the maintenance branches should be possible,
but I'm not opposed to using cloned branches either.  What I really
want to see is the common-subset approach for maintenance branches.

Changesets still have to be transplanted from 2.x to 3.x or the other
way round.

> - in addition, we have two security branches currently: 2.4 and
>   2.5, although 2.4 will be closed soon.
> - our committers consistently refuse to merge changes across
>   branches themselves, and likely continue to do so unless there
>   is some feature of hg that I missed (e.g. one were merging
>   would happen without any user specifically asking for it)

If the checkin is done in the proper (the maint) branch, at least merging
of that change is automatic whenever someone does a hg merge.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ziade.tarek at gmail.com  Sun Jul  5 15:02:49 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 15:02:49 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>
	<79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com>
Message-ID: <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com>

2009/7/4 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/4 Paul Moore <p.f.moore at gmail.com>:
>> 2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
>>> You can give me a bitbucket account so I can give you write access to the repo,
>>> There are tests as long as you install Nose.
>>
>> How do I get the tests to work? Just running nosetests gives an error
>> (probably because pkgutil is being imported from the stdlib, rather
>> than from this directory).
>>

I just run them from within the directory

>> If I set PYTHONPATH=. then I get errors. I suspect path normalisation
>> (for backslashes) in the zipfile handling.

>
> Actually, the test
>
> ? ?assert_equals(list(dist.get_egginfo_files(local=True)),
> ? ? ? ? ? ? ? ? ?[os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'),
> ? ? ? ? ? ? ? ? ? os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')])
>
> is broken, because the expected value uses slashes, which are *not*
> the local separator on win32.
>
> I've attached a patch.

Applied, thanks (I didn't run them under win32 yet)


>
> But there's 2 comments I'd make (one minor, one major)
>
> Minor one: The tests often seem to be exercising the internal classes,
> not so much the public API, so many of them will probably not be of
> much use to me :-(

I'll add some more tests then, or even user stories.

> I think you need some real-world use cases, with actual sample
> (pseudo-)code, to validate the design here. As things stand, it's both
> confusing and (I suspect) unusable in practice. Sorry, I know that
> sounds negative, but if this isn't to be a source of subtle bugs for
> years to come, it needs to be clarified now. PEP 302 is still hitting
> this type of issue - runpy and importlib have brought out errors and
> holes in the protocol quite recently - even though Just and I went to
> great lengths to try to tease out hidden assumptions up front.

Agreed, the zip case was added afterwards, but in practice, the APIs are still
dealing with the files are *filesystem files* located in a container
(eg a directory or a zip file)
located somewhere on the filesystem.

"local" in that case is a flag that means "translate a file path
expressed in the local filesystem"
which make no sense anymore with zip files. But the goal really, is to
be able to point out
that two distributions are using the very same file.

Right now PEP 376 and the prototype code handle these two real world use cases:

- browsing regular site-packages-like directories
- browsing site-packages-like directories, that are zipped.

For example:

- I have a  "packages.zip" file in /var/, wich is also in my sys.path.
It contains a distribution "foo-1.0" that has the "roman.py" file in
its root.  So the RECORD file located in "foo-1.0.egg-info" has a line
starting with "roman.py,..."

- Then if I install docutils 0.5 as a regular filesystem distribution,
"roman.py" will be added in Python's site-packages.
  and docutils-0.5.egg-info/RECORD will contain "roman.py,..." with
the same hash.

The local flag will return these paths:

- /var/packages.zip/roman.py   <--- not a "real" path
- /usr/local/lib/python2.6/site-packages/roman.py

So removing the docutils distribution will be doable, because these
paths are different.

>
> Concrete proposal:
>
> get_metadata_files() - returns slash-separated names, relative to the
> egginfo dir
> get_metadata_file(path) - path must be slash-separated, relative to
> the egginfo dir
>
> get_installed_files - returns the contents of RECORD unaltered
> uses(path) - checks if path is in RECORD
>
> The latter 2 are not very useful in practice - you can't say anything
> about entries in different RECORD files, which is likely the real use
> case you want. Maybe RECORD could have an extra "Location" entry,
> which determines where it exists globally (this would be the directory
> to which the filenames were relative, in the case of filesystem-based
> distributions) and RECORD entries are comparable if the Location
> values in the 2 RECORD files match. That's a lot more complex - but
> depending on what use people expect to make of these 2 APIs, it may be
> justified.

Yes,
In practice, if you look at my previous example, even if
"/var/packages.zip/roman.py" isn't a
real path, it's enough to compare RECORD entries globally.

The "Location" entry you are proposing in that case, would be
"/var/packages.zip".

But do we really need to store it the RECORD  ? Or can't we define an
API that returns
two elements :

- the path to the location (in the example: /var/packages.zip or
/usr/local/lib/python2.6/site-packages)
- the path within the location itself (in the example: roman.py)

A concrete proposal would be to take back your proposal, but return
tuples with the location as the first member.
e.g. "(location, relative path[s])"

The code that is comparing paths to see if they are the same can join
location+relative path[s], while
we can provide in a dedicated function something to read the content
of the file (that would be get_data I guess,
if I refer to PEP 302)

Tarek

From ziade.tarek at gmail.com  Sun Jul  5 15:05:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 15:05:10 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com>
Message-ID: <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com>

On Sat, Jul 4, 2009 at 3:04 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/3 Tarek Ziad? <ziade.tarek at gmail.com>:
>> 2009/7/3 Paul Moore <p.f.moore at gmail.com>:
>>> Does this sound sensible? Tarek, would you be OK with me attempting to
>>> modify your prototype to support this protocol? Are there any tests
>>> for PEP 376, so that I can confirm I haven't completely broken
>>> something? If I can, I'll knock up some simple prototype importers for
>>> non-standard examples, and see how they work with all this.
>>
>> Yes that's exactly what I was thinking about after the discussion we
>> had in the other thread. This change would allow much more flexibility.
>
> One important note - I plan on using the fact that DistributionDirMap
> is not public, and hacking it about drastically, or possibly even
> removing it. (After all, the likes of the load method don't make sense
> when you've got sys.meta_path, sys.path_importer_cache and the like to
> consider).
>
> If the loss of any of the "internal" classes is an issue, say so now!

Go for it please, and let me know if you set a bitbucket account, so
you can push your commits in there directly


>
> Paul.
>



-- 
Tarek Ziad? | http://ziade.org

From p.f.moore at gmail.com  Sun Jul  5 15:13:02 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 14:13:02 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com>
	<94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com>
Message-ID: <79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com>

2009/7/5 Tarek Ziad? <ziade.tarek at gmail.com>:
> Go for it please, and let me know if you set a bitbucket account, so
> you can push your commits in there directly

My bitbucket account is 'pmoore'.

Paul.

From ziade.tarek at gmail.com  Sun Jul  5 15:13:13 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 15:13:13 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
Message-ID: <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>

2009/7/4 Brett Cannon <brett at python.org>:
>>
>> P.S. +lots on using 'metadata' in the PEP 376 method names rather than
>> the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
>> gratuitous in this case.
>
> Ditto from here. Plus I have an aversion to terminology that goes down the
> reptile route instead of the Monty Python route.

If it turns out that we use PEP 302-like loaders, I am also suggesting
that the default metadata directory name
used in Distutils is changed to "DIST_NAME.metadata".

The loader would still work with "DIST_NAME.egg-info" directories for
compatibility with
existing format in the query APIs, but the Distutils install command
would rather create  "DIST_NAME.metadata"

From ziade.tarek at gmail.com  Sun Jul  5 15:14:04 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 15:14:04 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com>
	<94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com>
	<79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com>
Message-ID: <94bdd2610907050614r568c8ca9qa681b03e3a4fcb8b@mail.gmail.com>

On Sun, Jul 5, 2009 at 3:13 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/5 Tarek Ziad? <ziade.tarek at gmail.com>:
>> Go for it please, and let me know if you set a bitbucket account, so
>> you can push your commits in there directly
>
> My bitbucket account is 'pmoore'.

You're set.

From p.f.moore at gmail.com  Sun Jul  5 15:27:31 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 14:27:31 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>
	<79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com>
	<94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com>
Message-ID: <79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com>

2009/7/5 Tarek Ziad? <ziade.tarek at gmail.com>:
> Agreed, the zip case was added afterwards, but in practice, the APIs are still
> dealing with the files are *filesystem files* located in a container (eg a directory
> or a zip file) located somewhere on the filesystem.
>
> "local" in that case is a flag that means "translate a file path expressed in the
> local filesystem" which make no sense anymore with zip files. But the goal really,
> is to be able to point out that two distributions are using the very same file.
>
> Right now PEP 376 and the prototype code handle these two real world use cases:
>
> - browsing regular site-packages-like directories
> - browsing site-packages-like directories, that are zipped.
>
> For example:
>
> - I have a ?"packages.zip" file in /var/, wich is also in my sys.path.
> It contains a distribution "foo-1.0" that has the "roman.py" file in
> its root. ?So the RECORD file located in "foo-1.0.egg-info" has a line
> starting with "roman.py,..."
>
> - Then if I install docutils 0.5 as a regular filesystem distribution,
> "roman.py" will be added in Python's site-packages.
> ?and docutils-0.5.egg-info/RECORD will contain "roman.py,..." with
> the same hash.
>
> The local flag will return these paths:
>
> - /var/packages.zip/roman.py ? <--- not a "real" path
> - /usr/local/lib/python2.6/site-packages/roman.py
>
> So removing the docutils distribution will be doable, because these
> paths are different.
>
>>
>> Concrete proposal:
>>
>> get_metadata_files() - returns slash-separated names, relative to the
>> egginfo dir
>> get_metadata_file(path) - path must be slash-separated, relative to
>> the egginfo dir
>>
>> get_installed_files - returns the contents of RECORD unaltered
>> uses(path) - checks if path is in RECORD
>>
>> The latter 2 are not very useful in practice - you can't say anything
>> about entries in different RECORD files, which is likely the real use
>> case you want. Maybe RECORD could have an extra "Location" entry,
>> which determines where it exists globally (this would be the directory
>> to which the filenames were relative, in the case of filesystem-based
>> distributions) and RECORD entries are comparable if the Location
>> values in the 2 RECORD files match. That's a lot more complex - but
>> depending on what use people expect to make of these 2 APIs, it may be
>> justified.
>
> Yes,
> In practice, if you look at my previous example, even if
> "/var/packages.zip/roman.py" isn't a
> real path, it's enough to compare RECORD entries globally.
>
> The "Location" entry you are proposing in that case, would be
> "/var/packages.zip".
>
> But do we really need to store it the RECORD ?? Or can't we define an
> API that returns
> two elements :
>
> - the path to the location (in the example: /var/packages.zip or
> /usr/local/lib/python2.6/site-packages)
> - the path within the location itself (in the example: roman.py)
>
> A concrete proposal would be to take back your proposal, but return
> tuples with the location as the first member.
> e.g. "(location, relative path[s])"

That sounds reasonable. So we can forget the "local" parameter, and
return a tuple:

- absolute location of the container (directory, zipfile or whatever
containing the egginfo file) as a filesystem path in canonical native
form (where it's filesystem based) or as an opaque token for the odd
cases (frozen modules, for example) where a filesystem location isn't
available.
- entry from the RECORD file, as a slash-separated filename relative
to the root of the container.

> The code that is comparing paths to see if they are the same can join
> location+relative path[s], while we can provide in a dedicated function
> something to read the content of the file (that would be get_data I guess,
> if I refer to PEP 302)

Unfortunately, get_data loads data files located within a *package*,
using a name relative to the package directory. You can't get at the
metadata of a *distribution* like that.

But if you're using get_installed_files(), why would you then want to
read the files? What exactly would you *use* get_installed_files for
which would then leave you needing to read the files? If it's to check
they haven't changed (by comparing md5 values) you're doing that to
uninstall, so that's the responsibility of the uninstall function.

Again, it's a question of what is a public API, and what is the use
case it's designed for.

I'm currently writing a SQLite importer, which will allow me to store
"files" in any sort of database tables I want, so I can build in some
nice pathological behaviour. That should tease out some awkward corner
cases :-)

Paul

From ziade.tarek at gmail.com  Sun Jul  5 15:48:50 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 15:48:50 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com>
	<79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com>
	<94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com>
	<79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com>
Message-ID: <94bdd2610907050648sd5335aqaa3466b5fb9d05cc@mail.gmail.com>

On Sun, Jul 5, 2009 at 3:27 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>>
>> A concrete proposal would be to take back your proposal, but return
>> tuples with the location as the first member.
>> e.g. "(location, relative path[s])"
>
> That sounds reasonable. So we can forget the "local" parameter, and
> return a tuple:
>
> - absolute location of the container (directory, zipfile or whatever
> containing the egginfo file) as a filesystem path in canonical native
> form (where it's filesystem based) or as an opaque token for the odd
> cases (frozen modules, for example) where a filesystem location isn't
> available.
> - entry from the RECORD file, as a slash-separated filename relative
> to the root of the container.

exactly,

>
> But if you're using get_installed_files(), why would you then want to
> read the files? What exactly would you *use* get_installed_files for
> which would then leave you needing to read the files? If it's to check
> they haven't changed (by comparing md5 values) you're doing that to
> uninstall, so that's the responsibility of the uninstall function.
>
> Again, it's a question of what is a public API, and what is the use
> case it's designed for.

Right. These APIs were created for third-party package managers.
One use case of a package manager is the uninstallation, but I have no other
use case in mind.

>
> I'm currently writing a SQLite importer, which will allow me to store
> "files" in any sort of database tables I want, so I can build in some
> nice pathological behaviour. That should tease out some awkward corner
> cases :-)

Sounds good.

Semi-related: even if the files themselves are in the filesystem,
having a sqlite db to
index the list of installed distributions makes a good cache solution
to reduce the disk I/O
and speed up the query functions.

So maybe we could use a disk-based cache for site-packages-like
directories in the
form of a sqlite db. That's what I am experimenting on my side.


>
> Paul
>



-- 
Tarek Ziad? | http://ziade.org

From stephen at xemacs.org  Sun Jul  5 16:42:23 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 05 Jul 2009 23:42:23 +0900
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2q252$rpd$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>
	<4A4E6EF5.4000901@v.loewis.de> <h2q252$rpd$1@ger.gmane.org>
Message-ID: <87hbxrur74.fsf@uwakimon.sk.tsukuba.ac.jp>

Georg Brandl writes:

 > What I really want to see is the common-subset approach for
 > maintenance branches.

IMO, this unfortunately unlikely to work as you seem to expect, for a
two technical reasons and for a social reason.  The first technical
reason is that a maintenance branch really is a branch, not a subset.
The fix that is appropriate for a maintenance branch is often
inappropriate in detail for the mainline, and vice versa.  But
Mercurial doesn't care about "in detail" vs. "in design"; both will
result in a conflict the first time that branch is merged.

Second, some fixes for the maintenance branch will simply not be
appropriate for the development branch, as the problem has already
been fixed "en passant" by some other change.  This can probably be
handled by doing what git calls an "ours" merge to make it look like
the unnecessary patch is an ancestor of the tip, even though no code
was actually applied to the mainline.  However, this kind of operation
is some what delicate, and even if it's mostly scripted, it's likely
to be somewhat unreliable for people who don't use the script very
often ... which leads to the social problem:

 > > - our committers consistently refuse to merge changes across
 > >   branches themselves, and likely continue to do so unless there
 > >   is some feature of hg that I missed (e.g. one were merging
 > >   would happen without any user specifically asking for it)
 > 
 > If the checkin is done in the proper (the maint) branch, at least merging
 > of that change is automatic whenever someone does a hg merge.

Maybe.  But I see two problematic sides to this from the social point
of view, which is the same old problem really.

First, to the extent that it doesn't run into the technical problems,
it encourages people to *not* review patches for each branch they are
committed to.  "It will get automerged anyway."  Anything that
discourages review is a bad thing.  It will cause the development
branch to "age" faster because of accumulation of crufty patches that
are good enough as minimally invasive fixes for bugs in a maintenance
branch, but which should be more robust for the development branch.

I think you will also find that some people are not particularly
interested in fixing the maintenance branch for some of their patches
for exactly the same reasons they currently don't, and they will
continue to refuse to do the work to commit in the maintenance branch
first.  Especially after the first time they run into one of the
technical problems described above.

In the end, any policy to encourage a "subset" policy is likely to end
up as a burden on the same people who currently do the cross branch
merging.



From p.f.moore at gmail.com  Sun Jul  5 18:26:07 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 17:26:07 +0100
Subject: [Python-Dev] PEP 376 - get_egginfo_files
Message-ID: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>

The PEP says:

"""
get_egginfo_files(local=False) -> iterator of paths

Iterates over the RECORD entries and return paths for each line if the
path is pointing a file located in the .egg-info directory or one of
its subdirectory.
"""

Should this method really only return filenames noted in the RECORD
file? Would it not be better for it to iterate over *all* files in the
.egg-info directory? I understand that there shouldn't, in practice,
be any files in that directory *not* mentioned in the RECORD file, but
given that we already have get_installed_files to read the RECORD
file, I would imagine it's better for this file to so something more
than filter the return values from get_installed_files.

Actually, on that note, consider the pkgutil functions:

def get_distribution(name):
    for d in get_distributions():
        if d.name == name:
            return d
    return None

def get_file_users(path):
    for d in get_distributions():
        if d.uses(path):
            yield d

These don't actually add much to the API. While I can see the
advantage of having them as convenience methods, it might be worth
pointing out in the PEP that that's all they are.

Similarly, how valuable is Distribution.name, given that it's the same
as Distribution.metadata.name? I'm probably just going to make it a
property -

@property
def name(self):
    return self.metadata.name

but that's actually slower than just using self.metadata.name
directly, so it's a bit of an attractive nuisance, and I'd prefer it
if it wasn't present. (For the PEP 302 stuff, I'm making metadata a
cached property, so name *has* to be a property to ensure that the
metadata cache is managed properly...)

Paul.

From jpellerin at gmail.com  Sun Jul  5 19:46:40 2009
From: jpellerin at gmail.com (jason pellerin)
Date: Sun, 5 Jul 2009 13:46:40 -0400
Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418)
Message-ID: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com>

Bringing python-dev into the discussion at Barry's request. The
summary is that a recent change to unittest.TestProgram breaks nose by
moving self.testRunner initialization from it's old home in
TestProgram.runTests to TestProgram.__init__. The very small patch
attached to the ticket moves it back to runTests.

Here's the ticket: http://bugs.python.org/issue6418
And a link to the testing in python list discussion:

http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html

JP (primary author of nose)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 831 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090705/ebbf7c1e/attachment.pgp>

From martin at v.loewis.de  Sun Jul  5 20:00:14 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Jul 2009 20:00:14 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <h2q252$rpd$1@ger.gmane.org>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>	<4A4E6EF5.4000901@v.loewis.de>
	<h2q252$rpd$1@ger.gmane.org>
Message-ID: <4A50EA2E.8010401@v.loewis.de>

>> In proposing such a workflow, consider these requirements:
> 
> It seems that there is a consensus to separate the 2.x and 3.x repos,
> and that also makes sense to me.

(I think) I wasn't primarily talking about the representation of
branches in hg - to that, I fully trust recommendations from hg users
and experts.

What will need debate and discussion in the PEP is the workflow, ie.
the order in which changes should be applied to the branches.

> If the checkin is done in the proper (the maint) branch, at least merging
> of that change is automatic whenever someone does a hg merge.

I probably don't fully understand, but that seems to imply a workflow
were all changes made to one branch should also automatically be
integrated into a different branch. I'm curious as to how such a
mechanism can apply to Python.

Regards,
Martin

From pje at telecommunity.com  Sun Jul  5 20:32:48 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 05 Jul 2009 14:32:48 -0400
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.co
 m>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
Message-ID: <20090705182953.90F7D3A4099@sparrow.telecommunity.com>

At 05:26 PM 7/5/2009 +0100, Paul Moore wrote:
>def get_distribution(name):
>     for d in get_distributions():
>         if d.name == name:
>             return d
>     return None

Btw, this is broken code anyway, because it's not handling 
case-insensitivity or name canonicalization.  (I've mentioned these 
issue previously on the distutils-sig.)


From ziade.tarek at gmail.com  Sun Jul  5 20:33:22 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 20:33:22 +0200
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
Message-ID: <94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com>

2009/7/5 Paul Moore <p.f.moore at gmail.com>:
> The PEP says:
>
> """
> get_egginfo_files(local=False) -> iterator of paths
>
> Iterates over the RECORD entries and return paths for each line if the
> path is pointing a file located in the .egg-info directory or one of
> its subdirectory.
> """
>
> Should this method really only return filenames noted in the RECORD
> file? Would it not be better for it to iterate over *all* files in the
> .egg-info directory?
> I understand that there shouldn't, in practice,
> be any files in that directory *not* mentioned in the RECORD file, but
> given that we already have get_installed_files to read the RECORD
> file, I would imagine it's better for this file to so something more
> than filter the return values from get_installed_files.

I don't see a use case for having more out of get_egginfo_files.
I still find it useful because to iterate over metadata files.

Maybe we could remove it and add a filter option for get_installed_files.
A callable that gets each visited file and returns True or False to
filter them out:

  get_installed_files(path, filter=callable)

And then provide a "egginfo_files" callable to get what we have with
get_egginfo_files :

  get_installed_files(path, filter=egginfo_files)


>
> Actually, on that note, consider the pkgutil functions:
>
> def get_distribution(name):
> ? ?for d in get_distributions():
> ? ? ? ?if d.name == name:
> ? ? ? ? ? ?return d
> ? ?return None
>
> def get_file_users(path):
> ? ?for d in get_distributions():
> ? ? ? ?if d.uses(path):
> ? ? ? ? ? ?yield d
>
> These don't actually add much to the API. While I can see the
> advantage of having them as convenience methods, it might be worth
> pointing out in the PEP that that's all they are.

Sure,

>
> Similarly, how valuable is Distribution.name, given that it's the same
> as Distribution.metadata.name? I'm probably just going to make it a
> property -

It's just for conveniency, since this metadata field is also the
identifier of the distribution.

>
> @property
> def name(self):
> ? ?return self.metadata.name

I don't think this adds any value, since self.metadata is a read-only instance,
that gets loaded once when the Distribution object is created.

From pje at telecommunity.com  Sun Jul  5 20:36:27 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 05 Jul 2009 14:36:27 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.co
 m>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
Message-ID: <20090705183326.097CF3A4099@sparrow.telecommunity.com>

At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>The loader would still work with "DIST_NAME.egg-info" directories for
>compatibility with
>existing format in the query APIs, but the Distutils install command
>would rather create  "DIST_NAME.metadata"

Note that this would then break setuptools without adding any 
benefit; ".metadata" is less precise and less unique than 
'.egg-info'.  If you want a clearer name, '.pydist' or some such 
would at least be reasonably specific.  (It'd still have a backward 
compatibility problem, but at least then there'd be some benefit to 
the name change.)


From ziade.tarek at gmail.com  Sun Jul  5 20:37:45 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 20:37:45 +0200
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <20090705182953.90F7D3A4099@sparrow.telecommunity.com>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
	<20090705182953.90F7D3A4099@sparrow.telecommunity.com>
Message-ID: <94bdd2610907051137q16912234h7ce39c0f262783c9@mail.gmail.com>

2009/7/5 P.J. Eby <pje at telecommunity.com>:
> At 05:26 PM 7/5/2009 +0100, Paul Moore wrote:
>>
>> def get_distribution(name):
>> ? ?for d in get_distributions():
>> ? ? ? ?if d.name == name:
>> ? ? ? ? ? ?return d
>> ? ?return None
>
> Btw, this is broken code anyway, because it's not handling
> case-insensitivity or name canonicalization. ?(I've mentioned these issue
> previously on the distutils-sig.)

Yes thanks, we need to fix that, the case-insensitivity or name
canonicalization functions are present, just to be used
in that function too


>
>



-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Sun Jul  5 20:43:20 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Jul 2009 20:43:20 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090705183326.097CF3A4099@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
Message-ID: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>

2009/7/5 P.J. Eby <pje at telecommunity.com>:
> At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>>
>> The loader would still work with "DIST_NAME.egg-info" directories for
>> compatibility with
>> existing format in the query APIs, but the Distutils install command
>> would rather create ?"DIST_NAME.metadata"
>
> Note that this would then break setuptools without adding any benefit;
> ".metadata" is less precise and less unique than '.egg-info'.

But if it's based on PEP 302 protocols and if the pkgutil code works
with the sys.meta_path hook,
setuptools could then provide its loader, based on its EggFormats and
act as a provider without being broken.

In that case, Distutils could provide a standard loader, with the
change I mentioned.


>?If you want a
> clearer name, '.pydist' or some such would at least be reasonably specific.
> ?(It'd still have a backward compatibility problem, but at least then
> there'd be some benefit to the name change.)

I do find "DIST_NAME.metadata" well-named and specific. But I guess that's just
bikeshedding :)

From p.f.moore at gmail.com  Sun Jul  5 20:46:58 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 19:46:58 +0100
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <20090705182953.90F7D3A4099@sparrow.telecommunity.com>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
	<20090705182953.90F7D3A4099@sparrow.telecommunity.com>
Message-ID: <79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com>

2009/7/5 P.J. Eby <pje at telecommunity.com>:
> At 05:26 PM 7/5/2009 +0100, Paul Moore wrote:
>>
>> def get_distribution(name):
>> ? ?for d in get_distributions():
>> ? ? ? ?if d.name == name:
>> ? ? ? ? ? ?return d
>> ? ?return None
>
> Btw, this is broken code anyway, because it's not handling
> case-insensitivity or name canonicalization. ?(I've mentioned these issue
> previously on the distutils-sig.)

Fair point. (Although I don't recall your distutils-sig posting, so
I'm not sure what you mean by "name canonicalisation").

Note that even on case insensitive filesystems, module/package names
are handled case sensitively. I would be happy to see distribution
names handled the same (although I have no vested interest either
way).

Is there code around to handle filename matching based on the case
sensitivity of the filesystem? (My understanding is that there isn't,
and programs like Mercurial play fancy games to determine if a
filesystem is case sensitive before doing tests). Of course, if you're
OK with an inaccurate but simple choice based on OS, it would probably
be OK to use os.path.normcase.

Paul.

From p.f.moore at gmail.com  Sun Jul  5 21:05:30 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 20:05:30 +0100
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
	<94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com>
Message-ID: <79990c6b0907051205t7458f693u493d6905ec5df749@mail.gmail.com>

2009/7/5 Tarek Ziad? <ziade.tarek at gmail.com>:
> 2009/7/5 Paul Moore <p.f.moore at gmail.com>:
>> The PEP says:
>>
>> """
>> get_egginfo_files(local=False) -> iterator of paths
>>
>> Iterates over the RECORD entries and return paths for each line if the
>> path is pointing a file located in the .egg-info directory or one of
>> its subdirectory.
>> """
>>
>> Should this method really only return filenames noted in the RECORD
>> file? Would it not be better for it to iterate over *all* files in the
>> .egg-info directory?
>> I understand that there shouldn't, in practice,
>> be any files in that directory *not* mentioned in the RECORD file, but
>> given that we already have get_installed_files to read the RECORD
>> file, I would imagine it's better for this file to so something more
>> than filter the return values from get_installed_files.
>
> I don't see a use case for having more out of get_egginfo_files.
> I still find it useful because to iterate over metadata files.
>
> Maybe we could remove it and add a filter option for get_installed_files.
> A callable that gets each visited file and returns True or False to
> filter them out:
>
> ?get_installed_files(path, filter=callable)
>
> And then provide a "egginfo_files" callable to get what we have with
> get_egginfo_files :
>
> ?get_installed_files(path, filter=egginfo_files)

-1. Unnecessary generalisation. Let's stick with the 2 functions as documented.

[...]
>> Similarly, how valuable is Distribution.name, given that it's the same
>> as Distribution.metadata.name? I'm probably just going to make it a
>> property -
>
> It's just for conveniency, since this metadata field is also the
> identifier of the distribution.
>
>>
>> @property
>> def name(self):
>> ? ?return self.metadata.name
>
> I don't think this adds any value, since self.metadata is a read-only instance,
> that gets loaded once when the Distribution object is created.

... not any more :-)

Your zipfile handling was horribly broken on Windows, thanks to the
usual slash/backslash confusion. The sanest way to fix it seemed to me
to be to load the metadata lazily, rather than in the __init__ (as
otherwise, zipfile and filesystem implementation end up not being able
to share any code). Once that's done, the name attribute has to *also*
handle lazy-loading of the metadata, and the above property is the
easiest way to do this.

Actually, my implementation is looking less and less like yours, and
ultimately any implementation questions are irrelevant until you see
my code and spot all the errors :-) I'm trying to get it into a
postable state as fast as I can. (At last count, I've replaced about
140 lines of code with 70, and it now includes PEP 302 support all the
(non-internal) tests still pass. So it's looking OK...)

Paul.

From dreamingforward at gmail.com  Sun Jul  5 21:36:20 2009
From: dreamingforward at gmail.com (average)
Date: Sun, 5 Jul 2009 12:36:20 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
Message-ID: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com>

>> Is it really that confusing? I have never heard of anyone asking "what
>> is py3k?"
>
> Do you read python-list? It has been asked. Also, some people seem to
> think that py3k is different from python 3.

Personally, I vote for keeping the "3k" for 3000 (or is it 3072?).  I
believe that py3k represents a ideal that hasn't been reached, despite
being hoped for in python3.  By keeping it, it confers the idea
continual evolution *within* the language until that hypothetical
ideal is reached.  Clearly, there are times when a language reaches
only a local maximum, and must depart from itself to arrive at a more
global optimum (an annealing problem in the minimization of
frustration energy).  If py3k wasn't kept, another term would
eventually need to be invented.

marcos

From benjamin at python.org  Sun Jul  5 21:48:01 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 5 Jul 2009 14:48:01 -0500
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com>
References: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com>
Message-ID: <1afaf6160907051248s4745cc27sc9d9f89c47b0c124@mail.gmail.com>

2009/7/5 average <dreamingforward at gmail.com>:
>>> Is it really that confusing? I have never heard of anyone asking "what
>>> is py3k?"
>>
>> Do you read python-list? It has been asked. Also, some people seem to
>> think that py3k is different from python 3.
>
> Personally, I vote for keeping the "3k" for 3000 (or is it 3072?). ?I
> believe that py3k represents a ideal that hasn't been reached, despite
> being hoped for in python3. ?By keeping it, it confers the idea
> continual evolution *within* the language until that hypothetical
> ideal is reached. ?Clearly, there are times when a language reaches
> only a local maximum, and must depart from itself to arrive at a more
> global optimum (an annealing problem in the minimization of
> frustration energy). ?If py3k wasn't kept, another term would
> eventually need to be invented.

And that's why we're already fantasizing about Py4k. :)



-- 
Regards,
Benjamin

From martin at v.loewis.de  Sun Jul  5 22:37:56 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Jul 2009 22:37:56 +0200
Subject: [Python-Dev] Mercurial migration: help needed
Message-ID: <4A510F24.1060004@v.loewis.de>

In this thread, I'd like to collect things that ought to be done
but where Dirkjan has indicated that he would prefer if somebody else
did it.

So far, I have only one item: build identification. If you want to work
on this, please either provide a patch (for trunk and/or py3k), or
(if you are a committer) create a subversion branch.

It seems that Barry and I agree that for the maintenance branches,
sys.subversion should be frozen, so we need actually two sets of
patches: one that removes sys.subversion entirely, and the other that
freezes the branch to the respective one, and freezes the subversion
revision to None.

Of course, it seems that the actual representation of branches hasn't
been determined yet, so the build process integration may need to be
changed if named branches aren't going to be used in the end.

Anybody working on this should have good knowledge of the Python source
code, Mercurial, and either autoconf or Visual Studio (preferably both).

Regards,
Martin

From ajaksu at gmail.com  Sun Jul  5 22:45:13 2009
From: ajaksu at gmail.com (Daniel Diniz)
Date: Sun, 5 Jul 2009 17:45:13 -0300
Subject: [Python-Dev] Mercurial migration: help needed
In-Reply-To: <4A510F24.1060004@v.loewis.de>
References: <4A510F24.1060004@v.loewis.de>
Message-ID: <2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com>

Hi Martin,

"Martin v. L?wis" wrote:
> In this thread, I'd like to collect things that ought to be done
> but where Dirkjan has indicated that he would prefer if somebody else
> did it.
>
> So far, I have only one item: build identification. If you want to work
> on this, please either provide a patch (for trunk and/or py3k), or
> (if you are a committer) create a subversion branch.

I do want to help, but I believe I'll only have time a week from now.
If we need/want Roundup tweaks to go with Mercurial, I can work on
that (keep in mind we have a cool GSoC student working on
Mercurial-Roundup integration, and I'm willing to work on our needs
with him).

For build identification, I'd only be able to do the C/Python side of
things and (with luck) autoconf.

Cheers,
Daniel

From ncoghlan at gmail.com  Sun Jul  5 22:58:50 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 06 Jul 2009 06:58:50 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A50EA2E.8010401@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp>	<ea2499da0907030649s619b6485se1aee94f7d4e02e9@mail.gmail.com>	<4A4E6EF5.4000901@v.loewis.de>	<h2q252$rpd$1@ger.gmane.org>
	<4A50EA2E.8010401@v.loewis.de>
Message-ID: <4A51140A.9010801@gmail.com>

Martin v. L?wis wrote:
> What will need debate and discussion in the PEP is the workflow, ie.
> the order in which changes should be applied to the branches.

Particularly since there isn't even "one true workflow" *now*. I see
three main variants go by on Python-checkins:

- svnmerge based
  - commit to 2.x
  - backport to 2.6 with svnmerge
  - forward port to 3.x with svnmerge
  - backport to 3.0 (now 3.1) with svnmerge

- manual port based
  - as above, but without using svnmerge

- Py3k focused
  - commit to 3.x
  - manual backport to 2.x
  - possible svnmerge block of the backported 2.x commit
  - possible svnmerge based or manual backport to 2.6/3.1

While it would obviously be *nice* if every committer maintained 4
checkouts and either blocked or committed each change on the appropriate
branches, I think actually *requiring* a specific workflow has the
potential to cost us commits. Even if one workflow is designated the
'preferred' approach, the others still need to be supported to handle
various possibilities for the "first" commit for a given change:

- forward port only
  - commit to 2.6
  - forward port to 2.x
  - forward port to 3.1
  - forward port to 3.x

- backport only
  - commit to 3.x
  - backport to 3.1
  - backport to 2.x
  - backport to 2.6

- mixed, starting with 2.x (aka current svnmerge workflow)
  - commit to 2.x
  - backport to 2.6
  - forward port to 3.x
  - backport to 3.1

- mixed, starting with 3.1
  - commit to 3.1
  - forward port to 3.x
  - backport to 2.6
  - forward port or backport to 2.x

Note that there are actual multiple variations even of the above
workflows, based on the *source* of the various forward ports and
backports. Also, each "forward port" or "backport" can be replaced by
blocking the merge rather than applying it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sun Jul  5 23:10:52 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 06 Jul 2009 07:10:52 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>	
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>	
	<4A4EA689.1010704@gmail.com>	
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>	
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>	
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
Message-ID: <4A5116DC.1070109@gmail.com>

Tarek Ziad? wrote:
> 2009/7/5 P.J. Eby <pje at telecommunity.com>:
>>  If you want a
>> clearer name, '.pydist' or some such would at least be reasonably specific.
>>  (It'd still have a backward compatibility problem, but at least then
>> there'd be some benefit to the name change.)
> 
> I do find "DIST_NAME.metadata" well-named and specific. But I guess that's just
> bikeshedding :)

pydist has the advantage of both being more intuitive than 'egginfo'
while still making it clear that this is Python related metadata rather
than, say, something added by an OS packaging utility. So no, I don't
think it's bikeshedding:

'metadata': accurate but generic. Not clear that it relates to Python
specifically (except that it happens to stored in a Python related
directory)

'egginfo': accurate, specific and serves as a good mnemonic. However,
use of the 'egg' jargon means that someone is unlikely to guess what it
means without being told, and it's less obvious that this is Python related

'pydist': accurate, specific and without the disadvantages of the 'egg'
jargon

PJE points out that existing tools (setuptools, pip, etc) won't be
compatible with the new format at all if it uses a new name, but I am
having trouble seeing that as a *bad* thing. By using a new name for the
directory we *guarantee* that old packaging utilities won't get confused
by the new format (they simply won't acknowledge its existence).

So +1 for pydist as the directory extension in PEP 376 (for the above
reasons).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From p.f.moore at gmail.com  Sun Jul  5 23:25:34 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 5 Jul 2009 22:25:34 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090705183326.097CF3A4099@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
Message-ID: <79990c6b0907051425t175fbe5tb700b52da7a8d33e@mail.gmail.com>

2009/7/5 P.J. Eby <pje at telecommunity.com>:
> At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>>
>> The loader would still work with "DIST_NAME.egg-info" directories for
>> compatibility with
>> existing format in the query APIs, but the Distutils install command
>> would rather create ?"DIST_NAME.metadata"
>
> Note that this would then break setuptools without adding any benefit;
> ".metadata" is less precise and less unique than '.egg-info'. ?If you want a
> clearer name, '.pydist' or some such would at least be reasonably specific.
> ?(It'd still have a backward compatibility problem, but at least then
> there'd be some benefit to the name change.)

Personally, the filename doesn't bother me. It's the API name that I'd
like changed (particularly as the contraction "egginfo" is clumsy in
function names...)

Paul.

From ben+python at benfinney.id.au  Mon Jul  6 00:21:06 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 06 Jul 2009 08:21:06 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
Message-ID: <87ljn2rctp.fsf@benfinney.id.au>

Nick Coghlan <ncoghlan at gmail.com> writes:

> 'pydist': accurate, specific and without the disadvantages of the 'egg'
> jargon
> 
> PJE points out that existing tools (setuptools, pip, etc) won't be
> compatible with the new format at all if it uses a new name, but I am
> having trouble seeing that as a *bad* thing. By using a new name for the
> directory we *guarantee* that old packaging utilities won't get confused
> by the new format (they simply won't acknowledge its existence).
> 
> So +1 for pydist as the directory extension in PEP 376 (for the above
> reasons).

+1 for the same reasons. Thank you for expressing them.

-- 
 \     ?We are not gonna be great; we are not gonna be amazing; we are |
  `\           gonna be *amazingly* amazing!? ?Zaphod Beeblebrox, _The |
_o__)                Hitch-Hiker's Guide To The Galaxy_, Douglas Adams |
Ben Finney


From pje at telecommunity.com  Mon Jul  6 03:04:08 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 05 Jul 2009 21:04:08 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com
 >
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
Message-ID: <20090706010108.CCEC83A4099@sparrow.telecommunity.com>

At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>But if it's based on PEP 302 protocols and if the pkgutil code works
>with the sys.meta_path hook,
>setuptools could then provide its loader, based on its EggFormats and
>act as a provider without being broken.

You misunderstand me.  The whole point of putting .egg-info in 
distutils in the first place was to enable setuptools to detect the 
presence of disutils-installed packages.  That's what's broken by 
changing the name.


From pje at telecommunity.com  Mon Jul  6 03:06:20 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 05 Jul 2009 21:06:20 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <4A5116DC.1070109@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
Message-ID: <20090706010320.3220A3A4099@sparrow.telecommunity.com>

At 07:10 AM 7/6/2009 +1000, Nick Coghlan wrote:
>By using a new name for the
>directory we *guarantee* that old packaging utilities won't get confused
>by the new format (they simply won't acknowledge its existence).

This is incorrect; they will get confused because they will think 
that the relevant package is *not* installed, and proceed to install 
a duplicate.  That's why .egg-info was added to the stdlib in the first place.


From martin at v.loewis.de  Mon Jul  6 07:07:30 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 06 Jul 2009 07:07:30 +0200
Subject: [Python-Dev] Mercurial migration: help needed
In-Reply-To: <2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com>
References: <4A510F24.1060004@v.loewis.de>
	<2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com>
Message-ID: <4A518692.6020605@v.loewis.de>

> I do want to help, but I believe I'll only have time a week from now.
> If we need/want Roundup tweaks to go with Mercurial, I can work on
> that (keep in mind we have a cool GSoC student working on
> Mercurial-Roundup integration, and I'm willing to work on our needs
> with him).

I think that's straight-forward. Once we know what URLs to link to,
we just need to fix the regexps.

> For build identification, I'd only be able to do the C/Python side of
> things and (with luck) autoconf.

Ok, we'll see in a week from now whether anybody had volunteered.

Regards,
Martin

From ziade.tarek at gmail.com  Mon Jul  6 08:55:14 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 08:55:14 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090706010320.3220A3A4099@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
	<20090706010320.3220A3A4099@sparrow.telecommunity.com>
Message-ID: <94bdd2610907052355w6d69ee53nc4e85cf1e2a5e748@mail.gmail.com>

But as I said earlier, if we use a PEP 302-like loader, distutils will
be able to consume
several loaders, so setuptools will be able to provide its storage
strategy (naming and egg dir locations)

So I don't understand why you are saying that it will be incompatible
or get confused.


2009/7/6 P.J. Eby <pje at telecommunity.com>:
> At 07:10 AM 7/6/2009 +1000, Nick Coghlan wrote:
>>
>> By using a new name for the
>> directory we *guarantee* that old packaging utilities won't get confused
>> by the new format (they simply won't acknowledge its existence).
>
> This is incorrect; they will get confused because they will think that the
> relevant package is *not* installed, and proceed to install a duplicate.
> ?That's why .egg-info was added to the stdlib in the first place.
>
>



-- 
Tarek Ziad? | http://ziade.org

From p.f.moore at gmail.com  Mon Jul  6 09:15:05 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 08:15:05 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090706010108.CCEC83A4099@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
Message-ID: <79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com>

2009/7/6 P.J. Eby <pje at telecommunity.com>:
> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>>
>> But if it's based on PEP 302 protocols and if the pkgutil code works
>> with the sys.meta_path hook,
>> setuptools could then provide its loader, based on its EggFormats and
>> act as a provider without being broken.
>
> You misunderstand me. ?The whole point of putting .egg-info in distutils in
> the first place was to enable setuptools to detect the presence of
> disutils-installed packages. ?That's what's broken by changing the name.

This implies that there is no possibility that setuptools will be
changed to support the new standard. That's patently false. Whether
you *want* to change setuptools in such a way is up to you. And it's
worth a note in the PEP if this change is made, that it will require a
change to setuptools if that package is still to recognise
distutils-installed packages.

Paul.

From ronaldoussoren at mac.com  Mon Jul  6 08:16:08 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 06 Jul 2009 08:16:08 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <4A5116DC.1070109@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
Message-ID: <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>

I'm -1 on changing the name. For better or worse setuptools is the  
elephant in the room w.r.t. package management and it would IMHO be  
better to stay compatible (even if the stdlib only implements a subset  
of setuptools/pkg_resources)

Ronald



On 5 jul 2009, at 23:10, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Tarek Ziad? wrote:
>> 2009/7/5 P.J. Eby <pje at telecommunity.com>:
>>> If you want a
>>> clearer name, '.pydist' or some such would at least be reasonably  
>>> specific.
>>> (It'd still have a backward compatibility problem, but at least then
>>> there'd be some benefit to the name change.)
>>
>> I do find "DIST_NAME.metadata" well-named and specific. But I guess  
>> that's just
>> bikeshedding :)
>
> pydist has the advantage of both being more intuitive than 'egginfo'
> while still making it clear that this is Python related metadata  
> rather
> than, say, something added by an OS packaging utility. So no, I don't
> think it's bikeshedding:
>
> 'metadata': accurate but generic. Not clear that it relates to Python
> specifically (except that it happens to stored in a Python related
> directory)
>
> 'egginfo': accurate, specific and serves as a good mnemonic. However,
> use of the 'egg' jargon means that someone is unlikely to guess what  
> it
> means without being told, and it's less obvious that this is Python  
> related
>
> 'pydist': accurate, specific and without the disadvantages of the  
> 'egg'
> jargon
>
> PJE points out that existing tools (setuptools, pip, etc) won't be
> compatible with the new format at all if it uses a new name, but I am
> having trouble seeing that as a *bad* thing. By using a new name for  
> the
> directory we *guarantee* that old packaging utilities won't get  
> confused
> by the new format (they simply won't acknowledge its existence).
>
> So +1 for pydist as the directory extension in PEP 376 (for the above
> reasons).
>
> Cheers,
> Nick.
>
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

From ziade.tarek at gmail.com  Mon Jul  6 09:50:09 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 09:50:09 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com>
Message-ID: <94bdd2610907060050h307669e6r600692c1ed6fcaf0@mail.gmail.com>

2009/7/6 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/6 P.J. Eby <pje at telecommunity.com>:
>> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>>>
>>> But if it's based on PEP 302 protocols and if the pkgutil code works
>>> with the sys.meta_path hook,
>>> setuptools could then provide its loader, based on its EggFormats and
>>> act as a provider without being broken.
>>
>> You misunderstand me. ?The whole point of putting .egg-info in distutils in
>> the first place was to enable setuptools to detect the presence of
>> disutils-installed packages. ?That's what's broken by changing the name.
>

Forget about my previous mail, I didn't see that answer,

> This implies that there is no possibility that setuptools will be
> changed to support the new standard. That's patently false. Whether
> you *want* to change setuptools in such a way is up to you. And it's
> worth a note in the PEP if this change is made, that it will require a
> change to setuptools if that package is still to recognise
> distutils-installed packages.

+1

setuptools is built on the top of distutils, not the contrary. And if setuptools
wants to query installed distribution, it will have to be changed to use the
query functions added in pkgutil.

Tarek

From ziade.tarek at gmail.com  Mon Jul  6 09:53:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 09:53:21 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
	<31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>
Message-ID: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>

2009/7/6 Ronald Oussoren <ronaldoussoren at mac.com>:
> I'm -1 on changing the name. For better or worse setuptools is the elephant
> in the room w.r.t. package management and it would IMHO be better to stay
> compatible (even if the stdlib only implements a subset of
> setuptools/pkg_resources)
>

I'd rather see the elephant evolves.

I don't see why we should bend a standard we want to introduce in the stdlib,
for a third-party package that is able to evolve to stick to a new standard
without any problem.

From p.f.moore at gmail.com  Mon Jul  6 11:32:28 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 10:32:28 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
	<31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>
	<94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>
Message-ID: <79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.com>

2009/7/6 Tarek Ziad? <ziade.tarek at gmail.com>:
> 2009/7/6 Ronald Oussoren <ronaldoussoren at mac.com>:
>> I'm -1 on changing the name. For better or worse setuptools is the elephant
>> in the room w.r.t. package management and it would IMHO be better to stay
>> compatible (even if the stdlib only implements a subset of
>> setuptools/pkg_resources)
>>
>
> I'd rather see the elephant evolves.
>
> I don't see why we should bend a standard we want to introduce in the stdlib,
> for a third-party package that is able to evolve to stick to a new standard
> without any problem.

I agree with this.

The big problem is that setuptools development seems to have
essentially stagnated, and while that shouldn't be something we care
about, it does need to be considered. Although my instinct is to fight
against "do it our way because we won't change to do it your way"
arguments, I have to ask whether the (purely internal) name of the
metadata directory is really worth fighting over.

I'm +0 on changing the name, as long as it's the *only* "do it this
way because setuptools isn't going to change" issue. If more such
issues come up, I'll become +1 on a clean break, to avoid endless such
arguments and restrictions.

Paul.

From ncoghlan at gmail.com  Mon Jul  6 16:36:15 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 00:36:15 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <20090706010108.CCEC83A4099@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
Message-ID: <4A520BDF.80503@gmail.com>

P.J. Eby wrote:
> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>> But if it's based on PEP 302 protocols and if the pkgutil code works
>> with the sys.meta_path hook,
>> setuptools could then provide its loader, based on its EggFormats and
>> act as a provider without being broken.
> 
> You misunderstand me.  The whole point of putting .egg-info in distutils
> in the first place was to enable setuptools to detect the presence of
> disutils-installed packages.  That's what's broken by changing the name.

How much information does setuptools actually need in order to tell that
a distribution is already present? Presumably the existing .egginfo
files generated by distutils are sufficient for that task?

If so, I'd still prefer to keep the new metadata safely out of reach of
the legacy package management systems that don't understand it, while
having distutils retain the ability to generate a legacy ".egginfo" file
to warn off the existing package management tools from installing the
same distribution again.

So instead of ".egginfo becomes a directory" as in the current version
of PEP 376, the existing distutils .egginfo file generation would
instead remain unchanged with a ".pydist" directory being added
alongside to hold all the new goodies. That way, the legacy formats
(EGG-INFO, .egg-info file, .egg-info directory) continue to play as well
together as they do now, while the new distutils system will understand
.pydist directories natively and allow the injection of hook
implementations by the third party package management tools in order to
understand the old formats.

That seems like it will give us the best of both worlds - a clean base
to work from in the future, but without breaking the existing
interoperability mechanisms (at least in the short term - maybe by the
time 3.3 rolls around we will be able to start thinking about
deprecating the legacy .egginfo formats, but even considering that
question is a long way off).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From mal at egenix.com  Mon Jul  6 16:51:47 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 06 Jul 2009 16:51:47 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
Message-ID: <4A520F83.7020300@egenix.com>

Dirkjan Ochtman wrote:
> In response to some rumblings on python-committers and just to request
> more feedback, a progress report. I know it's long, I've tried to put
> to keep it concise and chunked, though.

Two things:

 * We need some form of documentation of how committers are expected
   to work with the hg repo. (This is also missing for the Subversion
   repo, which due to the 3.x branch has gotten somewhat unclear - at
   least for me)

   It is currently not clear where to check in patches, whether and
   where to backport or forward-patch, which branches to consider
   closed, etc.

   E.g. if I check in something in trunk/ (Python 2.7), do I have to
   forward patch this change to the 3.0 branch (guess not), the py3k/
   branch (Python 3.1), or will someone else take care of this, so that
   it's better not to interfere by doing it myself ?

 * Our tracker, checkin messages, online documentation and lots of
   Python resources on the web are full of references to the
   Subversion rXYZ notation of changesets.

   The PEP has to provide some way to gracefully provide a redirect
   from those URLs to the new ones (both for the source code browser
   and the bug search tool on python.org).

   The PEP will also have to address the same issue for checkin
   messages. Perhaps by converting the rXYZ notation to the hash
   based values during migration or by adding the has value after
   the rXYZ value in parens ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 06 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From p.f.moore at gmail.com  Mon Jul  6 17:14:08 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 16:14:08 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A520BDF.80503@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
Message-ID: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>

2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> P.J. Eby wrote:
>> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
>>> But if it's based on PEP 302 protocols and if the pkgutil code works
>>> with the sys.meta_path hook,
>>> setuptools could then provide its loader, based on its EggFormats and
>>> act as a provider without being broken.
>>
>> You misunderstand me. ?The whole point of putting .egg-info in distutils
>> in the first place was to enable setuptools to detect the presence of
>> disutils-installed packages. ?That's what's broken by changing the name.

This is getting confusing. Is Phillip saying that setuptools will cope
with the file changing to a directory without modification, but it
can't handle a change in the name?

My site-packages has a confusing mix of egginfo directories and files.
Note that I NEVER use setuptools other than where an existing
package's setup.py requires it. In that case, I still only do python
setup.py bdist_wininst and install the generated installer.

So is PEP 376 going to be able to cope with the stuff I have installed
at the moment? If not, what's the point???

Example:

I have the "coverage" package installed. It does not use setuptools at
runtime - I could happily install it from a bdist_wininst on a machine
without setuptools installed, and it would run fine. (In fact, that's
what I did!) It has created a coverage-3.0-py2.6.egg-info *directory*
in my site-packages, with no RECORD file - presumably, this is in a
setuptools format.

If setuptools is not going to change to conform to PEP 376, then any
tools built using PEP 376 will fail to recognise my coverage install.
Which, as far as I am concerned, is a completely standard
bdist_wininst installer.

In fact, the above strongly suggests to me that PEP 376 must NOT
follow setuptools conventions - otherwise, it will end up clashing
with the files setuptools is putting on my system.

> How much information does setuptools actually need in order to tell that
> a distribution is already present? Presumably the existing .egginfo
> files generated by distutils are sufficient for that task?

It appears so, but setuptools doesn't use (or create!!!) those files
in its own installer formats.

> If so, I'd still prefer to keep the new metadata safely out of reach of
> the legacy package management systems that don't understand it, while
> having distutils retain the ability to generate a legacy ".egginfo" file
> to warn off the existing package management tools from installing the
> same distribution again.

So for a minimal case of a single .py file packaged up as a
distribution, we now have the .py file, a legacy .egginfo file, a new
.pydist directory containing RECORD and PKG-INFO files?

That's getting silly.

>
> So instead of ".egginfo becomes a directory" as in the current version
> of PEP 376, the existing distutils .egginfo file generation would
> instead remain unchanged with a ".pydist" directory being added
> alongside to hold all the new goodies. That way, the legacy formats
> (EGG-INFO, .egg-info file, .egg-info directory) continue to play as well
> together as they do now, while the new distutils system will understand
> .pydist directories natively and allow the injection of hook
> implementations by the third party package management tools in order to
> understand the old formats.
>
> That seems like it will give us the best of both worlds - a clean base
> to work from in the future, but without breaking the existing
> interoperability mechanisms (at least in the short term - maybe by the
> time 3.3 rolls around we will be able to start thinking about
> deprecating the legacy .egginfo formats, but even considering that
> question is a long way off).

And yet, given that PEP 376 is explicitly being designed with a goal
being to act as the common standard that *all* package management
formats can use, is it not the whole point that once it's defined and
we have achieved consensus, existing package managers will switch to
using it rather than retaining a range of custom formats?

I honestly fail to see why we are catering to this "setuptools cannot
change" view. Usually, it's the standard library that cannot change,
because of the longer release cycle of the core. If a standard is
agreed, and setuptools won't conform to it, maybe it's time for
someone to fork setuptools (unilaterally, if Phillip can't agree to an
amicable transfer). Otherwise setuptools will remain a roadblock for
any development of distutils.

Frustrated-ly y'rs
Paul.

From ziade.tarek at gmail.com  Mon Jul  6 17:40:43 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 17:40:43 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
Message-ID: <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com>

On Mon, Jul 6, 2009 at 5:14 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> [...]
> And yet, given that PEP 376 is explicitly being designed with a goal
> being to act as the common standard that *all* package management
> formats can use, is it not the whole point that once it's defined and
> we have achieved consensus, existing package managers will switch to
> using it rather than retaining a range of custom formats?

That's the goal from the very beginning.

>
> I honestly fail to see why we are catering to this "setuptools cannot
> change" view. Usually, it's the standard library that cannot change,
> because of the longer release cycle of the core. If a standard is
> agreed, and setuptools won't conform to it, maybe it's time for
> someone to fork setuptools (unilaterally, if Phillip can't agree to an
> amicable transfer). Otherwise setuptools will remain a roadblock for
> any development of distutils.
>
> Frustrated-ly y'rs

+1

why can't we just go ahead and continue the work as we started with PEP 376,
introducing your work on PEP 302-like behavior.

Then if we get a consensus on this PEP and introduce it in 2.7/3.2,
setuptools will have to follow this consensus.

FWIW I am also frustrated because the setuptools development has been stopped
9 months ago, and *doesn't work anymore with the current distutils trunk*.
There are a lot of patches waiting in the setuptools tracker people wrote.
I hate to say this but this project needs an active maintainer or it's
going to die.

So I've said in distutils-sig that I would start my own branch of setuptools
before 2.7/3.2 is out for the people in the community that relies
on this package. Not to maintain the project because I have a lot to do already
on distutils side. But at least to make sure everyone can work with
the upcoming releases
of Python 2.x and 3.X without having their setup.py broken (and switch
eventually to a plain distutils
solution maybe)

Tarek

From p.f.moore at gmail.com  Mon Jul  6 17:53:01 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 16:53:01 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com>
Message-ID: <79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com>

2009/7/6 Tarek Ziad? <ziade.tarek at gmail.com>:
> why can't we just go ahead and continue the work as we started with PEP 376,
> introducing your work on PEP 302-like behavior.
>
> Then if we get a consensus on this PEP and introduce it in 2.7/3.2,
> setuptools will have to follow this consensus.

Essentially, because when I ask questions, responses along the lines
of "you have to do it like X because setuptools does that" come back,
and (not being a setuptools user) I can't tell whether there's a valid
reason in there.

I'm uncomfortable assuming that setuptools experience is irrelevant,
but I can't distinguish between valid arguments and "setuptools can't
change" arguments.

I need to write another email with a list of outstanding open issues.
If we can thrash them out *without* getting bogged down in setuptools
compatibility questions, then I think we can move forward again. I'll
do that this evening.

Paul.

From ziade.tarek at gmail.com  Mon Jul  6 18:05:49 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 18:05:49 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com>
	<79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com>
Message-ID: <94bdd2610907060905w32026c2t9bc5f76b93233f33@mail.gmail.com>

On Mon, Jul 6, 2009 at 5:53 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/6 Tarek Ziad? <ziade.tarek at gmail.com>:
>> why can't we just go ahead and continue the work as we started with PEP 376,
>> introducing your work on PEP 302-like behavior.
>>
>> Then if we get a consensus on this PEP and introduce it in 2.7/3.2,
>> setuptools will have to follow this consensus.
>
> Essentially, because when I ask questions, responses along the lines
> of "you have to do it like X because setuptools does that" come back,
> and (not being a setuptools user) I can't tell whether there's a valid
> reason in there.

Notice that we created PEP 376 in distutils-SIG with most of the valid
reasons/use cases setuptools had,
with the help of Phillip, before I brought it up again on python-dev.


>
> I'm uncomfortable assuming that setuptools experience is irrelevant,
> but I can't distinguish between valid arguments and "setuptools can't
> change" arguments.
>
> I need to write another email with a list of outstanding open issues.
> If we can thrash them out *without* getting bogged down in setuptools
> compatibility questions, then I think we can move forward again. I'll
> do that this evening.

Ok, I'll wait for your work to work on the PEP again then, and your
push in the hg repo as well.

From pje at telecommunity.com  Mon Jul  6 18:51:40 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 06 Jul 2009 12:51:40 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.co
 m>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
	<31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>
	<94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>
	<79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.com>
Message-ID: <20090706164840.9E8763A407B@sparrow.telecommunity.com>

At 10:32 AM 7/6/2009 +0100, Paul Moore wrote:
>I'm +0 on changing the name, as long as it's the *only* "do it this
>way because setuptools isn't going to change" issue.

Please note that I never said that.  I was the one who suggested 
".pydist", remember?

I just don't want to have to complicate the pkg_resources directory 
scan to support a stupid name like "metadata", no offense to Tarek.  ;-)

I'd sooner not make the change at all, just because I doubt I'm the 
only person who has code that depends on the old name.  (Consider the 
Linux OS packagers who treat .egg-info files specially at the moment.)


From pje at telecommunity.com  Mon Jul  6 18:58:47 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 06 Jul 2009 12:58:47 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.co
 m>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
Message-ID: <20090706165546.25CCB3A407B@sparrow.telecommunity.com>

At 04:14 PM 7/6/2009 +0100, Paul Moore wrote:
>2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> > P.J. Eby wrote:
> >> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote:
> >>> But if it's based on PEP 302 protocols and if the pkgutil code works
> >>> with the sys.meta_path hook,
> >>> setuptools could then provide its loader, based on its EggFormats and
> >>> act as a provider without being broken.
> >>
> >> You misunderstand me.  The whole point of putting .egg-info in distutils
> >> in the first place was to enable setuptools to detect the presence of
> >> disutils-installed packages.  That's what's broken by changing the name.
>
>This is getting confusing. Is Phillip saying that setuptools will cope
>with the file changing to a directory without modification, but it
>can't handle a change in the name?

The existing versions of setuptools will read a file or a directory 
with no problem; it's the name change that will require a code 
change, and it's a rather more complex issue than just one name 
change, because it'll need to support both names.

What's more, on the build/install side, it'll have to figure out 
whether to use the new name or the old name when creating a project's 
metadata for installation in single-version mode.  In other words, 
this will likely affect pip as well, or at least the parts of 
setuptools that pip uses.


>My site-packages has a confusing mix of egginfo directories and files.
>Note that I NEVER use setuptools other than where an existing
>package's setup.py requires it. In that case, I still only do python
>setup.py bdist_wininst and install the generated installer.
>
>So is PEP 376 going to be able to cope with the stuff I have installed
>at the moment? If not, what's the point???

If I understand Tarek's proposal correctly, then no, it will not cope.


>If setuptools is not going to change to conform to PEP 376, then any
>tools built using PEP 376 will fail to recognise my coverage install.

I'm all in favor of adding RECORD support to setuptools; it was in 
fact my idea to have the file there in the first place.

Adding a RECORD file doesn't introduce any new and weird name 
migration requirements, which is why I'd rather not change the 
extension if we can avoid it.

Reading both names is painful, writing both is more so, and I'm not 
sure how many tools/users *besides* setuptools will be affected by a 
name change.


> > How much information does setuptools actually need in order to tell that
> > a distribution is already present? Presumably the existing .egginfo
> > files generated by distutils are sufficient for that task?
>
>It appears so, but setuptools doesn't use (or create!!!) those files
>in its own installer formats.

Setuptools treats an .egg-info file as if it were a PKG-INFO file 
contained in an .egg-info directory.  This allows it to treat 
distutils-supplied .egg-info files as if they were 
setuptools-supplied .egg-info directories containing exactly one file.


From hnassrat at gmail.com  Mon Jul  6 18:16:38 2009
From: hnassrat at gmail.com (Hatem Nassrat)
Date: Mon, 6 Jul 2009 13:16:38 -0300
Subject: [Python-Dev] Any updates on this subprocess/signal bug/issue (Re:
	subprocess	and EINTR errnos)
In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
Message-ID: <20090706161638.GA16498@gmail.com>

On Wed, 17 Nov 2004, Peter Astrand wrote,
> I've made a patch (attached) to subprocess.py (and test_subprocess.py)
> that should guard against EINTR, but I haven't committed it yet. It's
> quite large.
> 
> Are Python modules supposed to handle EINTR? Why not let the C code handle
> this? Or, perhaps the signal module should provide a sigaction function,
> so that users can use SA_RESTART.

This is a snippet from a email sent in 2004, I was wondering if there
was any update on this issue. Are these issues supposed to be handled on
a per application basis, or will a fix go into Python in the near
future?

-- 
Hatem Nassrat

Refrences: 
    
    http://mail.python.org/pipermail/python-dev/2004-November/049983.html

From ziade.tarek at gmail.com  Mon Jul  6 19:18:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 6 Jul 2009 19:18:44 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090706165546.25CCB3A407B@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
Message-ID: <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>

On Mon, Jul 6, 2009 at 6:58 PM, P.J. Eby<pje at telecommunity.com> wrote:
>> My site-packages has a confusing mix of egginfo directories and files.
>> Note that I NEVER use setuptools other than where an existing
>> package's setup.py requires it. In that case, I still only do python
>> setup.py bdist_wininst and install the generated installer.

If you end up with confusing mix of egginfo directories and files,
that's because
for the packages that uses setuptools in their setup.py, it patches
distutils and
to creates its own egg-info format.

>> So is PEP 376 going to be able to cope with the stuff I have installed
>> at the moment? If not, what's the point???
>
> If I understand Tarek's proposal correctly, then no, it will not cope.

Why that ? Can you detail ? On a system that uses only plain distutils
distributions, it'll work.

From pje at telecommunity.com  Mon Jul  6 19:40:31 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 06 Jul 2009 13:40:31 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.co
 m>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
Message-ID: <20090706173730.A09F13A407B@sparrow.telecommunity.com>

At 07:18 PM 7/6/2009 +0200, Tarek Ziad? wrote:
> >> So is PEP 376 going to be able to cope with the stuff I have installed
> >> at the moment? If not, what's the point???
> >
> > If I understand Tarek's proposal correctly, then no, it will not cope.
>
>Why that ? Can you detail ? On a system that uses only plain distutils
>distributions, it'll work.

Paul's question, as I understood it, was whether pkgutil would handle 
the existing .egg-info's.  Your proposal, as I understood it, was 
that pkgutil would not detect .egg-info's without a plugin of some 
sort...  and you've not proposed a way of loading such a thing, which 
means that to use pkgutil with that plugin, an application would have 
to know that the plugin exists and import it.

Unless you're planning to have a plugin registration facility (ala 
entry points) so that plugins can be used without each application 
needing to discover them on its own?


From p.f.moore at gmail.com  Mon Jul  6 19:57:52 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 18:57:52 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090706165546.25CCB3A407B@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
Message-ID: <79990c6b0907061057h66a82f04ke6f4971209752255@mail.gmail.com>

2009/7/6 P.J. Eby <pje at telecommunity.com>:
> At 04:14 PM 7/6/2009 +0100, Paul Moore wrote:
>> This is getting confusing. Is Phillip saying that setuptools will cope
>> with the file changing to a directory without modification, but it
>> can't handle a change in the name?
>
> The existing versions of setuptools will read a file or a directory with no
> problem; it's the name change that will require a code change, and it's a
> rather more complex issue than just one name change, because it'll need to
> support both names.

If PEP 376 is the standard, setuptools can just use that. Whether you
have to handle backward compatibility issues so that newer versions of
setuptools work with older metadata layouts is largely an internal
setuptools issue. While I don't want to gratuitously break setuptools,
it already supports something like 3 formats, as I understand it, so
adding a 4th shouldn't be the end of the world.

> What's more, on the build/install side, it'll have to figure out whether to
> use the new name or the old name when creating a project's metadata for
> installation in single-version mode. ?In other words, this will likely
> affect pip as well, or at least the parts of setuptools that pip uses.

And yet, there have been no adverse comments from users of pip, or
buildout, or any of the other tools that this PEP is targeted at. So
either there's a huge chunk of the affected community that's unaware
of this discussion (which I doubt), or they have no problem with this,
or their interaction is via setuptools, and so as long as setuptools
is changed, they are fine. (The other benefit of the PEP is that if
setuptools *doesn't* change, they have alternative tools to build
their own interface, although they may not want to be forced to do
this...)

>> If setuptools is not going to change to conform to PEP 376, then any
>> tools built using PEP 376 will fail to recognise my coverage install.
>
> I'm all in favor of adding RECORD support to setuptools; it was in fact my
> idea to have the file there in the first place.
>
> Adding a RECORD file doesn't introduce any new and weird name migration
> requirements, which is why I'd rather not change the extension if we can
> avoid it.

You don't give any reasons why you pick and choose what changes you're
happy with - "weird name migration requirements"  isn't very specific.

> Reading both names is painful, writing both is more so, and I'm not sure how
> many tools/users *besides* setuptools will be affected by a name change.

Don't forget - I made that point when I said I wasn't too bothered
about what was an internal name change.

>> > How much information does setuptools actually need in order to tell that
>> > a distribution is already present? Presumably the existing .egginfo
>> > files generated by distutils are sufficient for that task?
>>
>> It appears so, but setuptools doesn't use (or create!!!) those files
>> in its own installer formats.
>
> Setuptools treats an .egg-info file as if it were a PKG-INFO file contained
> in an .egg-info directory. ?This allows it to treat distutils-supplied
> .egg-info files as if they were setuptools-supplied .egg-info directories
> containing exactly one file.

Hey, look - you get to *remove* some compatibility code!

Paul.

From brett at python.org  Mon Jul  6 19:49:53 2009
From: brett at python.org (Brett Cannon)
Date: Mon, 6 Jul 2009 10:49:53 -0700
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A520F83.7020300@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com> 
	<4A520F83.7020300@egenix.com>
Message-ID: <bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>

On Mon, Jul 6, 2009 at 07:51, M.-A. Lemburg <mal at egenix.com> wrote:

> Dirkjan Ochtman wrote:
> > In response to some rumblings on python-committers and just to request
> > more feedback, a progress report. I know it's long, I've tried to put
> > to keep it concise and chunked, though.
>
> Two things:
>
>  * We need some form of documentation of how committers are expected
>   to work with the hg repo. (This is also missing for the Subversion
>   repo, which due to the 3.x branch has gotten somewhat unclear - at
>   least for me)
>

I am planning to get a version of the dev FAQ written up that covers most of
what it already does now for svn.


>
>   It is currently not clear where to check in patches, whether and
>   where to backport or forward-patch, which branches to consider
>   closed, etc.
>
>   E.g. if I check in something in trunk/ (Python 2.7), do I have to
>   forward patch this change to the 3.0 branch (guess not), the py3k/
>   branch (Python 3.1), or will someone else take care of this, so that
>   it's better not to interfere by doing it myself ?
>

This question is partially answered by
http://www.python.org/dev/faq/#how-do-i-merge-between-branches, but I agree
that we should have either this spelled out in the FAQ or a
committer-specific doc at www.python.org/dev/ that makes this all very
obvious.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090706/411587dc/attachment.htm>

From p.f.moore at gmail.com  Mon Jul  6 20:15:52 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 19:15:52 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <20090706173730.A09F13A407B@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
	<20090706173730.A09F13A407B@sparrow.telecommunity.com>
Message-ID: <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com>

2009/7/6 P.J. Eby <pje at telecommunity.com>:
> At 07:18 PM 7/6/2009 +0200, Tarek Ziad? wrote:
>>
>> >> So is PEP 376 going to be able to cope with the stuff I have installed
>> >> at the moment? If not, what's the point???
>> >
>> > If I understand Tarek's proposal correctly, then no, it will not cope.
>>
>> Why that ? Can you detail ? On a system that uses only plain distutils
>> distributions, it'll work.
>
> Paul's question, as I understood it, was whether pkgutil would handle the
> existing .egg-info's. ?Your proposal, as I understood it, was that pkgutil
> would not detect .egg-info's without a plugin of some sort... ?and you've
> not proposed a way of loading such a thing, which means that to use pkgutil
> with that plugin, an application would have to know that the plugin exists
> and import it.

Close. My point was that distributions which use setuptools in their
setup.py, *even if there's no runtime dependency on setuptools* end up
with non-standard .egg-info's. There is no good reason for this, from
my POV as a package user. So if setuptools is breaking otherwise
entirely standard packages, so that they won't work with core
distutils mechanisms, that's wrong. Setuptools is refusing to play
nicely with other packaging systems.

Now at the moment, this is a moot point, because the stdlib doesn't
have any package discovery mechanisms. But when they *do* appear, I
expect setuptools to work with them. So, when PEP 376 is accepted and
implemented, I contend that, in order to be a well-behaved packaging
tool, setuptools must:

- Generate PEP 376 compatible metadata
- Recognise PEP 376 compatible metadata
- Unless installing a custom package format (such as eggs), which has
its own PEP 302 installer, refrain from installing hooks that alter
the way the standard distutils commands handle metadata.

That's my view as an end user of Python packages. It means that if I
choose not to use setuptools/easy_install to manage my packages, I
shouldn't need them so that distutils or other core utilities (such as
bdist_wininst) can do so.

> Unless you're planning to have a plugin registration facility (ala entry
> points) so that plugins can be used without each application needing to
> discover them on its own?

My understanding was that package management tools provide their own
end-user level install and uninstall features (the PEP 376 uninstall
is just for filesystems, to correspond to the distutils install
command). PEP 376 is just standardising the format of metadata, and
the APIs to allow read access.

The PEP 302 hooks I'm writing provide the "plugin" ability for
alternative storage formats. But there's no provision for alternative
*metadata* formats. What's the point of a standard if people don't
follow it?

Paul.

From p.f.moore at gmail.com  Mon Jul  6 20:38:24 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 19:38:24 +0100
Subject: [Python-Dev] PEP 376 - Open questions
Message-ID: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>

As promised, here are some open questions on PEP 376.

- Will the public API names be changed from *egginfo* to *metadata*?

- What precisely are the use cases for absolute path names? Concrete
examples are needed. With the current spec, some things can go wrong
(e.g., see below), so we need real use cases to know how to address
this.

- How will bdist_wininst/bdist_msi/bdist_rpm be updated?

- How will the RECORD file be managed? (Particularly for the case of
bdist_xxx) [1]

- Can distutils be made to install files in places the current RECORD
file spec can't handle? (I think the answer is "yes"). What happens
then?

- Should distribution names be case insensitive on case insensitive
filesystems? For comparison, module/package names are always case
sensitive even on case insensitive systems.

- What will happen with the md5 hash? Are more types of hash going to
be supported? What's the default? (Actually, the PEP doesn't need to
care about the default, as the PEP says nothing about how RECORD files
are written).

[1] Note - the idea of using $EXEC_PREFIX / $PREFIX implies that the
RECORD file is intended to be relocatable. Which is worrying, and also
implies that an individual Distribution class must be able to handle
filesystem files as well as whatever else it handles (consider
mylib.zip on sys.path, containing a distribution which installed some
files in $PREFIX). If this isn't possible, it should be clearly stated
that it isn't possible. If it is, the ramifications are complex...

I'm still unsure how the local vs relative, slash-separated filename
formats should be handled. I don't actually think there's any real
benefit in having 2 formats. I propose:

- get_installed_files(), uses(), get_file_users() - always use local
format absolute pathnames (for zipfiles and the like, these may not be
"real" filenames, but they will be in "real filename" format, so other
code will be able to manipulate them as filenames).

- get_egginfo_files, get_egginfo_file - always use slash-separated
forms, relative to the egginfo directory (so the name of the RECORD
file is just 'RECORD')

But there could be uses I haven't thought about, so this still counts
as an open question at the moment (i.e., I'm reluctant to implement
things this way until I've had some feedback).

Paul.

From hnassrat at gmail.com  Mon Jul  6 20:18:40 2009
From: hnassrat at gmail.com (Hatem Nassrat)
Date: Mon, 6 Jul 2009 15:18:40 -0300
Subject: [Python-Dev] Any updates on this subprocess/signal bug/issue (Re:
	subprocess	and EINTR errnos)
In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
Message-ID: <20090706181840.GA19211@gmail.com>

On Wed, 17 Nov 2004, Peter Astrand wrote,
> I've made a patch (attached) to subprocess.py (and test_subprocess.py)
> that should guard against EINTR, but I haven't committed it yet. It's
> quite large.
> 
> Are Python modules supposed to handle EINTR? Why not let the C code handle
> this? Or, perhaps the signal module should provide a sigaction function,
> so that users can use SA_RESTART.

This is a snippet from a email sent in 2004, I was wondering if there
was any update on this issue. Are these issues supposed to be handled on
a per application basis, or will a fix go into Python in the near
future?

-- 
Hatem Nassrat

Refrences: 
    
    http://mail.python.org/pipermail/python-dev/2004-November/049983.html

From brett at python.org  Mon Jul  6 19:33:25 2009
From: brett at python.org (Brett Cannon)
Date: Mon, 6 Jul 2009 10:33:25 -0700
Subject: [Python-Dev] semi-regular check that all committers are
	subscribed to python-committers
In-Reply-To: <bbaeab100907021308i64cb8262ma72e673ab33c7464@mail.gmail.com>
References: <bbaeab100907021204r56247c41l6913c8408854567@mail.gmail.com> 
	<ea2499da0907021256sbd69effo7239cd7c89e8fba6@mail.gmail.com> 
	<bbaeab100907021308i64cb8262ma72e673ab33c7464@mail.gmail.com>
Message-ID: <bbaeab100907061033g71bdb760qdfc026d213dfd8f2@mail.gmail.com>

Since no one objected, I will give Dirkjan commit privs so he can work on
the hg migration. Now I just need you, Dirkjan, to get a contributor
agreement sent in and then send python-dev your SSH2 key and we will get you
set up.
On Thu, Jul 2, 2009 at 13:08, Brett Cannon <brett at python.org> wrote:

>
>
> On Thu, Jul 2, 2009 at 12:56, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
>
>> On Thu, Jul 2, 2009 at 21:04, Brett Cannon<brett at python.org> wrote:
>> > If you have commit privileges on Python but are not subscribed to
>> > python-committers, please let me know and I will subscribe you.
>> Membership
>> > on python-committers is mandatory as that's where we announce branch
>> > freezes, etc.
>>
>> Would it be helpful if I was subscribed? I'm not strictly a committer,
>> but what with the Mercurial migration I imagine it won't be long
>> before the first cset bearing my name appears in the Python repository
>> (plus I just saw there was some discussion about the hg migration
>> there which I could weigh in on).
>
>
>  Maybe you should just get the commit privileges now. Any objections?
>
> -Brett
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090706/afc77a30/attachment.htm>

From aahz at pythoncraft.com  Mon Jul  6 21:29:45 2009
From: aahz at pythoncraft.com (Aahz)
Date: Mon, 6 Jul 2009 12:29:45 -0700
Subject: [Python-Dev] Any updates on this subprocess/signal
	bug/issue	(Re: subprocess and EINTR errnos)
In-Reply-To: <20090706161638.GA16498@gmail.com>
References: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
	<20090706161638.GA16498@gmail.com>
Message-ID: <20090706192945.GA17222@panix.com>

On Mon, Jul 06, 2009, Hatem Nassrat wrote:
> On Wed, 17 Nov 2004, Peter Astrand wrote,
>>
>> I've made a patch (attached) to subprocess.py (and test_subprocess.py)
>> that should guard against EINTR, but I haven't committed it yet. It's
>> quite large.
>> 
>> Are Python modules supposed to handle EINTR? Why not let the C code handle
>> this? Or, perhaps the signal module should provide a sigaction function,
>> so that users can use SA_RESTART.
> 
> This is a snippet from a email sent in 2004, I was wondering if there
> was any update on this issue. Are these issues supposed to be handled on
> a per application basis, or will a fix go into Python in the near
> future?

For starters, if there is to be any progress, there needs to be an open
issue on bugs.python.org -- have you searched to see if one already
exists and created one if it doesn't?
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha

From pje at telecommunity.com  Mon Jul  6 22:55:38 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 06 Jul 2009 16:55:38 -0400
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.co
 m>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
	<20090706173730.A09F13A407B@sparrow.telecommunity.com>
	<79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com>
Message-ID: <20090706205237.014663A407B@sparrow.telecommunity.com>

At 07:15 PM 7/6/2009 +0100, Paul Moore wrote:
>My point was that distributions which use setuptools in their
>setup.py, *even if there's no runtime dependency on setuptools* end up
>with non-standard .egg-info's. There is no good reason for this, from
>my POV as a package user. So if setuptools is breaking otherwise
>entirely standard packages, so that they won't work with core
>distutils mechanisms, that's wrong. Setuptools is refusing to play
>nicely with other packaging systems.
>
>Now at the moment, this is a moot point, because the stdlib doesn't
>have any package discovery mechanisms.

Not so; it has .egg-info.  (See footnote below.)


>The PEP 302 hooks I'm writing provide the "plugin" ability for
>alternative storage formats. But there's no provision for alternative
>*metadata* formats. What's the point of a standard if people don't
>follow it?

I'm confused.  Are you saying pkgutil will recognize .egg-info, or not?

The only practical difference between the existing formats and the 
proposed new one (AFAICR) is the extension of the directory name, and 
the presence or absence of the RECORD file.

My POV is that it would be a shame to discard compatibility with 
existing installed software, just for a bikeshedding of the directory 
name.  Especially if it means that any tools that currently generate 
.egg-info's (and I'm not 100% positive setuptools is the only case) 
will need to grow options for whether to generate the old name, 
possibly based on what Python version...  except you can install new 
distutils into old Python... etc.

If this were strictly a matter of, "from Python 3.x on, generate 
.pydist", it's a lot less of a problem for me and everybody else, 
because Python 3 is effectively going to force a fork (or at least a 
branch) of setuptools anyway.

For Python 2.x, having *two* names for .egg-info (which is what I 
understand the proposal to effectively be) is just asking for trouble, IMO.

And my concern is that currently these compatibility issues are being 
handwaved rather than considered up front.

To reiterate, summarize, and attempt to make as concrete as I can:

I'm fine with a rename, *if and only if* it can be done in a way that 
never requires a tool to generate BOTH files for compatibility 
reasons.  AFAICT, this can only be done by making the extension 
dependent upon the Python version, and this version distinction must 
be made *even if* the distutils themselves are backported to an 
earlier Python version.  (Alternatively, this could be resolved by 
*not* backporting the PEP 376-compatible distutils/pkgutil to 2.x.)

Currently, there is no official Python 3 release of setuptools, so 
compatibility can (IMO) be considered moot there.  What's more, it is 
easy to conditionally determine whether to use '.egg-info' or 
'.pydist' based on Python version.  What is not easy and likely to be 
confusing for everyone, is having the possibility for *both* to exist 
in the same system, *without* also having bidirectional interoperability.

So, I'm +1 for no change (obviously), and +0 for "always .pydist in 
3.x and always .egg-info in 2.x", whether the latter part is achieved 
by making distutils/pkgutil use a version-dependent extension, or by 
refusing to backport distutils/pkgutil to 2.x.  I'm -1 for having 
.pydist ever appear on a 2.x Python, either via stdlib or backporting.

(These comments also apply for any name other than '.pydist'.)

Footnote: there has been some discussion that implies that somehow 
.egg-info was hijacked into the stdlib, and that it should therefore 
not be supported.  This is incorrect: at Pycon the year this happened 
(2005 IIRC), Guido *blessed setuptools for inclusion in the stdlib* 
in his keynote.  Yes, *all* of setuptools, including easy_install, 
eggs, the whole nine yards.

A few months later, when I actually checked in the code, there were 
protests from a few people on Python-Dev, some of which I agreed were 
valid.  I reverted the checkin, floating instead a compromise 
proposal to add .egg-info support to distutils (i.e. the 
install_egg_info command) to allow setuptools and other 3rd party 
tools to detect the presence of distutils-installed packages.

At the time, the assumption was that eggs and setuptools would still, 
eventually, make it into the stdlib -- at the time, Guido had not yet 
changed his mind about that.  So the ".egg-info" name was not 
controversial, to the best of my recollection, and the inclusion did 
not violate any of the expressed concerns of the three persons who 
objected to including setuptools whole-hog.

So, from my POV at least, dropping .egg-info is dropping backward 
compatibility on a feature that was *intentionally* added to the 
stdlib -- with approval from Guido and non-protest from Python-Dev -- 
specifically to provide that compatibility.

That doesn't obligate it to stay...  but I suggest that it *does* 
mean Python 3, not 2, is the appropriate place for dropping it (if it 
is indeed to be dropped)...  and that it is not appropriate to do so 
in 2.x on merely a handwave by persons not involved in the previous 
discussions.  (And of course Guido can Pronounce otherwise; I'm just 
saying I don't think anything less than such a pronouncement is 
required to remove it from 2.x.  And again, I'm fine with making a 
clean break of it in 3.x, since setuptools is already broken there anyway.)


From pje at telecommunity.com  Mon Jul  6 23:07:05 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 06 Jul 2009 17:07:05 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.co
 m>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
Message-ID: <20090706210405.4BFD63A407B@sparrow.telecommunity.com>

At 07:38 PM 7/6/2009 +0100, Paul Moore wrote:
>As promised, here are some open questions on PEP 376.
>
>- Will the public API names be changed from *egginfo* to *metadata*?

+1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of 
stuff as.)


>- What precisely are the use cases for absolute path names? Concrete
>examples are needed. With the current spec, some things can go wrong
>(e.g., see below), so we need real use cases to know how to address
>this.

The purpose is to be able to remove scripts, data files, 
documentation, etc. that's stored in some place that's not 
package-relative, and survive a move of the package directory itself.

For example, suppose I install with --prefix=~, and have files in 
~/bin and ~/lib.  I then move ~/lib/python2.7 to ~/pylib.  If the 
scripts were installed with a relative path, this would break the 
link to the script still living in ~/bin.

Now that I'm thinking about this, it occurs to me that there is 
another way to address this besides mixing relative and absolute 
paths in RECORD, and that is to make *all* the paths relative, and 
include an original installation location in INSTALLER.  Then, it 
would be possible to detect whether the installation location had 
been moved and handle it accordingly...  for some definition of "accordingly".

I agree that further discussion is probably needed to resolve this.


>- How will bdist_wininst/bdist_msi/bdist_rpm be updated?

bdist_wininst, bdist_dumb, and various others use 'install --root' 
pattern to generate files for installation, which means that they 
would transparently end up writing a correct RECORD file, except for 
the inclusion of incorrect absolute paths for non-libdir-relative 
files.  However, if we used the "all relative in RECORD, with a base 
in INSTALLER", these cases could transparently be treated as another 
instance of install directory relocation.

I don't know if bdist_msi does a --root install before generating the 
.msi; if it does, then it should work the same way.


>- Can distutils be made to install files in places the current RECORD
>file spec can't handle? (I think the answer is "yes"). What happens
>then?

Surely an absolute path is sufficient?


>- Should distribution names be case insensitive on case insensitive
>filesystems? For comparison, module/package names are always case
>sensitive even on case insensitive systems.

IMO, they should be case-insensitive *regardless* of filesystem case, 
and I believe PyPI has already begun disallowing packages with only 
case-distinct names (e.g. 'Zope' vs. 'ZoPe') to be registered.


From ncoghlan at gmail.com  Mon Jul  6 23:22:25 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 07:22:25 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A520F83.7020300@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com>
Message-ID: <4A526B11.7060308@gmail.com>

M.-A. Lemburg wrote:
> Dirkjan Ochtman wrote:
>  * Our tracker, checkin messages, online documentation and lots of
>    Python resources on the web are full of references to the
>    Subversion rXYZ notation of changesets.
> 
>    The PEP has to provide some way to gracefully provide a redirect
>    from those URLs to the new ones (both for the source code browser
>    and the bug search tool on python.org).
> 
>    The PEP will also have to address the same issue for checkin
>    messages. Perhaps by converting the rXYZ notation to the hash
>    based values during migration or by adding the has value after
>    the rXYZ value in parens ?!

I've asked this question before: Dirkjan indicated that he will be
writing a web redirection filter that translates from the SVN rXYZ
revision numbers to the Hg hash value that corresponds to that revision
in the history conversion.

I suggest that we also run a version of that redirection filter to remap
the old svn.python.org links in addition to making them accessible
through hg.python.org as Dirkjan proposed.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2009-July/090397.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Jul  6 23:43:12 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 07:43:12 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <20090706205237.014663A407B@sparrow.telecommunity.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
	<20090706173730.A09F13A407B@sparrow.telecommunity.com>
	<79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com>
	<20090706205237.014663A407B@sparrow.telecommunity.com>
Message-ID: <4A526FF0.10204@gmail.com>

P.J. Eby wrote:
> So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x
> and always .egg-info in 2.x", whether the latter part is achieved by
> making distutils/pkgutil use a version-dependent extension, or by
> refusing to backport distutils/pkgutil to 2.x.  I'm -1 for having
> .pydist ever appear on a 2.x Python, either via stdlib or backporting.
> 
> (These comments also apply for any name other than '.pydist'.)

Thanks for the explanation - Before this message I didn't fully grasp
*how* installing a directory containing files that the other packages
might not understand would help with the backwards compatibility problem.

Now I understand that aspect better (i.e. it isn't a matter of
"setuptools won't change" but "the user might already have an old
version of setuptools/pip/etc on their system, or be installing a
package that uses an old version of one of those libraries for its
installer"), I no longer think it is worth the hassle of trying to use a
"prettier" name for the metadata directory. So +1 on keeping the same
extension name and -1 on the idea of changing it to anything other than
the existing ".egg-info".

I believe the idea of having different names in 2.x and 3.x would likely
cause too many problems for simple bdist_* installers of modules that
use only the common 2.x/3.x language subset, so I'm also -1 on that idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Jul  6 23:54:58 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 07:54:58 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
Message-ID: <4A5272B2.8000202@gmail.com>

(I cancelled sending this the first time, so apologies if a half-written
version turns up)

Paul Moore wrote:
> As promised, here are some open questions on PEP 376.

I'd add one more question to the list: is allowing backslash separated
names in the RECORD file actually a good idea, or would it be better to
always use forward slashes?

My reason for this question is what happens if (for example) a bdist_*
installer is generated on Linux and then used on Windows or vice-versa?
If the expected RECORD format is different on the two platforms, won't
it get things wrong?

For the other questions, I don't have anything much to add to PJE's
comments, except that the "all relative" paths idea won't work due to
the Windows drive letter issue (i.e. if an installer puts files in
C:\Program Files, there is no guarantee that a relative path between
site-packages and Program Files even exists if Python is installed on a
different drive).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Mon Jul  6 23:57:55 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 06 Jul 2009 23:57:55 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A526B11.7060308@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>
	<4A526B11.7060308@gmail.com>
Message-ID: <4A527363.7070305@v.loewis.de>

> I suggest that we also run a version of that redirection filter to remap
> the old svn.python.org links in addition to making them accessible
> through hg.python.org as Dirkjan proposed.

Not sure what links you are talking about, but we also need to keep the
current svn.python.org up as-is, for all the stuff that won't be migrated.

Regards,
Martin

From ncoghlan at gmail.com  Tue Jul  7 00:05:08 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 08:05:08 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A527363.7070305@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>
	<4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de>
Message-ID: <4A527514.5040709@gmail.com>

Martin v. L?wis wrote:
>> I suggest that we also run a version of that redirection filter to remap
>> the old svn.python.org links in addition to making them accessible
>> through hg.python.org as Dirkjan proposed.
> 
> Not sure what links you are talking about, but we also need to keep the
> current svn.python.org up as-is, for all the stuff that won't be migrated.

In that case, no need to redirect anything (although it will still be
possible to look up the old SVN revisions in the Hg web view).

The previous discussion was based on the assumption that it was going to
be possible to switch the SVN server off at some point rather than just
disallowing commits to the projects that had migrated to Mercurial.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From p.f.moore at gmail.com  Tue Jul  7 00:16:56 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 23:16:56 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A526FF0.10204@gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
	<20090706173730.A09F13A407B@sparrow.telecommunity.com>
	<79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com>
	<20090706205237.014663A407B@sparrow.telecommunity.com>
	<4A526FF0.10204@gmail.com>
Message-ID: <79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com>

2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> P.J. Eby wrote:
>> So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x
>> and always .egg-info in 2.x", whether the latter part is achieved by
>> making distutils/pkgutil use a version-dependent extension, or by
>> refusing to backport distutils/pkgutil to 2.x. ?I'm -1 for having
>> .pydist ever appear on a 2.x Python, either via stdlib or backporting.
>>
>> (These comments also apply for any name other than '.pydist'.)
>
> Thanks for the explanation - Before this message I didn't fully grasp
> *how* installing a directory containing files that the other packages
> might not understand would help with the backwards compatibility problem.
>
> Now I understand that aspect better (i.e. it isn't a matter of
> "setuptools won't change" but "the user might already have an old
> version of setuptools/pip/etc on their system, or be installing a
> package that uses an old version of one of those libraries for its
> installer"), I no longer think it is worth the hassle of trying to use a
> "prettier" name for the metadata directory. So +1 on keeping the same
> extension name and -1 on the idea of changing it to anything other than
> the existing ".egg-info".
>
> I believe the idea of having different names in 2.x and 3.x would likely
> cause too many problems for simple bdist_* installers of modules that
> use only the common 2.x/3.x language subset, so I'm also -1 on that idea.

That suits me too. I'm happy for the name to stay as it is.

Paul.

From p.f.moore at gmail.com  Tue Jul  7 00:36:08 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 23:36:08 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090706210405.4BFD63A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090706210405.4BFD63A407B@sparrow.telecommunity.com>
Message-ID: <79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com>

2009/7/6 P.J. Eby <pje at telecommunity.com>:
> At 07:38 PM 7/6/2009 +0100, Paul Moore wrote:
>>
>> As promised, here are some open questions on PEP 376.
>>
>> - Will the public API names be changed from *egginfo* to *metadata*?
>
> +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff
> as.)

Thanks. I think this one is pretty non-controversial.

>> - What precisely are the use cases for absolute path names? Concrete
>> examples are needed. With the current spec, some things can go wrong
>> (e.g., see below), so we need real use cases to know how to address
>> this.
>
> The purpose is to be able to remove scripts, data files, documentation, etc.
> that's stored in some place that's not package-relative, and survive a move
> of the package directory itself.
>
> For example, suppose I install with --prefix=~, and have files in ~/bin and
> ~/lib. ?I then move ~/lib/python2.7 to ~/pylib. ?If the scripts were
> installed with a relative path, this would break the link to the script
> still living in ~/bin.
>
> Now that I'm thinking about this, it occurs to me that there is another way
> to address this besides mixing relative and absolute paths in RECORD, and
> that is to make *all* the paths relative, and include an original
> installation location in INSTALLER. ?Then, it would be possible to detect
> whether the installation location had been moved and handle it
> accordingly... ?for some definition of "accordingly".
>
> I agree that further discussion is probably needed to resolve this.

I'm not 100% sure I follow your logic here. If you're renaming
directories after install, without involving the package manager, I
think you probably deserve whatever you get :-)

Having said that, your proposal (all paths relative, plus an install
location) is similar to something I mentioned earlier (but better
explained :-)) The only case it doesn't cater for is when an installed
file can't be expressed as relative to the "install location" (which
would involve being on a different drive on Windows). That's likely to
never work, so I doubt it counts in any case. I've got some concerns
about the possibility of having locations "above" the install location
- something like "../../scripts/driver.bat". I'm not sure if such
things will work (consider if the install location is a zipfile or
egg) but nor am I sure anything generates such names.

As you say, more discussion (and use cases!) would help here.

>> - How will bdist_wininst/bdist_msi/bdist_rpm be updated?
>
> bdist_wininst, bdist_dumb, and various others use 'install --root' pattern
> to generate files for installation, which means that they would
> transparently end up writing a correct RECORD file, except for the inclusion
> of incorrect absolute paths for non-libdir-relative files. ?However, if we
> used the "all relative in RECORD, with a base in INSTALLER", these cases
> could transparently be treated as another instance of install directory
> relocation.
>
> I don't know if bdist_msi does a --root install before generating the .msi;
> if it does, then it should work the same way.

Yes, that sounds like a plausible approach.

>> - Can distutils be made to install files in places the current RECORD
>> file spec can't handle? (I think the answer is "yes"). What happens
>> then?
>
> Surely an absolute path is sufficient?

Current spec says relative or $PREFIX/... or $EXEC_PREFIX/...

I think that was based on earlier discussion, but it's hampered by the
fact that nobody's pointing at a real case of a distribution that
actually tries to *install* files in odd places. (Actually, something
that installs a driver script, like twisted's twistd or nose's
nosetests, would do - I'll see what I can find tomorrow).

>> - Should distribution names be case insensitive on case insensitive
>> filesystems? For comparison, module/package names are always case
>> sensitive even on case insensitive systems.
>
> IMO, they should be case-insensitive *regardless* of filesystem case, and I
> believe PyPI has already begun disallowing packages with only case-distinct
> names (e.g. 'Zope' vs. 'ZoPe') to be registered.

OK, that's another option :-) It's not hard to change the code, so as
long as a final decision is reached at some point, we're OK here...

Thanks for the comments,
Paul.

From p.f.moore at gmail.com  Tue Jul  7 00:44:09 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Jul 2009 23:44:09 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A5272B2.8000202@gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
Message-ID: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>

2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> I'd add one more question to the list: is allowing backslash separated
> names in the RECORD file actually a good idea, or would it be better to
> always use forward slashes?

They do always use forward slashes.

> For the other questions, I don't have anything much to add to PJE's
> comments, except that the "all relative" paths idea won't work due to
> the Windows drive letter issue (i.e. if an installer puts files in
> C:\Program Files, there is no guarantee that a relative path between
> site-packages and Program Files even exists if Python is installed on a
> different drive).

The big question, though, is can an installer actually *do* that in
practical terms?

- There are *no* guaranteed absolute locations on Windows, so any such
oddly-located file would require user interaction to work. Certainly
bdist_wininst and bdist_msi don't do that.
- My experiments indicate that bdist_{wininst,msi} are broken with
respect to absolute paths anyway: they do a --root install to a
temporary directory (and the absolute paths don't end up in there) and
then package up that temporary directory.

I still want to see a real life example that demonstrates that there
is a genuine issue here. We're spending a lot of energy and complexity
trying to design a solution to a problem that actually doesn't appear
to exist in practice...

(To be honest, I'd be fairly confident in saying that absolute paths
can be ignored on Windows, subject to some corner cases that I haven't
thought through yet. My worry is that I don't know what Unix and Mac
users might do, so I can't just wish away the issue because it can't
arise on Windows. Can a Unix/Mac user offer a real-world example on
their own system?)

Paul.

From ben+python at benfinney.id.au  Tue Jul  7 03:45:19 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 07 Jul 2009 11:45:19 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
Message-ID: <87ws6lp8pc.fsf@benfinney.id.au>

Paul Moore <p.f.moore at gmail.com> writes:

> In fact, the above strongly suggests to me that PEP 376 must NOT
> follow setuptools conventions - otherwise, it will end up clashing
> with the files setuptools is putting on my system.

I think the argument for a separate ?.pydist? metadata directory,
normative in PEP 376, is a good one.

> > If so, I'd still prefer to keep the new metadata safely out of reach
> > of the legacy package management systems that don't understand it,
> > while having distutils retain the ability to generate a legacy
> > ".egginfo" file to warn off the existing package management tools
> > from installing the same distribution again.
> 
> So for a minimal case of a single .py file packaged up as a
> distribution, we now have the .py file, a legacy .egginfo file, a new
> .pydist directory containing RECORD and PKG-INFO files?
> 
> That's getting silly.

The burden of switching is that we need a compatibility bridge. If that
means that a tool must support the old while also supporting the new,
that's not silliness in the tool, but acknowledgement of the silliness
we're migrating away from.

That doesn't mean, though, that PEP 376 needs to mention the old format
at all; just that distutils should support it, deprecated, until a time
when it's deemed safe to remove.

In fact, I think it's best that PEP 376 discuss *only* the format we're
trying to migrate to, and support for existing formats is firmly outside
its scope.

> And yet, given that PEP 376 is explicitly being designed with a goal
> being to act as the common standard that *all* package management
> formats can use, is it not the whole point that once it's defined and
> we have achieved consensus, existing package managers will switch to
> using it rather than retaining a range of custom formats?

Again, that needs to be a gradual process. Supporting a new format as
defined in PEP 376 needs not (indeed, I would argue *should* not) mean
the tool drop support for old formats immediately. There needs to be a
compatibility chain to allow for fast migration *and* slow migration,
since there will be a broad mix of both.

Let PEP 376 discuss a format that is clean and new, but let discussions
leading to that format acknowledge that it must allow (*not* require)
parallel support of old formats for some time.

-- 
 \       ?I love to go down to the schoolyard and watch all the little |
  `\   children jump up and down and run around yelling and screaming. |
_o__)             They don't know I'm only using blanks.? ?Emo Philips |
Ben Finney


From p.f.moore at gmail.com  Tue Jul  7 09:15:19 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 08:15:19 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <87ws6lp8pc.fsf@benfinney.id.au>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
Message-ID: <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>

2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
> Paul Moore <p.f.moore at gmail.com> writes:
>
>> In fact, the above strongly suggests to me that PEP 376 must NOT
>> follow setuptools conventions - otherwise, it will end up clashing
>> with the files setuptools is putting on my system.
>
> I think the argument for a separate ?.pydist? metadata directory,
> normative in PEP 376, is a good one.

I've been convinced otherwise. There's no long-term benefit (the
directory layout and format is the same, so it's a purely cosmetic
change) and we're doing no favours to anyone just adding a new format
for the sake of it.

Paul.

From ziade.tarek at gmail.com  Tue Jul  7 09:21:39 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 09:21:39 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<20090706165546.25CCB3A407B@sparrow.telecommunity.com>
	<94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com>
	<20090706173730.A09F13A407B@sparrow.telecommunity.com>
	<79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com>
	<20090706205237.014663A407B@sparrow.telecommunity.com>
	<4A526FF0.10204@gmail.com>
	<79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com>
Message-ID: <94bdd2610907070021s4d90dd4fw55e5961e597b1afb@mail.gmail.com>

On Tue, Jul 7, 2009 at 12:16 AM, Paul Moore<p.f.moore at gmail.com> wrote:
>>
>> I believe the idea of having different names in 2.x and 3.x would likely
>> cause too many problems for simple bdist_* installers of modules that
>> use only the common 2.x/3.x language subset, so I'm also -1 on that idea.
>
> That suits me too. I'm happy for the name to stay as it is.

While I'd rather like having an "egg"-free standard, it's a detail
compared to the rest,
so I'm 0+

From ziade.tarek at gmail.com  Tue Jul  7 09:30:39 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 09:30:39 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090706210405.4BFD63A407B@sparrow.telecommunity.com>
	<79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com>
Message-ID: <94bdd2610907070030vfe98bc8jeb4503a6f40c5bc1@mail.gmail.com>

2009/7/7 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/6 P.J. Eby <pje at telecommunity.com>:
>> At 07:38 PM 7/6/2009 +0100, Paul Moore wrote:
>>>
>>> As promised, here are some open questions on PEP 376.
>>>
>>> - Will the public API names be changed from *egginfo* to *metadata*?
>>
>> +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff
>> as.)
>
> Thanks. I think this one is pretty non-controversial.

+1, I'll change the PEP accordingly;


>>> - How will bdist_wininst/bdist_msi/bdist_rpm be updated?
>>
>> bdist_wininst, bdist_dumb, and various others use 'install --root' pattern
>> to generate files for installation, which means that they would
>> transparently end up writing a correct RECORD file, except for the inclusion
>> of incorrect absolute paths for non-libdir-relative files. ?However, if we
>> used the "all relative in RECORD, with a base in INSTALLER", these cases
>> could transparently be treated as another instance of install directory
>> relocation.
>>
>> I don't know if bdist_msi does a --root install before generating the .msi;
>> if it does, then it should work the same way.
>
> Yes, that sounds like a plausible approach.

As I said earlier in the previous thread, all these commands rely on a
call to the install
command, so bdist_msi does too.

From mal at egenix.com  Tue Jul  7 09:50:34 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 09:50:34 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks
 to	provide distribution metadata
In-Reply-To: <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>	<4A4EA689.1010704@gmail.com>	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>	<20090705183326.097CF3A4099@sparrow.telecommunity.com>	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>	<4A520BDF.80503@gmail.com>	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
Message-ID: <4A52FE4A.6020204@egenix.com>

Paul Moore wrote:
> 2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
>> Paul Moore <p.f.moore at gmail.com> writes:
>>
>>> In fact, the above strongly suggests to me that PEP 376 must NOT
>>> follow setuptools conventions - otherwise, it will end up clashing
>>> with the files setuptools is putting on my system.
>> I think the argument for a separate ?.pydist? metadata directory,
>> normative in PEP 376, is a good one.
> 
> I've been convinced otherwise. There's no long-term benefit (the
> directory layout and format is the same, so it's a purely cosmetic
> change) and we're doing no favours to anyone just adding a new format
> for the sake of it.

The PEP should take the chance to define not only the
directory, but also the complete set of files in there and
their format.

A typical setuptools dir looks like this:

dependency_links.txt
namespace_packages.txt
not-zip-safe
PKG-INFO
requires.txt
SOURCES.txt
top_level.txt

Ie. a complete mess of upper-case/lower-case names, names with
underscores or hyphens as word separator, files with extensions,
files without them.

This needs some serious cleanup.

A new dir name will allow to do this, so +1 on a new dir name.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Tue Jul  7 09:50:50 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 09:50:50 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
Message-ID: <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>

2009/7/7 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
>> I'd add one more question to the list: is allowing backslash separated
>> names in the RECORD file actually a good idea, or would it be better to
>> always use forward slashes?
>
> They do always use forward slashes.
>
>> For the other questions, I don't have anything much to add to PJE's
>> comments, except that the "all relative" paths idea won't work due to
>> the Windows drive letter issue (i.e. if an installer puts files in
>> C:\Program Files, there is no guarantee that a relative path between
>> site-packages and Program Files even exists if Python is installed on a
>> different drive).
>
> The big question, though, is can an installer actually *do* that in
> practical terms?
>
> - There are *no* guaranteed absolute locations on Windows, so any such
> oddly-located file would require user interaction to work. Certainly
> bdist_wininst and bdist_msi don't do that.
> - My experiments indicate that bdist_{wininst,msi} are broken with
> respect to absolute paths anyway: they do a --root install to a
> temporary directory (and the absolute paths don't end up in there) and
> then package up that temporary directory.

yes that's unfortunately the case for all windows-based installation.
wether it's a bdist call to the install command, to create a binary package,
wether it's an installation.

c:\something or d:\something will be installed in sys.prefix\something.

I will add an issue for distutils for this, probably ending up raising
an exception when we hit this case, because I don't see how these
paths can work.

Unless we define a "drive that contains the python installation" maybe, or
the "Program Files" directory

would that  make sense from a win32 point of view ?


>
> I still want to see a real life example that demonstrates that there
> is a genuine issue here. We're spending a lot of energy and complexity
> trying to design a solution to a problem that actually doesn't appear
> to exist in practice...
>
> (To be honest, I'd be fairly confident in saying that absolute paths
> can be ignored on Windows, subject to some corner cases that I haven't
> thought through yet. My worry is that I don't know what Unix and Mac
> users might do, so I can't just wish away the issue because it can't
> arise on Windows. Can a Unix/Mac user offer a real-world example on
> their own system?)
>

I know some people are writing to /etc to add their configuration file
on the system,

So a real-world example under linux would be:

setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)


That is basically how the examples are shown at:

http://docs.python.org/distutils/setupscript.html#installing-additional-files

But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.

So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX
would be to nominate a list of variables that every python environment
has (querying
modules like sys) and let the developers use them as root locations
for some files.

- sys.prefix
- sys.exec_prefix
- some elements returned by distutils.sysconfig.get_config_vars()
(distutils.sysconfig queries the python Makefile amongst others)
- ...

Semi-related: distutils.sysconfig should be removed from distutils,
and be a standalone module in the sdtlib for example.

Regards
Tarek

From mal at egenix.com  Tue Jul  7 10:16:19 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 10:16:19 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A527514.5040709@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>	<4A526B11.7060308@gmail.com>
	<4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com>
Message-ID: <4A530453.10005@egenix.com>

Nick Coghlan wrote:
> Martin v. L?wis wrote:
>>> I suggest that we also run a version of that redirection filter to remap
>>> the old svn.python.org links in addition to making them accessible
>>> through hg.python.org as Dirkjan proposed.
>> Not sure what links you are talking about, but we also need to keep the
>> current svn.python.org up as-is, for all the stuff that won't be migrated.

Good point. So the old svn revision links will continue to work.

> In that case, no need to redirect anything (although it will still be
> possible to look up the old SVN revisions in the Hg web view).
> 
> The previous discussion was based on the assumption that it was going to
> be possible to switch the SVN server off at some point rather than just
> disallowing commits to the projects that had migrated to Mercurial.

Is there a standard notation for hg revisions that roundup could
detect and turn into links (much like it does for svn) ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ronaldoussoren at mac.com  Tue Jul  7 10:17:17 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 07 Jul 2009 10:17:17 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com>
	<4A4EA689.1010704@gmail.com>
	<bbaeab100907041254q7f8a63b4he17f910e76ff3776@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<4A5116DC.1070109@gmail.com>
	<31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com>
	<94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com>
Message-ID: <18234706-FD64-453F-BFBD-D084D5CC0B1C@mac.com>


On 6 Jul, 2009, at 9:53, Tarek Ziad? wrote:

> 2009/7/6 Ronald Oussoren <ronaldoussoren at mac.com>:
>> I'm -1 on changing the name. For better or worse setuptools is the  
>> elephant
>> in the room w.r.t. package management and it would IMHO be better  
>> to stay
>> compatible (even if the stdlib only implements a subset of
>> setuptools/pkg_resources)
>>
>
> I'd rather see the elephant evolves.
>
> I don't see why we should bend a standard we want to introduce in  
> the stdlib,
> for a third-party package that is able to evolve to stick to a new  
> standard
> without any problem.

But why break existing code without having any other benifits?  If I  
read the discussion correctly the name would be changed without any  
changes to the contents of the metadata directory.  I would be more  
inclined to be in favour if the name change had a sound technical  
reason, such as a change of the contents of the directory which would  
make setuptools "egg-info" directories incompatible with the PEP376  
ones.

Ronald


From dirkjan at ochtman.nl  Tue Jul  7 10:30:30 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 7 Jul 2009 10:30:30 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A530453.10005@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com>
	<4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com>
	<4A530453.10005@egenix.com>
Message-ID: <ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>

On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg<mal at egenix.com> wrote:
> Is there a standard notation for hg revisions that roundup could
> detect and turn into links (much like it does for svn) ?

[a-f0-9]{12} should mostly do.

(Sorry for my absence from the discussion for some time. I'll try to
update the PEP and summarize ongoing discussions here soon.)

Cheers,

Dirkjan

From p.f.moore at gmail.com  Tue Jul  7 10:33:55 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 09:33:55 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
Message-ID: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>

2009/7/7 Tarek Ziad? <ziade.tarek at gmail.com>:
> Unless we define a "drive that contains the python installation" maybe, or
> the "Program Files" directory
>
> would that ?make sense from a win32 point of view ?

I can't imagine that it would be useful in practice.

> I know some people are writing to /etc to add their configuration file
> on the system,
>
> So a real-world example under linux would be:
>
> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
>
>
> That is basically how the examples are shown at:
>
> http://docs.python.org/distutils/setupscript.html#installing-additional-files

Thanks. Yes, that makes for a good example.

> But this is already os-specific, and exists because distutils doesn't
> have a way (yet)
> to express systems locations independantly from their physical location,
> like what the RPM system does with %VARIABLES.
>
> So another way to handle this maybe, like I have added with $PREFIX
> and $EXEC_PREFIX would be to nominate a list of variables that every
> python environment has (querying modules like sys) and let the developers
> use them as root locations for some files.
>
> - sys.prefix
> - sys.exec_prefix
> - some elements returned by distutils.sysconfig.get_config_vars()
> (distutils.sysconfig queries the python Makefile amongst others)
> - ...

This adds extra complexity to the RECORD format, for little practical
benefit that I can see.

>From the POV of core distutils:
- Windows users use bdist_wininst and/or bdist_msi, and absolute
locations are a lost cause.
- Presumably, some people use bdist_rpm (I don't know if there are
other ways of creating RPMs).
- Everyone else uses setup.py install or a 3rd party tool.
- PEP 302 style loaders aren't relevant as the core only uses the
filesystem (not even zip files).
- Only 3rd party tools will consume this data.

So, we need input from developers of 3rd party tools here. Phillip has
stated the case for setuptools, from his POV having everything
relative to the "install location" which is stored elsewhere (in the
installer file) is fine. I'd like to know whether he needs
"upwards-pointing" relative paths like ../../../../xx.py, but that's a
small detail.

So - do any other potential users of the PEP 376 metadata want to speak up?

At the moment, it feels like we're designing things more or less in a vacuum.

Paul.

From ziade.tarek at gmail.com  Tue Jul  7 10:40:00 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 10:40:00 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
Message-ID: <94bdd2610907070140g73e787a7rd171a6d828621efa@mail.gmail.com>

On Tue, Jul 7, 2009 at 10:33 AM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/7 Tarek Ziad? <ziade.tarek at gmail.com>:
>> Unless we define a "drive that contains the python installation" maybe, or
>> the "Program Files" directory
>>
>> would that ?make sense from a win32 point of view ?
>
> I can't imagine that it would be useful in practice.
>
>> I know some people are writing to /etc to add their configuration file
>> on the system,
>>
>> So a real-world example under linux would be:
>>
>> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
>>
>>
>> That is basically how the examples are shown at:
>>
>> http://docs.python.org/distutils/setupscript.html#installing-additional-files
>
> Thanks. Yes, that makes for a good example.
>
>> But this is already os-specific, and exists because distutils doesn't
>> have a way (yet)
>> to express systems locations independantly from their physical location,
>> like what the RPM system does with %VARIABLES.
>>
>> So another way to handle this maybe, like I have added with $PREFIX
>> and $EXEC_PREFIX would be to nominate a list of variables that every
>> python environment has (querying modules like sys) and let the developers
>> use them as root locations for some files.
>>
>> - sys.prefix
>> - sys.exec_prefix
>> - some elements returned by distutils.sysconfig.get_config_vars()
>> (distutils.sysconfig queries the python Makefile amongst others)
>> - ...
>
> This adds extra complexity to the RECORD format, for little practical
> benefit that I can see.
>
> From the POV of core distutils:
> - Windows users use bdist_wininst and/or bdist_msi, and absolute
> locations are a lost cause.
> - Presumably, some people use bdist_rpm (I don't know if there are
> other ways of creating RPMs).

They are othe ways to generate RPMs.

There are also debian packaging scripts

> - Everyone else uses setup.py install or a 3rd party tool.
> - PEP 302 style loaders aren't relevant as the core only uses the
> filesystem (not even zip files).
> - Only 3rd party tools will consume this data.
>
> So, we need input from developers of 3rd party tools here. Phillip has
> stated the case for setuptools, from his POV having everything
> relative to the "install location" which is stored elsewhere (in the
> installer file) is fine. I'd like to know whether he needs
> "upwards-pointing" relative paths like ../../../../xx.py, but that's a
> small detail.
>
> So - do any other potential users of the PEP 376 metadata want to speak up?
>
> At the moment, it feels like we're designing things more or less in a vacuum.

I am CC'ing the people that worked with us for the versionning matters,
they can speak from a Fedora and Ubuntu POV, I am also adding Jim for
zc.buildout,
he can provide precious input.

(I am sorry if some people get the mail twice)

From ronaldoussoren at mac.com  Tue Jul  7 10:40:12 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 07 Jul 2009 10:40:12 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
Message-ID: <4B578D30-0B20-461F-AFEB-7B23710FE14D@mac.com>


On 6 Jul, 2009, at 20:38, Paul Moore wrote:
>
> - Should distribution names be case insensitive on case insensitive
> filesystems? For comparison, module/package names are always case
> sensitive even on case insensitive systems.

I'd then go for case sensitive names for distributions as well, just  
to be consistent. But isn't the actual name in the PKG-INFO file anyway?

Do you mean really case insensive filesystems like DOS' FAT, or case  
preserving (HFS+ on OSX, NTFS on Windows)? For the latter you can at  
least get the original name back using os.readdir.

BTW. Actually determining if you are working with a case-sensitive  
fileystem requires filesystem access, os.path.normcase is hopelessly  
naieve in that respect.  I'm reguarly dealing with NFS mounted  
filesystems on OSX systems, which means that parts of a path are case  
preserving (the HFS+ filesystem upto the mount point) and other parts  
are truly case sensitive (paths on the NFS mounted filesystem  
originating on a Linux server).

Ronald

From mal at egenix.com  Tue Jul  7 10:57:33 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 10:57:33 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<4A5272B2.8000202@gmail.com>	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
Message-ID: <4A530DFD.40802@egenix.com>

Paul Moore wrote:
>> I know some people are writing to /etc to add their configuration file
>> on the system,
>>
>> So a real-world example under linux would be:
>>
>> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
>>
>>
>> That is basically how the examples are shown at:
>>
>> http://docs.python.org/distutils/setupscript.html#installing-additional-files
> 
> Thanks. Yes, that makes for a good example.
> 
>> But this is already os-specific, and exists because distutils doesn't
>> have a way (yet)
>> to express systems locations independantly from their physical location,
>> like what the RPM system does with %VARIABLES.
>>
>> So another way to handle this maybe, like I have added with $PREFIX
>> and $EXEC_PREFIX would be to nominate a list of variables that every
>> python environment has (querying modules like sys) and let the developers
>> use them as root locations for some files.

I think you have to differentiate between packages and applications.

Packages will usually have their own way of getting configured,
either by passing parameters via some API or pointing the package
to a configuration file. They may come with some example config
files, but should normally don't interfere with the system
configuration. I.e. putting a Python package configuration into /etc
does not really sound like a good idea.

Applications tend to ship everything needed to run the application
together with the installer and typically use a system-dependent
installer rather than a distutils based approach. These then
place config files in the usual default dirs of the system.
This is out-of-scope for PEP 376.

Then you have tools like zc.buildout which basically build
an application in some directory at "install" time. Since only
these tools know what they are doing, the whole "uninstall"
mechanism also lies in their hands. Again, PEP 376 would only
apply to the dynamic package installation part, but not to the
complete application build.

Another aspect to consider is that config files should normally
not be uninstalled automatically. The user should either be asked
whether she wants to keep the files or the uninstaller should leave
them untouched and issue a warning that certain files were not
uninstalled.

Summarizing, I think it's better not to record config files
and other user-edited files in the RECORD file.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From p.f.moore at gmail.com  Tue Jul  7 11:04:04 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 10:04:04 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A52FE4A.6020204@egenix.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
	<4A52FE4A.6020204@egenix.com>
Message-ID: <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>

2009/7/7 M.-A. Lemburg <mal at egenix.com>:
> The PEP should take the chance to define not only the
> directory, but also the complete set of files in there and
> their format.
>
> A typical setuptools dir looks like this:
>
> dependency_links.txt
> namespace_packages.txt
> not-zip-safe
> PKG-INFO
> requires.txt
> SOURCES.txt
> top_level.txt
>
> Ie. a complete mess of upper-case/lower-case names, names with
> underscores or hyphens as word separator, files with extensions,
> files without them.
>
> This needs some serious cleanup.
>
> A new dir name will allow to do this, so +1 on a new dir name.

I agree the current situation is a mess. I believe that everything you
mention is related to setuptools - so essentially, we have the
situation:

- the existing setuptools format is a mess (at least in the opinion of
MAL and me :-))
- PEP 376 is an opportunity to consider cleanup
- there are some strong supporters of keeping things setuptools-compatible
- nobody has come forward with any other real-world use case than setuptools
- consequently, those of us arguing for cleanup are talking theoretically :-(
- Do you (MAL) have any real use for this data? Any non-setuptools use
case, no matter how small, would really help here.

I suspect practicality will beat purity here. Which would be fine, if
the "practical" cases weren't just setuptools.

I understood that there was a lot of interest in a "distutils cleanup"
from the packaging community. And yet so far in this discussion, the
main participants have seemed to be (apologies to anyone if I've
misunderstood their position):

- Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed
- Me, advocating Windows issues and PEP 302 integration (to cover eggs
and general flexibility)
- Phillip (and occasional others), representing setuptools POV
- Interested python-dev participants

Nobody representing any other 3rd party packaging solution than setuptools.

Much as I'd like to, I can't really argue the case for breaking
setuptools compatibility without practical reasons. And if we're not
going to do that, PEP 376 reduces to a further stage of the
incremental move of setuptools into distutils core (by removing the
partial solution of a .egg-info file, and replacing it with a full
setuptools .egg-info directory, plus a few introspection APIs as a
sweetener).

2009/7/7 Ronald Oussoren <ronaldoussoren at mac.com>:
> But why break existing code without having any other benifits?  If I read
> the discussion correctly the name would be changed without any changes to
> the contents of the metadata directory.  I would be more inclined to be in
> favour if the name change had a sound technical reason, such as a change of
> the contents of the directory which would make setuptools "egg-info"
> directories incompatible with the PEP376 ones.

See MAL's comments above, and my response. If (and only if!) his
proposal is accepted, then a name change starts to be worth discussing
further. As things stand at the moment, the structure appears to be
setuptools-compatibile (other than the new RECORD file, which Phillip
has committed to adding) so a name change isn't worth it.

I can't comment myself on setuptools compatibility, so I'm assuming
here that Phillip will speak up if the proposed format is *not*
compatible...

Paul.

From mal at egenix.com  Tue Jul  7 11:28:10 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 11:28:10 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
 provide distribution metadata
In-Reply-To: <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>	<20090705183326.097CF3A4099@sparrow.telecommunity.com>	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>	<4A520BDF.80503@gmail.com>	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>	<87ws6lp8pc.fsf@benfinney.id.au>	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>	<4A52FE4A.6020204@egenix.com>
	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
Message-ID: <4A53152A.8020003@egenix.com>

Paul Moore wrote:
> 2009/7/7 M.-A. Lemburg <mal at egenix.com>:
>> The PEP should take the chance to define not only the
>> directory, but also the complete set of files in there and
>> their format.
>>
>> A typical setuptools dir looks like this:
>>
>> dependency_links.txt
>> namespace_packages.txt
>> not-zip-safe
>> PKG-INFO
>> requires.txt
>> SOURCES.txt
>> top_level.txt
>>
>> Ie. a complete mess of upper-case/lower-case names, names with
>> underscores or hyphens as word separator, files with extensions,
>> files without them.
>>
>> This needs some serious cleanup.
>>
>> A new dir name will allow to do this, so +1 on a new dir name.
> 
> I agree the current situation is a mess. I believe that everything you
> mention is related to setuptools - so essentially, we have the
> situation:
> 
> - the existing setuptools format is a mess (at least in the opinion of
> MAL and me :-))
> - PEP 376 is an opportunity to consider cleanup
> - there are some strong supporters of keeping things setuptools-compatible
> - nobody has come forward with any other real-world use case than setuptools
> - consequently, those of us arguing for cleanup are talking theoretically :-(
> - Do you (MAL) have any real use for this data? Any non-setuptools use
> case, no matter how small, would really help here.

No, we currently don't have a package installer that would
implement what setuptools does on the user side, but without
all the black magic stuff.

What we do have is a somewhat different approach to packaging
binary packages: we call them pre-built archives.

The idea is that the developer runs a complete build command on
the target platform, the bdist_prebuilt command then packages up
everything (including the current state of distutils) and the
user then runs the usual

    python setup.py install

on her machine to complete the setup.

The major advantage here is that the user actually has full
control over where things are installed and how.

So whatever distutils implements as part of the install command,
this format will also support. We also provide an uninstall command,
that basically runs an installation in reverse.

Something else we've done recently is write a bdist_egg command
that creates .egg files without relying on setuptools. It's
fairly easy to do and if there's interest, I can add that to
core distutils. Changing the layout of the EGG-INFO dir embedded
in those .egg files is pretty easy to do, so I don't really see
much of a problem with changing it or renaming the dir.

> I suspect practicality will beat purity here. Which would be fine, if
> the "practical" cases weren't just setuptools.
> 
> I understood that there was a lot of interest in a "distutils cleanup"
> from the packaging community. And yet so far in this discussion, the
> main participants have seemed to be (apologies to anyone if I've
> misunderstood their position):
> 
> - Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed
> - Me, advocating Windows issues and PEP 302 integration (to cover eggs
> and general flexibility)
> - Phillip (and occasional others), representing setuptools POV
> - Interested python-dev participants
> 
> Nobody representing any other 3rd party packaging solution than setuptools.

Well I've added another one above, the pre-built approach :-)

Writing an uninstall command really isn't all that hard, provided
you stick with the standard distutils "python setup.py install"
dance. The whole packaging approach only complicates things, IMHO.

> Much as I'd like to, I can't really argue the case for breaking
> setuptools compatibility without practical reasons. And if we're not
> going to do that, PEP 376 reduces to a further stage of the
> incremental move of setuptools into distutils core (by removing the
> partial solution of a .egg-info file, and replacing it with a full
> setuptools .egg-info directory, plus a few introspection APIs as a
> sweetener).

Uhm, we are talking about a Python standard here and since there
is no prior implementation in Python stdlib, we're not breaking
anything and are free to design whatever we like.

Of course, it does make a lot of sense to build upon things that
have been used in the past, so looking at setuptools is certainly
a useful strategy. However, let's learn from their mistakes
and take the change to clean up things a bit.

There are good reasons to do so:

 * the naming scheme should be concise

 * the meta-data directory should get a name that doesn't
   imply any specific distribution form...

Calling the directory .egg-info is natural if you're only dealing
with eggs. It is not when running standard "python setup.py install"
or having a mixed setup using different packaging mechanisms.

> 2009/7/7 Ronald Oussoren <ronaldoussoren at mac.com>:
>> But why break existing code without having any other benifits?  If I read
>> the discussion correctly the name would be changed without any changes to
>> the contents of the metadata directory.  I would be more inclined to be in
>> favour if the name change had a sound technical reason, such as a change of
>> the contents of the directory which would make setuptools "egg-info"
>> directories incompatible with the PEP376 ones.
> 
> See MAL's comments above, and my response. If (and only if!) his
> proposal is accepted, then a name change starts to be worth discussing
> further. As things stand at the moment, the structure appears to be
> setuptools-compatibile (other than the new RECORD file, which Phillip
> has committed to adding) so a name change isn't worth it.
> 
> I can't comment myself on setuptools compatibility, so I'm assuming
> here that Phillip will speak up if the proposed format is *not*
> compatible...

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Tue Jul  7 11:33:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 11:33:48 +0200
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A53152A.8020003@egenix.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
	<4A52FE4A.6020204@egenix.com>
	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
	<4A53152A.8020003@egenix.com>
Message-ID: <94bdd2610907070233m4b0bd28cie264c42cf228abe5@mail.gmail.com>

On Tue, Jul 7, 2009 at 11:28 AM, M.-A. Lemburg<mal at egenix.com> wrote:
>
> Writing an uninstall command really isn't all that hard, provided
> you stick with the standard distutils "python setup.py install"
> dance. The whole packaging approach only complicates things, IMHO.

You can't expect people to keep the distribution they used to install
in a corner
of their hard drive, to make sure they will be able to uninstall it (or
to ask them to download it again)

That's all about the RECORD files and the asymetric uninstall script
described in PEP 376.

having an uninstall command on the top of the uninstall script is a
nice shortcut though.

From p.f.moore at gmail.com  Tue Jul  7 11:44:45 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 10:44:45 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A530DFD.40802@egenix.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
Message-ID: <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>

2009/7/7 M.-A. Lemburg <mal at egenix.com>:
> I think you have to differentiate between packages and applications.

Agreed. I believe that only packages should be considered here.
Applications are the focus of tools like py2exe on Windows, and (AIUI)
things like workingenv. These tools should (will) have their own
approaches.

The only cases I know where there is reason for a package to store
paths outside the package directory are:
- executable scripts, which go in sys.prefix/Scripts on Windows, and
??? on Unix/MAC OS
- supporting files (MoinMoin puts its HTML documents etc in
sys.prefix/share on Windows, cx_Oracle puts its documentation in
sys.prefix/cx_Oracle-doc)

Executable files could probably be superseded by using "python -m",
but compatibility and users' preference for having a "real script"
probably means they aren't going away in the near future.

Support files are getting put in the package directory more often
these days, but stuff that needs to be found by non-Python tools is
arguably still better in a more discoverable location.

For Windows, having a few distinguished locations under the
installation root (scripts, share, doc) would probably do. For Unix
and Mac OS, I have no opinion (but I suspect that absolute paths like
/usr/local/bin might be the norm there).

[...]
> Summarizing, I think it's better not to record config files
> and other user-edited files in the RECORD file.

The RECORD file should contain precisely those files that are created
as part of the install. That's ultimately the point of the file (for
ownership queries and uninstallation).

Hmm, on the other hand, if foo.py is in the RECORD file, the
uninstaller should uninstall foo.pyc and foo.pyo as well. And a query
as to whether the distribution owns foo.pyc should return True. How
will this be handled?

Paul.

From p.f.moore at gmail.com  Tue Jul  7 11:52:25 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 10:52:25 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <4A53152A.8020003@egenix.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
	<4A52FE4A.6020204@egenix.com>
	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
	<4A53152A.8020003@egenix.com>
Message-ID: <79990c6b0907070252i2e1d945fw9a5d8920915c3d39@mail.gmail.com>

2009/7/7 M.-A. Lemburg <mal at egenix.com>:
> Well I've added another one above, the pre-built approach :-)

Thanks! That's a lot of help. Are there any public examples, or is
your format only used internally?

Would you expect to use any of the new pkgutil APIs - for example, to
check if a distribution is installed, and/or the installed version? Or
to check you're not overwriting a file owned by another distribution?
Or would you get any value from storing your own format-specific
metadata files in the egg-info directory?

If all you care about is that setup.py install continues to work, my
suspicion is that PEP 376 won't have much to offer you, whatever
happens.

Paul.

From ncoghlan at gmail.com  Tue Jul  7 13:38:34 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 07 Jul 2009 21:38:34 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
Message-ID: <4A5333BA.3030806@gmail.com>

Paul Moore wrote:
> 2009/7/6 Nick Coghlan <ncoghlan at gmail.com>:
> - There are *no* guaranteed absolute locations on Windows, so any such
> oddly-located file would require user interaction to work. Certainly
> bdist_wininst and bdist_msi don't do that.
> - My experiments indicate that bdist_{wininst,msi} are broken with
> respect to absolute paths anyway: they do a --root install to a
> temporary directory (and the absolute paths don't end up in there) and
> then package up that temporary directory.
> 
> I still want to see a real life example that demonstrates that there
> is a genuine issue here. We're spending a lot of energy and complexity
> trying to design a solution to a problem that actually doesn't appear
> to exist in practice...
> 
> (To be honest, I'd be fairly confident in saying that absolute paths
> can be ignored on Windows, subject to some corner cases that I haven't
> thought through yet. My worry is that I don't know what Unix and Mac
> users might do, so I can't just wish away the issue because it can't
> arise on Windows. Can a Unix/Mac user offer a real-world example on
> their own system?)

I thought installing pywin32 based COM objects still involved messing
with the Windows directory, but MS may have improved that in more recent
OS versions. It's been years since I played with win32com objects, and
even then it was just idle experimentation that didn't get very far so I
could easily be misremembering.

For *nix, the obvious use case is installing scripts somewhere like
/usr/bin or /usr/local/bin.

One option is to punt on this whole issue and say if people want to
install stuff outside the Python module heirarchy they should create
their own OS-specific package to manage it (i.e. leave the non-relative
paths to be managed by APT or a Windows installer or whatever).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ben+python at benfinney.id.au  Tue Jul  7 13:52:38 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 07 Jul 2009 21:52:38 +1000
Subject: [Python-Dev] PEP 376 - Open questions
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
Message-ID: <87k52kpv5l.fsf@benfinney.id.au>

Paul Moore <p.f.moore at gmail.com> writes:

> The only cases I know where there is reason for a package to store
> paths outside the package directory are:

I think the *only* files that actually belong in the package directory
are the Python modules inside that package. Other files need to be
easily, and automatically, separable for purposes of installation.

> - executable scripts, which go in sys.prefix/Scripts on Windows

    os.path.join(sys.prefix, 'Scripts')

> and ??? on Unix/MAC OS

Depending on whether the developer designates the programs for
sysadmin-only use or not:

    os.path.join(sys.prefix, 'bin')
    os.path.join(sys.prefix, 'sbin')

> - supporting files (MoinMoin puts its HTML documents etc in
> sys.prefix/share on Windows, cx_Oracle puts its documentation in
> sys.prefix/cx_Oracle-doc)

This category, it seems to me, needs to be expanded with metadata that
allows ?purpose?-based tagging, so that platform-specific standards
can be applied using those purposes to determine the correct location
for these non-Python-module files.

> Support files are getting put in the package directory more often
> these days

Only, IMO, because there's no way of flagging it as anything but
arbitrary ?data?.

Examples of purpose-based classifications that need to be distinctly
declarable: executable program, importable module source,
platform-independent compiled module, platform-dependent compiled
module, documentation, run-time variable data, static data,
configuration, and so on.

All of these (and others I've forgotten) should be possible for the
developer to declare in distribution metadata, and the installer can
then use those declarations to make the files go to platform-specific
locations, not the Python package directories.

> For Windows, having a few distinguished locations under the
> installation root (scripts, share, doc) would probably do. For Unix
> and Mac OS, I have no opinion (but I suspect that absolute paths like
> /usr/local/bin might be the norm there).

The lines are drawn in different places on each platform; we don't want
to have the union of all these different platform-specific locations in
the standard, and likewise we don't want to leave any of them with
second-class support.

I think it's not the developer's burden to decide *where* such files go;
rather, they should be declaring only the *purpose* of these files in
the distribution metadata, and it's up to the site-specific installer
(possibly as configured by the installing user) to decide the location
of each file by its declared purpose.

-- 
 \      ?I can picture in my mind a world without war, a world without |
  `\   hate. And I can picture us attacking that world, because they'd |
_o__)                                   never expect it.? ?Jack Handey |
Ben Finney


From ben+python at benfinney.id.au  Tue Jul  7 13:58:51 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 07 Jul 2009 21:58:51 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com>
	<20090705183326.097CF3A4099@sparrow.telecommunity.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
	<4A52FE4A.6020204@egenix.com>
	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
Message-ID: <87fxd8puv8.fsf@benfinney.id.au>

Paul Moore <p.f.moore at gmail.com> writes:

> I agree the current situation is a mess. I believe that everything you
> mention is related to setuptools - so essentially, we have the
> situation:
> 
> - the existing setuptools format is a mess (at least in the opinion of
> MAL and me :-))

No argument from me on this point.

> - PEP 376 is an opportunity to consider cleanup
> - there are some strong supporters of keeping things
> setuptools-compatible

Your implication seems to be that these two are incompatible. It should
be noted that it's possible to clean up, by having a separate (e.g.
?.pydist?) directory as the new standard, with ?cleaned up? standard
content, and continue backward compatibility in the actual tools while
deprecating the old ?.egg-info? format for some time until finally
removing that support.

> Much as I'd like to, I can't really argue the case for breaking
> setuptools compatibility without practical reasons.

Agreed. I think we should be making it clear that ?.egg-info? is not
going to evolve any further, and that it will eventually go away; but
not remove support for it until there's a clearly established
replacement.

-- 
 \         ?I got up the other day, and everything in my apartment has |
  `\   been stolen and replaced with an exact replica.? ?Steven Wright |
_o__)                                                                  |
Ben Finney


From p.f.moore at gmail.com  Tue Jul  7 14:05:19 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 13:05:19 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <87k52kpv5l.fsf@benfinney.id.au>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
Message-ID: <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>

2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
[... lots of interesting stuff deleted ...]
> I think it's not the developer's burden to decide *where* such files go;
> rather, they should be declaring only the *purpose* of these files in
> the distribution metadata, and it's up to the site-specific installer
> (possibly as configured by the installing user) to decide the location
> of each file by its declared purpose.

That's a whole different PEP, though.

Paul.

From p.f.moore at gmail.com  Tue Jul  7 14:11:50 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 7 Jul 2009 13:11:50 +0100
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to
	provide distribution metadata
In-Reply-To: <87fxd8puv8.fsf@benfinney.id.au>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>
	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>
	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>
	<4A520BDF.80503@gmail.com>
	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>
	<87ws6lp8pc.fsf@benfinney.id.au>
	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>
	<4A52FE4A.6020204@egenix.com>
	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>
	<87fxd8puv8.fsf@benfinney.id.au>
Message-ID: <79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com>

2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
>> - PEP 376 is an opportunity to consider cleanup
>> - there are some strong supporters of keeping things
>> setuptools-compatible
>
> Your implication seems to be that these two are incompatible. It should
> be noted that it's possible to clean up, by having a separate (e.g.
> ?.pydist?) directory as the new standard, with ?cleaned up? standard
> content, and continue backward compatibility in the actual tools while
> deprecating the old ?.egg-info? format for some time until finally
> removing that support.

Phillip argued strongly against this. I'm not going to repeat his
arguments - they seemed to make sense to me at the time, but I haven't
internalised them well enough that I'd be willing to try to repeat
them here.

The crux of all this seems to be setuptools' backwards compatibility
issues, As someone who has a very strong dislike of setuptools (the
implementation, not the ideas), I'm trying very, very hard to leave it
to someone else to say "stuff setuptools, this is a core distutils
proposal, setuptools can keep up or die" :-)

Paul.

From rdmurray at bitdance.com  Tue Jul  7 14:52:41 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 7 Jul 2009 08:52:41 -0400 (EDT)
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>

On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote:
> 2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
> [... lots of interesting stuff deleted ...]
>> I think it's not the developer's burden to decide *where* such files go;
>> rather, they should be declaring only the *purpose* of these files in
>> the distribution metadata, and it's up to the site-specific installer
>> (possibly as configured by the installing user) to decide the location
>> of each file by its declared purpose.
>
> That's a whole different PEP, though.

Which one?  It seems to me that supporting this is implicit in
the language summit goals of (1) having distutils be better support
infrastructure for system packaging utilities and (2) needing a way to
deal with resource files "that might be installed in a specific place on
the target system by the system packager".  I'll grant that I'm reading
between the lines, it isn't an explicitly stated goal.  But it was the
direction my mind went when I read Tarek's notes, given that the first
stated goal is "standardize more metadata".

But I'm not one of the people involved in system packaging tools, so
I'll leave it to them to say how useful/important this is.

--David

From ziade.tarek at gmail.com  Tue Jul  7 15:23:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 15:23:01 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
Message-ID: <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>

On Tue, Jul 7, 2009 at 2:52 PM, R. David Murray<rdmurray at bitdance.com> wrote:
> On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote:
>>
>> 2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
>> [... lots of interesting stuff deleted ...]
>>>
>>> I think it's not the developer's burden to decide *where* such files go;
>>> rather, they should be declaring only the *purpose* of these files in
>>> the distribution metadata, and it's up to the site-specific installer
>>> (possibly as configured by the installing user) to decide the location
>>> of each file by its declared purpose.
>>
>> That's a whole different PEP, though.
>
> Which one? ?It seems to me that supporting this is implicit in
> the language summit goals of (1) having distutils be better support
> infrastructure for system packaging utilities and (2) needing a way to
> deal with resource files "that might be installed in a specific place on
> the target system by the system packager". ?I'll grant that I'm reading
> between the lines, it isn't an explicitly stated goal. ?But it was the
> direction my mind went when I read Tarek's notes, given that the first
> stated goal is "standardize more metadata".

Yes but the topic is so wide that it has to be cut in several PEP, and
things have to be done gradually

So far:

- PEP 376 : standard for the metadata format and location + query APIs
- PEP 345 : standard for the metadata *content* - work in progress too
(there's a branch with new fields waiting)
- PEP 386 : standard for version comparisons

topics that are not yet in PEP are grouped on the wiki page (under
"current work") with notes :

http://wiki.python.org/moin/Distutils

When I started to work on this I didn't realize the gigantic amount of
work and coordination
it requires, and I do understand now the current state. At first I was
trying to coordinate interested
people to work on each topic mentioned there in parallel. (like we did
a bit after the summit)
But at the end, it seems that having everyone interested in packaging
matters focusing on the
less number of possible topics pays more.

PEP 376 is just a piece of  the puzzle but I am confident it will
speed up other tasks since
it raises our common ground knowledge.

From mal at egenix.com  Tue Jul  7 15:26:20 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 15:26:20 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com>
	<bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>
Message-ID: <4A534CFC.30909@egenix.com>

Brett Cannon wrote:
> On Mon, Jul 6, 2009 at 07:51, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> Dirkjan Ochtman wrote:
>>> In response to some rumblings on python-committers and just to request
>>> more feedback, a progress report. I know it's long, I've tried to put
>>> to keep it concise and chunked, though.
>> Two things:
>>
>>  * We need some form of documentation of how committers are expected
>>   to work with the hg repo. (This is also missing for the Subversion
>>   repo, which due to the 3.x branch has gotten somewhat unclear - at
>>   least for me)
>>
> 
> I am planning to get a version of the dev FAQ written up that covers most of
> what it already does now for svn.
> 
> 
>>   It is currently not clear where to check in patches, whether and
>>   where to backport or forward-patch, which branches to consider
>>   closed, etc.
>>
>>   E.g. if I check in something in trunk/ (Python 2.7), do I have to
>>   forward patch this change to the 3.0 branch (guess not), the py3k/
>>   branch (Python 3.1), or will someone else take care of this, so that
>>   it's better not to interfere by doing it myself ?
>>
> 
> This question is partially answered by
> http://www.python.org/dev/faq/#how-do-i-merge-between-branches, but I agree
> that we should have either this spelled out in the FAQ or a
> committer-specific doc at www.python.org/dev/ that makes this all very
> obvious.

The merge process itself is more or less clear. What I'm missing
is the agreed upon strategy for applying the patches to the various
branches.

I've seen a few discussions about this, but no final statement
of what strategy to follow and whether hg makes this easier (AFAIR,
that was the main argument for switching to hg).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Tue Jul  7 15:32:46 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Jul 2009 15:32:46 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>
	<4A526B11.7060308@gmail.com>	<4A527363.7070305@v.loewis.de>
	<4A527514.5040709@gmail.com>	<4A530453.10005@egenix.com>
	<ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>
Message-ID: <4A534E7E.7010700@egenix.com>

Dirkjan Ochtman wrote:
> On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg<mal at egenix.com> wrote:
>> Is there a standard notation for hg revisions that roundup could
>> detect and turn into links (much like it does for svn) ?
> 
> [a-f0-9]{12} should mostly do.

Hmm, no prefix or suffix ?

So we'll always have to write "see deadbeefdeadbeefff for details"
or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?!

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From dirkjan at ochtman.nl  Tue Jul  7 15:40:24 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 7 Jul 2009 15:40:24 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A534E7E.7010700@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com>
	<4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com>
	<4A530453.10005@egenix.com>
	<ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>
	<4A534E7E.7010700@egenix.com>
Message-ID: <ea2499da0907070640x706e8623o21a6f59aeacae600@mail.gmail.com>

On Tue, Jul 7, 2009 at 15:32, M.-A. Lemburg<mal at egenix.com> wrote:
> Hmm, no prefix or suffix ?

No, not really. hg often shows revision integers as well, but as they
aren't globally consistent, they're of little value in communicating
changeset pointers.

> So we'll always have to write "see deadbeefdeadbeefff for details"
> or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?!

Yes. And it's really not that bad.

Cheers,

Dirkjan

From rdmurray at bitdance.com  Tue Jul  7 16:17:21 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 7 Jul 2009 10:17:21 -0400 (EDT)
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A534CFC.30909@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com>
	<bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>
	<4A534CFC.30909@egenix.com>
Message-ID: <Pine.LNX.4.64.0907071003460.6306@kimball.webabinitio.net>

On Tue, 7 Jul 2009 at 15:26, M.-A. Lemburg wrote:
> The merge process itself is more or less clear. What I'm missing
> is the agreed upon strategy for applying the patches to the various
> branches.
>
> I've seen a few discussions about this, but no final statement
> of what strategy to follow and whether hg makes this easier (AFAIR,
> that was the main argument for switching to hg).

I think the main reason for switching was that it would make it easier
for non-core-committers to maintain branches and submit patches (as
changesets core committers can pull).  I don't think it was ever clear
that the merge workflow would in fact get easier, except insofar as
hg's merge support is better than SVN's (at least, I believe people
have said that last is true).  There is _hope_ that it will be easier,
but I think it remains to be proven/worked out.  And I believe there is
no tool like svnmerge for tracking changesets to be merged, which could
be an issue that needs a resolution.

IIUC, the discussion about named versus cloned branches is part of
figuring out the workflow....

--David

From ideasman42 at gmail.com  Tue Jul  7 19:38:05 2009
From: ideasman42 at gmail.com (Campbell Barton)
Date: Tue, 7 Jul 2009 10:38:05 -0700
Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
Message-ID: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>

In Python3 PySys_SetArgv takes (wchar_t **) for the argv, I looked
into converting a (char **) into a (wchar_t **) and while its possible
its lengthy enough not to be trivial, see python.c:18 - char2wcharm(),
its 102 lines with ifdef's and goto's, not including the loop lower
down that loops over the argv.

Looking further python has a very similar function for
./PC/bdist_wininst/install.c, for do_run_installscript  -
widen_string().

Since many C applications take argv as a (char **) it seems reasonable
for python to keep a function for the C api that accepts a  (char **)
argument for argv.
otherwise each C applications that embeds python will need to write
their own conversion function.

also noticed Demo/embed/demo.c is out of date, giving (char **) to
PySys_SetArgv().
-- 
- Campbell

From pje at telecommunity.com  Tue Jul  7 22:30:05 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 07 Jul 2009 16:30:05 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
Message-ID: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>

Just an idea...  suppose that instead of using "real" absolute paths 
in the RECORD file for non-local files (scripts, data, etc) we 
changed the format to include a "prefix" field, containing something 
like LIBDIR, SHARE, SCRIPTS, etc., ala bdist_wininst internals?

Also, we could include a separate (optional) PREFIXES file defining 
what those locations translated to at install time.  Dumb bdists 
would still have their build paths there (or delete the file before 
packaging, or use real paths instead of the --root prefixed versions).

Upside: relocation can be detected and handled, RECORD remains 
cross-platform, and bdists are ok.

Downside: more complex API required to read/manipulate paths and 
delete files, since you need to be able to check that you have the 
right prefixes, and may have to ask the user for a prefix you don't 
recognize, if the default in PREFIXES doesn't match.

Thoughts?

(Also, as a special case, any file that's actually installed to 
LIBDIR or a subdirectory thereof (even if it's technically a "data" 
file or script), will probably need to be designated under LIBDIR 
prefix to prevent runtime breakage in the event any system package 
maintainers are tempted to use RECORD files as a way of forcing a HFS 
conformance.  These are installation prefixes, *not* content types.)


From pje at telecommunity.com  Tue Jul  7 22:31:52 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 07 Jul 2009 16:31:52 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.co
 m>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
Message-ID: <20090707202851.7DBB93A410E@sparrow.telecommunity.com>

At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>When I started to work on this I didn't realize the gigantic amount of
>work and coordination it requires

No one expects the package inquisition.  ;-)


From ncoghlan at gmail.com  Tue Jul  7 23:02:52 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Jul 2009 07:02:52 +1000
Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks
 to	provide distribution metadata
In-Reply-To: <79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com>
References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com>	<94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com>	<20090706010108.CCEC83A4099@sparrow.telecommunity.com>	<4A520BDF.80503@gmail.com>	<79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com>	<87ws6lp8pc.fsf@benfinney.id.au>	<79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com>	<4A52FE4A.6020204@egenix.com>	<79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com>	<87fxd8puv8.fsf@benfinney.id.au>
	<79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com>
Message-ID: <4A53B7FC.6090500@gmail.com>

Paul Moore wrote:
> 2009/7/7 Ben Finney <ben+python at benfinney.id.au>:
>>> - PEP 376 is an opportunity to consider cleanup
>>> - there are some strong supporters of keeping things
>>> setuptools-compatible
>> Your implication seems to be that these two are incompatible. It should
>> be noted that it's possible to clean up, by having a separate (e.g.
>> ?.pydist?) directory as the new standard, with ?cleaned up? standard
>> content, and continue backward compatibility in the actual tools while
>> deprecating the old ?.egg-info? format for some time until finally
>> removing that support.
> 
> Phillip argued strongly against this. I'm not going to repeat his
> arguments - they seemed to make sense to me at the time, but I haven't
> internalised them well enough that I'd be willing to try to repeat
> them here.
> 
> The crux of all this seems to be setuptools' backwards compatibility
> issues, As someone who has a very strong dislike of setuptools (the
> implementation, not the ideas), I'm trying very, very hard to leave it
> to someone else to say "stuff setuptools, this is a core distutils
> proposal, setuptools can keep up or die" :-)

Actually, the approach Ben suggests is the same as the one I suggested
for dealing with the backwards compatibility problem (i.e. distutils
would install both the existing .egg-info for detection by existing
packaging tools, while putting the new stuff in a .pydist directory).

The argument against *that* approach to backwards compatibility is the
accumulation of cruft in site-packages (and other module installation
directories) since even a simple installation of a single module or
small package would now be generating at least *three* entries in the
relevant directory (.egg-info, .pydist and the module or package itself).

Retaining the .egg-info naming provides detection of installed
distributions by old packaging systems with a minimum of cruft on the
end user's system. The only possible advantage changing the extension
name might give us is making it fractionally easier to clean up the
contents of the .egg-info directory, but I don't think it will actually
make that much difference (i.e. it really shouldn't be that hard to keep
the new PEP 376 based filenames all UPPERCASE and distinct from any
filenames currently stored in the directory by setuptools or other
packaging utilities)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Tue Jul  7 23:21:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 23:21:23 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
Message-ID: <94bdd2610907071421r70daf38bt91fb29fb18db46f2@mail.gmail.com>

2009/7/7 Paul Moore <p.f.moore at gmail.com>:
>
> The RECORD file should contain precisely those files that are created
> as part of the install. That's ultimately the point of the file (for
> ownership queries and uninstallation).
>
> Hmm, on the other hand, if foo.py is in the RECORD file, the
> uninstaller should uninstall foo.pyc and foo.pyo as well. And a query
> as to whether the distribution owns foo.pyc should return True. How
> will this be handled?

It's planned to list them as well in RECORD, since install calls a sub
command that build thems (install_lib),
So the same rules apply than the .py ones.

But there's a special case : if the --no-compile or the --no-optimize
option is used, then the pyc|pyo files are not
added. Which means they will be created afterwards when the module is
used on the target system.

So the pyc|pyo files could be removed when they are present besides
the py file that is being removed.

Although, it will still be required to write them in the RECORD file
by the install command when the
--no-compile or the --no-optimize options are *not* used.  So they are
properly detected and removed
when the py files are not distributed in binary distributions, but
just pyc files.

From ziade.tarek at gmail.com  Tue Jul  7 23:21:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 23:21:25 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A5333BA.3030806@gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A5272B2.8000202@gmail.com>
	<79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com>
	<4A5333BA.3030806@gmail.com>
Message-ID: <94bdd2610907071421l3d7a5122n31e1901e2baf9f34@mail.gmail.com>

2009/7/7 Nick Coghlan <ncoghlan at gmail.com>:
> For *nix, the obvious use case is installing scripts somewhere like
> /usr/bin or /usr/local/bin.

Using distutils' scripts option, they will end up in  : sys.exec_path/bin

Another use case I've found in a distro I've installed this afternoon :

setup(..., data_files=[('/usr/share/man/man1/', ['SOMEFILE'])], ...)

That's not the most elegant way to add a man page, but for someone
who doesn't bather with APT or whatever, it works to build a binary
distribution.

>
> One option is to punt on this whole issue and say if people want to
> install stuff outside the Python module heirarchy they should create
> their own OS-specific package to manage it (i.e. leave the non-relative
> paths to be managed by APT or a Windows installer or whatever).

If so, what do we do with the "data_files" option in distutils ?

If it's used with absolute paths, files can be installed anywhere on
the system, and we want to track them.
Even if we don't uninstall them automatically, they should be tracked
so a third-party
uninstaller can deal with them properly.

Or do we change this distutils feature and state that the directories
used in "data_files"
will always be relative to sys.prefix ?

That would bring us back to three cases in the RECORD:

- files located under sys.prefix, but not located under site-packages
- files located under sys.exec_prefix, but not located under site-packages
- files located under site-packages

Where "site-packages" is the directory that contains .egg-info
directory of the distribution

(that's basically the current PEP state, beside the absolute paths
case we would need to remove)

From ziade.tarek at gmail.com  Tue Jul  7 23:21:27 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 23:21:27 +0200
Subject: [Python-Dev] PEP 376 - RECORD format proposal
Message-ID: <94bdd2610907071421i3e1dc6b4y3ba3125d0f72bfb8@mail.gmail.com>

Hello,

Let's state that we don't allow absolute paths in RECORD file and see
what we can
do with other paths.

First as a reminder, here's the full list of directories used by
Distutils at installation time.
The "install" command uses five options to decide where to install the files.

The default values are:

Unix:

 install_purelib: sys.prefix/lib/python2.6/site-packages
 install_platlib: sys.exec_prefix/lib/python2.6/site-packages
 install_headers: sys.prefix/include/python2.6/docutils
 install_scripts: sys.prefix/bin
 install_data   : sys.prefix

Unix, using the home option, instead of sys.prefix/sys.exec_prefix:

 install_purelib: home/lib/python
 install_platlib: home/lib/python
 install_headers: home/include/python/$dist_name
 install_scripts: home/bin
 install_data   : home

Windows:

 install_purelib: sys.prefix/Lib/site-packages
 install_platlib: sys.prefix/Lib/site-packages
 install_headers: sys.prefix/Include/python2.6/docutils
 install_scripts: sys.prefix/bin
 install_data   : sys.prefix

There are also some old scheme for os.name in ('mac', 'os2') I need to
cleanup asap in there,
but I won't mention them.

Last, there's now the new per user site-package that adds more
potential locations. e.g. replaces
sys.prefix and sys.exec_prefix with two user directories.

So they are all located under two root locations at the max.

But since these are all overridable options (in pydistutils.cfg,
distutils.cfg, etc) files can be located under *five* different
roots.

So my proposal is to write those five roots in a header line in the
RECORD file, and use them as prefixes
to locate the files that are not under the directory where the
egg-info dir is located (which will be kept : relative paths)

purelib,xxx,headers,xxx,scripts,xxx,install_data,xxx
...
$purelib/some/file
...


How does that  sounds ?

Tarek.

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Tue Jul  7 23:29:47 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Jul 2009 23:29:47 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A534E7E.7010700@egenix.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>
	<4A526B11.7060308@gmail.com>	<4A527363.7070305@v.loewis.de>
	<4A527514.5040709@gmail.com>	<4A530453.10005@egenix.com>
	<ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>
	<4A534E7E.7010700@egenix.com>
Message-ID: <4A53BE4B.6000604@v.loewis.de>

M.-A. Lemburg wrote:
> Dirkjan Ochtman wrote:
>> On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg<mal at egenix.com> wrote:
>>> Is there a standard notation for hg revisions that roundup could
>>> detect and turn into links (much like it does for svn) ?
>> [a-f0-9]{12} should mostly do.
> 
> Hmm, no prefix or suffix ?
> 
> So we'll always have to write "see deadbeefdeadbeefff for details"
> or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?!

No, rather "see deadbeefdead for details" and
"Reverting f00fl33df00fl33d00 after problems on Pentium CPUs".
In practice, rather 30ba63d28b1b or 12fb3b32d75d (from html5lib,
fwiw).

Dirkjan meant "[a-f0-9]{12}" literally.

Regards,
Martin

From ziade.tarek at gmail.com  Tue Jul  7 23:30:16 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 7 Jul 2009 23:30:16 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090707202851.7DBB93A410E@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
Message-ID: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>

On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:
> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>>
>> When I started to work on this I didn't realize the gigantic amount of
>> work and coordination it requires
>
> No one expects the package inquisition. ?;-)
>
>

Sorry, i've looked in the english dictionary but I don't get this one.
what do you mean ?

From martin at v.loewis.de  Tue Jul  7 23:32:16 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 07 Jul 2009 23:32:16 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <ea2499da0907070640x706e8623o21a6f59aeacae600@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com>	
	<4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com>	
	<4A530453.10005@egenix.com>	
	<ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>	
	<4A534E7E.7010700@egenix.com>
	<ea2499da0907070640x706e8623o21a6f59aeacae600@mail.gmail.com>
Message-ID: <4A53BEE0.3000703@v.loewis.de>

>> Hmm, no prefix or suffix ?
> 
> No, not really. hg often shows revision integers as well, but as they
> aren't globally consistent, they're of little value in communicating
> changeset pointers.

I think MAL wasn't asking for a "1354:" prefix, but for a, say,
"r" or "R" prefix, or perhaps "V" or "merc:".

I think you are proposing that there is no prefix because the chance
for a misidentification of some word as a hg revision number is small,
as it has to be exactly 12 hex digits.

Regards,
Martin

From rdmurray at bitdance.com  Tue Jul  7 23:40:06 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 7 Jul 2009 17:40:06 -0400 (EDT)
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> 
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> 
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> 
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net> 
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> 
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0907071736530.6306@kimball.webabinitio.net>

On Tue, 7 Jul 2009 at 23:30, Tarek Ziad? wrote:
> On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:
>> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>>>
>>> When I started to work on this I didn't realize the gigantic amount of
>>> work and coordination it requires
>>
>> No one expects the package inquisition. ?;-)
>>
>>
>
> Sorry, i've looked in the english dictionary but I don't get this one.
> what do you mean ?

See the recent thread on python-committers about mandatory Monty Python
cultural education for committers.... :)

See also http://www.youtube.com/watch?v=gldlyTjXk9A

--David

From ncoghlan at gmail.com  Tue Jul  7 23:41:14 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Jul 2009 07:41:14 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>	<4A530DFD.40802@egenix.com>	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>	<87k52kpv5l.fsf@benfinney.id.au>	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
Message-ID: <4A53C0FA.8020304@gmail.com>

Tarek Ziad? wrote:
> On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:
>> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>>> When I started to work on this I didn't realize the gigantic amount of
>>> work and coordination it requires
>> No one expects the package inquisition.  ;-)
>>
>>
> 
> Sorry, i've looked in the english dictionary but I don't get this one.
> what do you mean ?

It's a Monty Python reference:
http://en.wikipedia.org/wiki/The_Spanish_Inquisition_(Monty_Python)
http://www.youtube.com/watch?v=gldlyTjXk9A

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Tue Jul  7 23:42:56 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Jul 2009 23:42:56 +0200
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <Pine.LNX.4.64.0907071003460.6306@kimball.webabinitio.net>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	<4A520F83.7020300@egenix.com>	<bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>	<4A534CFC.30909@egenix.com>
	<Pine.LNX.4.64.0907071003460.6306@kimball.webabinitio.net>
Message-ID: <4A53C160.6040400@v.loewis.de>

>> I've seen a few discussions about this, but no final statement
>> of what strategy to follow and whether hg makes this easier (AFAIR,
>> that was the main argument for switching to hg).
> 
> I think the main reason for switching was that it would make it easier
> for non-core-committers to maintain branches and submit patches (as
> changesets core committers can pull).

Indeed, it's all in PEP 374:
- motivation 1: give direct tool support to non-committers
- motivation 2: allow disconnected (off-line) access to the repository
- motivation 3: allow revisions of a patch
- motivation 4: allow tracking the mainline while working on a patch
- motivation 5: avoid using arcane third-party tools for merge tracking

So merge tracking was not the main argument, but the fifth.

Regards,
Martin

From ncoghlan at gmail.com  Tue Jul  7 23:43:56 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Jul 2009 07:43:56 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A53BEE0.3000703@v.loewis.de>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>	
	<4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com>	
	<4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com>	
	<4A530453.10005@egenix.com>	
	<ea2499da0907070130x21eaba97x78afc9041ee6bee0@mail.gmail.com>	
	<4A534E7E.7010700@egenix.com>
	<ea2499da0907070640x706e8623o21a6f59aeacae600@mail.gmail.com>
	<4A53BEE0.3000703@v.loewis.de>
Message-ID: <4A53C19C.6040502@gmail.com>

Martin v. L?wis wrote:
> I think you are proposing that there is no prefix because the chance
> for a misidentification of some word as a hg revision number is small,
> as it has to be exactly 12 hex digits.

So triggering it accidentally would require a 12 letter word or object
name that used only the letter a-f? Agreed that seems very unlikely.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Tue Jul  7 23:46:51 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Jul 2009 23:46:51 +0200
Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
In-Reply-To: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>
References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>
Message-ID: <4A53C24B.7080202@v.loewis.de>

> Since many C applications take argv as a (char **) it seems reasonable
> for python to keep a function for the C api that accepts a  (char **)
> argument for argv.

I'm not quite sure what you are suggesting: either that there is a
function in the C API that accepts a (char**), or that PySys_SetArgv
be that function.

The latter alternative is out of question, as it breaks the Windows
port of Python - there is simply no way of representing argv with
char** in a meaningful way. It also breaks backwards compatibility.

As for creating another function: please provide a patch to
bugs.python.org. I believe this patch is *very* difficult to
implement, unless the function can accept some severe limitations.

Regards,
Martin

From dalcinl at gmail.com  Wed Jul  8 00:18:51 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 19:18:51 -0300
Subject: [Python-Dev]  Suggested PySys_SetArgv use with a (char **) argv ?
In-Reply-To: <e7ba66e40907071516o7f76e4a0q8bf277611cb60470@mail.gmail.com>
References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>
	<4A53C24B.7080202@v.loewis.de>
	<e7ba66e40907071516o7f76e4a0q8bf277611cb60470@mail.gmail.com>
Message-ID: <e7ba66e40907071518p6e4cab82y3100eaa39bbe802c@mail.gmail.com>

Sorry...

---------- Forwarded message ----------
From: Lisandro Dalcin <dalcinl at gmail.com>
Date: Tue, Jul 7, 2009 at 7:16 PM
Subject: Re: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
To: "Martin v. L?wis" <martin at v.loewis.de>


On Tue, Jul 7, 2009 at 6:46 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>> Since many C applications take argv as a (char **) it seems reasonable
>> for python to keep a function for the C api that accepts a ?(char **)
>> argument for argv.
>

I've run in the same issue, and basically had to copy&paste from
Python sources...

>
> I'm not quite sure what you are suggesting: either that there is a
> function in the C API that accepts a (char**), or that PySys_SetArgv
> be that function.
>
> The latter alternative is out of question, as it breaks the Windows
> port of Python - there is simply no way of representing argv with
> char** in a meaningful way. It also breaks backwards compatibility.
>

Indeed...

> As for creating another function: please provide a patch to
> bugs.python.org. I believe this patch is *very* difficult to
> implement, unless the function can accept some severe limitations.
>

What about the Python lib exposing a public PyOS_char2wchar() ?

No idea how to write a Windows implementation, though..


--
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stephen at xemacs.org  Wed Jul  8 03:55:27 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 08 Jul 2009 10:55:27 +0900
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
Message-ID: <873a98ueeo.fsf@uwakimon.sk.tsukuba.ac.jp>

Tarek Ziad? writes:
 > On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:

 > > No one expects the package inquisition. ?;-)

 > Sorry, i've looked in the english dictionary but I don't get this one.

I think that far more important than PEP 385 conversion of Roundup and
other utilities to recognize Mercurial revision identifiers is a
filter for Python channels that recognizes Monty Python references and
automatically inserts YouTube video URLs for this essential background.

From stephen at xemacs.org  Wed Jul  8 04:19:39 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 08 Jul 2009 11:19:39 +0900
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <Pine.LNX.4.64.0907071003460.6306@kimball.webabinitio.net>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<4A520F83.7020300@egenix.com>
	<bbaeab100907061049o1666b988x785e38d8d3d1ac8f@mail.gmail.com>
	<4A534CFC.30909@egenix.com>
	<Pine.LNX.4.64.0907071003460.6306@kimball.webabinitio.net>
Message-ID: <871vorvrus.fsf@uwakimon.sk.tsukuba.ac.jp>

R. David Murray writes:
 > On Tue, 7 Jul 2009 at 15:26, M.-A. Lemburg wrote:

 > > I've seen a few discussions about this, but no final statement
 > > of what strategy to follow and whether hg makes this easier (AFAIR,
 > > that was the main argument for switching to hg).

Yes, yes, yes, and no.  In reverse order, no: the main argument for
switching to hg is that it makes private branching easier.

Yes: hg will make public branching easier, too, but that can't be
proved until the workflow adjustments get worked out.  For that
reason, it is essential that the current workflow be supportable
essentially without change, and it is.  With respect to "how", I'm not
a Mercurial geek, but I have been working with Mercurial queues a bit
recently in another area, and I think they have some promise for
helping organize the workflow.  (Although by themselves they're
clearly not sufficient, since they're oriented to a single branch.)

Yes: there has been no final statement of what strategy to follow
because opinions are extremely varied, even as to what is feasible
with Mercurial.  For example, Dirkjan and Georg want a workflow that
makes moving patches among the public branches worry-free Mercurial
merges.  I believe that means (to the extent it is implemented)
essentially gutting the current strategy of cutting maintenance
branches and simply lagging the maintenance branches relative to the
dev branches, and that it's infeasible for py3k vs. py2.  I can't
substantiate that; maybe the patch flow would support what they want,
I'm not that familiar with how much patches currently morph across
branches.

And yes: there are a few inconclusive discussions.  That's why PEP 374
was written consciously with the intent of postponing the hard issues
of workflow across the public branches in favor of picking off the low
hanging fruit of private branching and disconnected version control.

 > I think the main reason for switching was that it would make it easier
 > for non-core-committers to maintain branches and submit patches (as
 > changesets core committers can pull).

Patches or "bundles" aka merge directives.  "Pulling" submissions is
probably not going to happen; that's a practice that is common with
highly distributed workflows, but Python has a fairly centralized
workflow.

 > but I think it remains to be proven/worked out.  And I believe there is
 > no tool like svnmerge for tracking changesets to be merged, which could
 > be an issue that needs a resolution.

I think that Mercurial queues or some related extension can be adapted
to this, but I can't say for sure (after all, I was the git person on
the PEP 374 team :-).

 > IIUC, the discussion about named versus cloned branches is part of
 > figuring out the workflow....

Peripherally.  But actually it's not really relevant to workflow.
Anything that can be done with named branches can be done with cloned
branches, possibly requiring substantially more space.  That
discussion really is about whether anything is *lost* by using named
branches.  I worry that something is lost, but the discussion so far
has been inconclusive.

From ideasman42 at gmail.com  Wed Jul  8 06:06:07 2009
From: ideasman42 at gmail.com (Campbell Barton)
Date: Tue, 7 Jul 2009 21:06:07 -0700
Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
In-Reply-To: <e7ba66e40907071518p6e4cab82y3100eaa39bbe802c@mail.gmail.com>
References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>
	<4A53C24B.7080202@v.loewis.de>
	<e7ba66e40907071516o7f76e4a0q8bf277611cb60470@mail.gmail.com>
	<e7ba66e40907071518p6e4cab82y3100eaa39bbe802c@mail.gmail.com>
Message-ID: <7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com>

The reason I was a big vague is that I'm not really bothered HOW its
done, I'd just like there to be some way to use PySys_SetArgv with a
(char **)

Definitely not suggesting that PySys_SetArgv be reverted to the way it
worked in py2.x, if there was 2 versions of PySys_SetArgv that would
be acceptable to me. PySys_SetArgvASCII() maybe?

Alternately a function to convert the char **argv to wchar_t **argv
could be ok too.

PyOS_char2wchar would be useful though it still means you need to
manually copy the argv, convert and free it.


On Tue, Jul 7, 2009 at 3:18 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Sorry...
>
> ---------- Forwarded message ----------
> From: Lisandro Dalcin <dalcinl at gmail.com>
> Date: Tue, Jul 7, 2009 at 7:16 PM
> Subject: Re: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
> To: "Martin v. L?wis" <martin at v.loewis.de>
>
>
> On Tue, Jul 7, 2009 at 6:46 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>>> Since many C applications take argv as a (char **) it seems reasonable
>>> for python to keep a function for the C api that accepts a ?(char **)
>>> argument for argv.
>>
>
> I've run in the same issue, and basically had to copy&paste from
> Python sources...
>
>>
>> I'm not quite sure what you are suggesting: either that there is a
>> function in the C API that accepts a (char**), or that PySys_SetArgv
>> be that function.
>>
>> The latter alternative is out of question, as it breaks the Windows
>> port of Python - there is simply no way of representing argv with
>> char** in a meaningful way. It also breaks backwards compatibility.
>>
>
> Indeed...
>
>> As for creating another function: please provide a patch to
>> bugs.python.org. I believe this patch is *very* difficult to
>> implement, unless the function can accept some severe limitations.
>>
>
> What about the Python lib exposing a public PyOS_char2wchar() ?
>
> No idea how to write a Windows implementation, though..
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
- Campbell

From pje at telecommunity.com  Wed Jul  8 07:09:09 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 01:09:09 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com
 >
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com>
	<79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
Message-ID: <20090708050608.AC2C63A407B@sparrow.telecommunity.com>

At 11:30 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:
> > At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
> >>
> >> When I started to work on this I didn't realize the gigantic amount of
> >> work and coordination it requires
> >
> > No one expects the package inquisition.  ;-)
> >
> >
>
>Sorry, i've looked in the english dictionary but I don't get this one.
>what do you mean ?

It's a Monty Python reference, i.e. "no one expects the Spanish Inquisition".

In the Spanish Inquisition sketch, a character is asked several 
questions he can't answer, and then complains that he wasn't 
expecting some kind of Spanish inquisition, at which point three 
scarlet-robed cardinals burst into the room and exclaim, "NO ONE 
expects the Spanish Inquisition!"

So I punned on that in order to imply that nobody who takes on the 
job of packaging expects to be asked the kind of detailed, 
unanswerable questions that come with the territory of packaging, or 
to be subjected to torture by third parties merely for trying to do 
something useful.

And, by further implication, I was thus expressing camaraderie with 
you on your initiation into the wonderful world of trying to satisfy 
everyone in Pythonland, and our shared experience in the area of "if 
I'd really known what I was getting into, I probably wouldn't have".  ;-)


From martin at v.loewis.de  Wed Jul  8 08:13:18 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Jul 2009 08:13:18 +0200
Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ?
In-Reply-To: <7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com>
References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com>	<4A53C24B.7080202@v.loewis.de>	<e7ba66e40907071516o7f76e4a0q8bf277611cb60470@mail.gmail.com>	<e7ba66e40907071518p6e4cab82y3100eaa39bbe802c@mail.gmail.com>
	<7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com>
Message-ID: <4A5438FE.3050608@v.loewis.de>

> The reason I was a big vague is that I'm not really bothered HOW its
> done, I'd just like there to be some way to use PySys_SetArgv with a
> (char **)

Ok, that's easy:

void PySys_SetArgvChar(int argc, char** argv)
{
  PySys_SetArgv(0, NULL);
}

So you don't bother *how* it's done as long as you can pass char**
in some way.

> Alternately a function to convert the char **argv to wchar_t **argv
> could be ok too.

Convert in what manner? What is the encoding of your char**?

Regards,
Martin

From ziade.tarek at gmail.com  Wed Jul  8 08:26:45 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 8 Jul 2009 08:26:45 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090708050608.AC2C63A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A530DFD.40802@egenix.com>
	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>
	<87k52kpv5l.fsf@benfinney.id.au>
	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
Message-ID: <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>

On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby<pje at telecommunity.com> wrote:
> At 11:30 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>>
>> On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje at telecommunity.com> wrote:
>> > At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote:
>> >>
>> >> When I started to work on this I didn't realize the gigantic amount of
>> >> work and coordination it requires
>> >
>> > No one expects the package inquisition. ?;-)
>> >
>> >
>>
>> Sorry, i've looked in the english dictionary but I don't get this one.
>> what do you mean ?
>
> It's a Monty Python reference, i.e. "no one expects the Spanish
> Inquisition".
>
> In the Spanish Inquisition sketch, a character is asked several questions he
> can't answer, and then complains that he wasn't expecting some kind of
> Spanish inquisition, at which point three scarlet-robed cardinals burst into
> the room and exclaim, "NO ONE expects the Spanish Inquisition!"
>
> So I punned on that in order to imply that nobody who takes on the job of
> packaging expects to be asked the kind of detailed, unanswerable questions
> that come with the territory of packaging, or to be subjected to torture by
> third parties merely for trying to do something useful.
>
> And, by further implication, I was thus expressing camaraderie with you on
> your initiation into the wonderful world of trying to satisfy everyone in
> Pythonland, and our shared experience in the area of "if I'd really known
> what I was getting into, I probably wouldn't have". ?;-)

Hehe very nice, thanks for the explanation :)

(thanks to the others too)

>
>



-- 
Tarek Ziad? | http://ziade.org

From ncoghlan at gmail.com  Wed Jul  8 12:41:26 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Jul 2009 20:41:26 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<4A530DFD.40802@egenix.com>	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>	<87k52kpv5l.fsf@benfinney.id.au>	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
Message-ID: <4A5477D6.1080708@gmail.com>

Tarek Ziad? wrote:
> On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby<pje at telecommunity.com> wrote:
>> So I punned on that in order to imply that nobody who takes on the job of
>> packaging expects to be asked the kind of detailed, unanswerable questions
>> that come with the territory of packaging, or to be subjected to torture by
>> third parties merely for trying to do something useful.
>>
>> And, by further implication, I was thus expressing camaraderie with you on
>> your initiation into the wonderful world of trying to satisfy everyone in
>> Pythonland, and our shared experience in the area of "if I'd really known
>> what I was getting into, I probably wouldn't have".  ;-)
> 
> Hehe very nice, thanks for the explanation :)
> 
> (thanks to the others too)

I think is exchange actually a good reminder that the bar for acceptance
of PEP 376 shouldn't be "Addresses every possible packaging issue we can
come up with". That would be setting expectations far too high because
packaging cross-platform is so messy.

A more appropriate bar is probably:
  a) Better that what we have now; and
  b) Provides possible avenues for even more improvement in the future

For example, I quite like the concept behind the various ideas for
"location" or "prefix" definitions either in the RECORD file itself or
in a separate PREFIXES file, since such approaches feeds directly in to
part b) above. Once some kind prefix definition mechanism is in place in
the metadata, it becomes possible to leverage it to categorise files for
the benefit of system packagers while still allowing simple
internal-use-only packages to just bundle everything together.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Wed Jul  8 12:50:59 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 8 Jul 2009 10:50:59 +0000 (UTC)
Subject: [Python-Dev] PEP 376 - Open questions
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<4A530DFD.40802@egenix.com>	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>	<87k52kpv5l.fsf@benfinney.id.au>	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
Message-ID: <loom.20090708T104939-364@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> I think is exchange actually a good reminder that the bar for acceptance
> of PEP 376 shouldn't be "Addresses every possible packaging issue we can
> come up with". That would be setting expectations far too high because
> packaging cross-platform is so messy.

+1

> For example, I quite like the concept behind the various ideas for
> "location" or "prefix" definitions either in the RECORD file itself or
> in a separate PREFIXES file, since such approaches feeds directly in to
> part b) above.

I don't understand the point. Since RECORD is specific to a particular machine
and installation (it is generated at install time), what would you gain by
adding a fake genericity to its entries?

Regards

Antoine.



From ncoghlan at gmail.com  Wed Jul  8 13:08:16 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Jul 2009 21:08:16 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <loom.20090708T104939-364@post.gmane.org>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<4A530DFD.40802@egenix.com>	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>	<87k52kpv5l.fsf@benfinney.id.au>	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>	<4A5477D6.1080708@gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
Message-ID: <4A547E20.5030606@gmail.com>

Antoine Pitrou wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> For example, I quite like the concept behind the various ideas for
>> "location" or "prefix" definitions either in the RECORD file itself or
>> in a separate PREFIXES file, since such approaches feeds directly in to
>> part b) above.
> 
> I don't understand the point. Since RECORD is specific to a particular machine
> and installation (it is generated at install time), what would you gain by
> adding a fake genericity to its entries?

My understanding is that RECORD isn't generated at install time for the
bdist_* commands -it's generated at the time the binary distribution is
created. Hence the problems raised with the idea of having absolute
paths in RECORD - if they were written directly into RECORD they would
reflect where the files were when the distribution was created, not
where they end up when it is installed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From p.f.moore at gmail.com  Wed Jul  8 14:13:55 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 8 Jul 2009 13:13:55 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A547E20.5030606@gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com> <loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
Message-ID: <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>

2009/7/8 Nick Coghlan <ncoghlan at gmail.com>:
> Antoine Pitrou wrote:
>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>> For example, I quite like the concept behind the various ideas for
>>> "location" or "prefix" definitions either in the RECORD file itself or
>>> in a separate PREFIXES file, since such approaches feeds directly in to
>>> part b) above.
>>
>> I don't understand the point. Since RECORD is specific to a particular machine
>> and installation (it is generated at install time), what would you gain by
>> adding a fake genericity to its entries?
>
> My understanding is that RECORD isn't generated at install time for the
> bdist_* commands -it's generated at the time the binary distribution is
> created. Hence the problems raised with the idea of having absolute
> paths in RECORD - if they were written directly into RECORD they would
> reflect where the files were when the distribution was created, not
> where they end up when it is installed.

This is a key open question, IMHO. No-one has done any work on
updating the core bdist_xxx commands to the new structure, and the PEP
is silent on the matter, so it's still all speculation.

The current "best guess" assumption is that, given that bdist_xxx uses
a "setup.py install --root=xxx" incantation to generate the structure
to package, the RECORD can be picked up from this. There are a number
of issues with this assumption, though - it doesn't handle absolute
locations well (there are existing bugs), paths may be affected by the
environment at the time the installer is built, and it requires RECORD
to be relocatable.

An alternative solution would be for the bdist_xxx commands to ignore
the RECORD file generated in the temp area, and build its own on the
target machine when the installer is run. This is conceptuially far
cleaner, it's in line with the implicit assumptions in the PEP, and it
avoids a lot of the problems with path formats (the filenames in
RECORD could be absolute local pathnames, and that's the end of the
story). But - and it's a big but - this dumps RECORD file handling on
the bdist_xxx command, and they all need to implement the same code,
resulting in a lot of duplication and opportunity for inconsistency.

I have no answer here. But I think we need feedback from the people
who will ultimately be building bdist_xxx formats - Debian packagers,
people wrting alternate RPM builders, etc. There's also Phillip
commenting from the setuptools POV and MAL who has mentioned his build
process.

Bottom line - Is RECORD to be created on the target machine at install
time, or must it be relocatable?
Subsequent question - if it's to be relocatable, how will that work?

Paul.

From solipsis at pitrou.net  Wed Jul  8 14:24:34 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 8 Jul 2009 12:24:34 +0000 (UTC)
Subject: [Python-Dev] PEP 376 - Open questions
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
Message-ID: <loom.20090708T121713-308@post.gmane.org>

Paul Moore <p.f.moore <at> gmail.com> writes:
> 
> I have no answer here. But I think we need feedback from the people
> who will ultimately be building bdist_xxx formats - Debian packagers,
> people wrting alternate RPM builders, etc.

I think any bdist_xxx command which targets externally handled package formats
(rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
dpkg, you will uninstall with the same tool, won't you?

> Bottom line - Is RECORD to be created on the target machine at install
> time, or must it be relocatable?

This is quite an uninformed opinion, but a relocatable RECORD looks like a can
of worms to me.

As for creating RECORD at install time, I don't understand the argument about
duplicate code. Distutils can provide useful primitives for generating RECORD,
so that interested subcommands don't have a lot of additional work to do.



From p.f.moore at gmail.com  Wed Jul  8 14:54:49 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 8 Jul 2009 13:54:49 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <loom.20090708T121713-308@post.gmane.org>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com> <loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
Message-ID: <79990c6b0907080554t49de466cpde75258fa8e02bee@mail.gmail.com>

2009/7/8 Antoine Pitrou <solipsis at pitrou.net>:
> Paul Moore <p.f.moore <at> gmail.com> writes:
>>
>> I have no answer here. But I think we need feedback from the people
>> who will ultimately be building bdist_xxx formats - Debian packagers,
>> people wrting alternate RPM builders, etc.
>
> I think any bdist_xxx command which targets externally handled package formats
> (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
> dpkg, you will uninstall with the same tool, won't you?

I thought the purpose of RECORD was (also) so that *other* installs
could check who "owned" an existing file, before potentially
overwriting it. That seems like it's the reasoning behind the uses()
function.

But the PEP is very light on practical applications, so it's hard to
tell what the methods are intended to be used for.

>> Bottom line - Is RECORD to be created on the target machine at install
>> time, or must it be relocatable?
>
> This is quite an uninformed opinion, but a relocatable RECORD looks like a can
> of worms to me.
>
> As for creating RECORD at install time, I don't understand the argument about
> duplicate code. Distutils can provide useful primitives for generating RECORD,
> so that interested subcommands don't have a lot of additional work to do.

The bdist_wininst code is in C, and the bdist_msi code is driven by
the MSI data format. I don't know what scripting options exist for
.deb and .rpm files. Different languages, so no code sharing seems
possible. Maybe I'm wrong, and something could be done along the lines
of a "pkgutil.generate_record_file()" function which could be run from
a post-install script written in Python, but I don't know how
plausible or robust that would be...

Paul.

From eric at trueblade.com  Wed Jul  8 14:59:57 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 08:59:57 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <loom.20090708T121713-308@post.gmane.org>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>	<4A5477D6.1080708@gmail.com>	<loom.20090708T104939-364@post.gmane.org>	<4A547E20.5030606@gmail.com>	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
Message-ID: <4A54984D.4020305@trueblade.com>

Antoine Pitrou wrote:
> Paul Moore <p.f.moore <at> gmail.com> writes:
>> I have no answer here. But I think we need feedback from the people
>> who will ultimately be building bdist_xxx formats - Debian packagers,
>> people wrting alternate RPM builders, etc.
> 
> I think any bdist_xxx command which targets externally handled package formats
> (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or
> dpkg, you will uninstall with the same tool, won't you?

I agree with this. For RPM's, there's a whole other database of what 
files were installed. Is it really the intent that a file will be 
managed by multiple different installers? That I can install with RPM 
but remove with some python-installer (or other) tool? That way lies 
madness. In fact, I see RECORD as an installer-specific detail that 
doesn't need to be standardized at all.

For bdist_rpm (which I'm contemplating taking over, but no promises), 
the only thing that needs to be done with the metadata is be able to 
build a .spec file. Once the .spec file is generated, rpmbuild (or 
equivalent) will produce the .rpm files. I image that .deb's are 
similar, but I'm no expert.

Which brings up the "static metadata" argument I was making at PyCon. I 
want there to be enough static metadata that the new bdist_rpm can read 
so that it produces a well-behaved .spec file. I need to know the intent 
of some of the files (which are documentation, which are config files, 
which are scripts, which are libraries, etc.) and not much else.

I envision the new bdist_rpm as a standalone tool, not a distutils 
extension.

>> Bottom line - Is RECORD to be created on the target machine at install
>> time, or must it be relocatable?
> 
> This is quite an uninformed opinion, but a relocatable RECORD looks like a can
> of worms to me.

Agreed.

> As for creating RECORD at install time, I don't understand the argument about
> duplicate code. Distutils can provide useful primitives for generating RECORD,
> so that interested subcommands don't have a lot of additional work to do.

I envision a "distlib" that contains new support code that's not part of 
the current distutils morass.

From erik at cq2.nl  Wed Jul  8 14:43:29 2009
From: erik at cq2.nl (Erik Groeneveld)
Date: Wed, 8 Jul 2009 14:43:29 +0200
Subject: [Python-Dev] Define metatype and a type that uses it
Message-ID: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>

L.S.

I am working on an extension that allows one to open Java .jar files
and use the objects defined in it (via CNI or JNI) in Python.  It
works like a charm, but I am stuck with the definition of a metatype
for the wrappers.  I have to do this in Python now, with a helper
class.

I did find examples of how to define metatypes, but I can't find
anything on how to declare one type using this metatype.

Could anyone point me to examples or documentation or help me fix my code below?

What I have is a type JObject that has a metatype JObjectMeta, but the
objects created from it seem not properly initialized.   If I call
PyString_Str(self) in JObjectMeta_init, it seg faults.  De Python code
that initiates this is:

class MyJObject(JObject):
    pass

Below is the complete, contained code of the extension.

Thanks in advance,
Erik Groeneveld


#include <Python.h>

/*************** JObjectMeta *******************/
typedef struct {
    PyObject_HEAD
} JObjectMeta;

PyTypeObject JObjectMetaType = {
    PyObject_HEAD_INIT(NULL)
};

int JObjectMeta_init(PyObject* self, PyObject* args, PyObject *kwargs) {
    printf("before str\n");
    PyObject_Str(self);
    printf("after str\n");
    return 0;
}

PyObject* JObjectMeta_getattro(PyObject *self, PyObject *name) {
    return PyBaseObject_Type.tp_getattro(self, name);
}


/*************** JObject ***********************/
typedef struct {
    PyObject_HEAD
} JObject;

PyTypeObject JObjectType = {
    PyObject_HEAD_INIT(&JObjectMetaType)
};

int JObject_init(PyObject* self, PyObject* args, PyObject *kwargs) {
    return 0;
}

static PyMethodDef methods[] = {
    {NULL, NULL, 0, NULL}
};

/*************** INIT *******************/

PyMODINIT_FUNC
initmetaclass(void) {
    JObjectMetaType.tp_name         = "metaclass.JObjectMeta";
    JObjectMetaType.tp_basicsize    = sizeof(JObjectMeta);
    JObjectMetaType.tp_flags        = Py_TPFLAGS_DEFAULT;
    JObjectMetaType.tp_doc          = "JObject objects";
    JObjectMetaType.tp_init         = JObjectMeta_init;
    JObjectMetaType.tp_base         = &PyType_Type; // must ?
    JObjectMetaType.tp_new          = PyType_GenericNew;
    JObjectMetaType.tp_getattro     = JObjectMeta_getattro;

    JObjectType.tp_name         = "metaclass.JObject";
    JObjectType.tp_basicsize    = sizeof(JObject);
    JObjectType.tp_flags        = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
    JObjectType.tp_doc          = "JObjectMeta objects";
    JObjectType.tp_init         = JObject_init;
    JObjectType.tp_base         = &JObjectMetaType; // must ?
    JObjectType.tp_new          = PyType_GenericNew;

    PyObject* module = Py_InitModule3("metaclass", methods, "Metaclass
try out.");
    if (module == NULL) {
        PyErr_Print();
        exit(-1);
    }
    if (PyType_Ready(&JObjectMetaType) < 0) {
        PyErr_Print();
        exit(-1);
    }
    if (PyModule_AddObject(module, "JObjectMeta", (PyObject*)
&JObjectMetaType) < 0) {
        PyErr_Print();
        exit(-1);
    }
    if (PyType_Ready(&JObjectType) < 0) {
        PyErr_Print();
        exit(-1);
    }
    if (PyModule_AddObject(module, "JObject", (PyObject*) &JObjectType) < 0 ) {
        PyErr_Print();
        exit(-1);
    }
}

From lists at cheimes.de  Wed Jul  8 15:47:45 2009
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 08 Jul 2009 15:47:45 +0200
Subject: [Python-Dev] Define metatype and a type that uses it
In-Reply-To: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
References: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
Message-ID: <h32820$7mr$1@ger.gmane.org>

Erik Groeneveld wrote:
> L.S.
> 
> I am working on an extension that allows one to open Java .jar files
> and use the objects defined in it (via CNI or JNI) in Python.  It
> works like a charm, but I am stuck with the definition of a metatype
> for the wrappers.  I have to do this in Python now, with a helper
> class.

Did you know that Andi Vajda has already created a fantastic wrapper for
Java called JCC? http://pypi.python.org/pypi/JCC/2.3

Contrary to your project it doesn't create the wrappers in runtime. It
analyzes the jar file(s) with JNI, created C++ templates and wraps the
C++ code in Python wrappers. JCC even allows you to implement native
methods of Java classes in Python.

Christian


From p.f.moore at gmail.com  Wed Jul  8 16:57:08 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 8 Jul 2009 15:57:08 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A54AF4F.9080800@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com> <loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
Message-ID: <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>

2009/7/8 Eric Smith <eric at trueblade.com>:
> [Any reason you didn't respond to the list? I'd like to continue the
> discussion there. But I'm currently replying just to you.]

No, I just clicked the wrong button. Sorry. Redirecting back to the list.

> Paul Moore wrote:
>>
>> 2009/7/8 Eric Smith <eric at trueblade.com>:
>>>
>>> I agree with this. For RPM's, there's a whole other database of what
>>> files
>>> were installed. Is it really the intent that a file will be managed by
>>> multiple different installers? That I can install with RPM but remove
>>> with
>>> some python-installer (or other) tool? That way lies madness. In fact, I
>>> see
>>> RECORD as an installer-specific detail that doesn't need to be
>>> standardized
>>> at all.
>>
>> I thought RECORD was to ensure that you can install with RPM, and
>> *not* have any other random uninstaller remove your files (because you
>> populate RECORD, and the other uninstaller uses
>> pkgutil.get_file_users(path) to find out that you care about the file,
>> and hence avoid deleting it).
>
> I really don't think this is a use case worth supporting. A single file
> managed by multiple installers is a mess waiting to happen. RPM won't let
> you do it. About the only use case I can think of is namespace packages, and
> that can be handled by factoring out the __init__.py file into it's own
> install package.

Personally, I think you're right. But I have essentially zero
experience with the non-Windows use cases, so I don't feel qualified
to judge.

It seems to me that a lot of the fancy detecting what other people do
stuff is motivated by setuptools. I'm not going to comment on that,
other than to say that it may be that setuptools is generating more
problems than it's solving...

>> Basically, other install methods can't read your (RPM) database, and
>> you can't read their database. RECORD is a simple common format,
>> providing just enough information for everyone to communicate the
>> message "I manage this file".
>
> The RPM uninstaller won't look at the RECORD file. I think the best we can
> say is that _if_ an installer wants to share state about what's installed
> with other installers, this is how it's done.

OK. I'd be stunned if bdist_wininst or bdist_msi ends up checking RECORD.

So that's 3 "not interested" votes.

Who's going to use the APIs based around the RECORD file? Anyone?

Seriously - who is actually going to USE any of the APIs being
proposed in PEP 376? If no-one's interested, we should stop right now.
Do any of the people from the summit read this list? If not, can
anyone contact them and get some feedback?

Paul.

From pje at telecommunity.com  Wed Jul  8 17:11:44 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 11:11:44 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A54984D.4020305@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
Message-ID: <20090708150844.EC3843A407B@sparrow.telecommunity.com>

At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:
>An alternative solution would be for the bdist_xxx commands to ignore
>the RECORD file generated in the temp area, and build its own on the
>target machine when the installer is run. This is conceptuially far
>cleaner, it's in line with the implicit assumptions in the PEP, and it
>avoids a lot of the problems with path formats (the filenames in
>RECORD could be absolute local pathnames, and that's the end of the
>story). But - and it's a big but - this dumps RECORD file handling on
>the bdist_xxx command, and they all need to implement the same code,
>resulting in a lot of duplication and opportunity for inconsistency.

The PREFIXES approach I suggested doesn't have these problems; the 
bdist installer only needs to write a PREFIXES file, and doesn't need 
to touch RECORD (which would be generated by the install --root).  In 
fact, since PREFIXES is strictly advisory, it can be generated by the 
install --root and left alone at installation time.  And, if it's 
generated without the --root prefix included, RPMs would have a 
correct PREFIXES file without further manipulation.  Plus, the 
bdist_wininst .exe installer can easily write a PREFIXES file, since 
it has to know the prefixes in order to do an installation in the first place.

(We still need to hash out an initial list of prefixes, of course.)


At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
>I agree with this. For RPM's, there's a whole other database of what 
>files were installed. Is it really the intent that a file will be 
>managed by multiple different installers? That I can install with 
>RPM but remove with some python-installer (or other) tool? That way 
>lies madness. In fact, I see RECORD as an installer-specific detail 
>that doesn't need to be standardized at all.

This is a misunderstanding.  The purpose is to let an *installer* 
(like easy_install) know that these files belong to some other 
installer, and not allow overwriting them.  That's why there's also 
an INSTALLER file.


From pje at telecommunity.com  Wed Jul  8 17:19:48 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 11:19:48 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.co
 m>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
Message-ID: <20090708151647.9E0703A407B@sparrow.telecommunity.com>

At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:
>Who's going to use the APIs based around the RECORD file? Anyone?

The distutils uninstall facility, for starters.  easy_install and pip 
also will, eventually.

In all three cases, the use will be in order to avoid overwriting 
files belonging to a system package manager, and in order to 
uninstall/upgrade their own files.

(I can't personally commit that pip will need this; but it's 
currently the only installer besides the distutils that installs 
things in this format, and would therefore need this to prevent 
overwriting things.)

Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES 
appropriately for both development and actual installations.


>Seriously - who is actually going to USE any of the APIs being
>proposed in PEP 376? If no-one's interested, we should stop right now.
>Do any of the people from the summit read this list? If not, can
>anyone contact them and get some feedback?

Package management people aren't likely to be doing anything with 
this stuff directly; if they need a list of installed files they're 
probably using --root or --record at package build time, not needing 
to sniff stuff and installation time, and they already have 
installation facilities.  Their only real need in relation to this 
will be to set an option to generate an INSTALLER file that says you 
can't uninstall their packages manually, thereby preventing the 
package from being uninstalled by the distutils utility, or any other 
well-behaved PEP 376-based package management tool.


From amauryfa at gmail.com  Wed Jul  8 17:35:21 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 8 Jul 2009 17:35:21 +0200
Subject: [Python-Dev] Define metatype and a type that uses it
In-Reply-To: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
References: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
Message-ID: <e27efe130907080835p3dedf14fi7a8044f7c60b049d@mail.gmail.com>

Hello,

2009/7/8 Erik Groeneveld <erik at cq2.nl>:
> I am working on an extension that allows one to open Java .jar files
> and use the objects defined in it (via CNI or JNI) in Python. ?It
> works like a charm, but I am stuck with the definition of a metatype
> for the wrappers. ?I have to do this in Python now, with a helper
> class.
>
> I did find examples of how to define metatypes, but I can't find
> anything on how to declare one type using this metatype.
>
> Could anyone point me to examples or documentation or help me fix my code below?
[...]

There are several changes to do in your code:
- Don't define a JObjectMeta struct, use JObjectType directly instead.
An instance of the metatype is the type itself!
- Don't set JObjectMetaType.tp_basicsize, let it inherit from the base
type (the correct value would be sizeof(PyHeapTypeObject), this is
important in order to create derived classes in python)
- Don't set JObjectMetaType.tp_new, let it inherit from the base type
(PyType_GenericNew just allocates memory; types are more complex...)
- JObjectType.tp_base = &JObjectMetaType" is wrong, it should be
"JObjectType.ob_type = &JObjectMetaType;", or better to be compatible
with python3: "Py_TYPE(JObjectType) = &JObjectMetaType;"

In summary, I made your example work with:
   JObjectMetaType.tp_name         = "metaclass.JObjectMeta";
   JObjectMetaType.tp_flags        = Py_TPFLAGS_DEFAULT;
   JObjectMetaType.tp_doc          = "JObjectMeta objects";
   JObjectMetaType.tp_init         = JObjectMeta_init;
   JObjectMetaType.tp_base         = &PyType_Type;
   JObjectMetaType.tp_getattro     = JObjectMeta_getattro;

   Py_TYPE(&JObjectType)       = &JObjectMetaType;
   JObjectType.tp_name         = "metaclass.JObject";
   JObjectType.tp_basicsize    = sizeof(JObject);
   JObjectType.tp_flags        = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE;
   JObjectType.tp_doc          = "JObject objects";
   JObjectType.tp_init         = JObject_init;


-- 
Amaury Forgeot d'Arc

From ziade.tarek at gmail.com  Wed Jul  8 17:36:36 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 8 Jul 2009 17:36:36 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090708150844.EC3843A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com> <loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<20090708150844.EC3843A407B@sparrow.telecommunity.com>
Message-ID: <94bdd2610907080836q55d68de4od7a8b55d1ee1ae0a@mail.gmail.com>

On Wed, Jul 8, 2009 at 5:11 PM, P.J. Eby<pje at telecommunity.com> wrote:
> At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:
>>
>> An alternative solution would be for the bdist_xxx commands to ignore
>> the RECORD file generated in the temp area, and build its own on the
>> target machine when the installer is run. This is conceptuially far
>> cleaner, it's in line with the implicit assumptions in the PEP, and it
>> avoids a lot of the problems with path formats (the filenames in
>> RECORD could be absolute local pathnames, and that's the end of the
>> story). But - and it's a big but - this dumps RECORD file handling on
>> the bdist_xxx command, and they all need to implement the same code,
>> resulting in a lot of duplication and opportunity for inconsistency.
>
> The PREFIXES approach I suggested doesn't have these problems; the bdist
> installer only needs to write a PREFIXES file, and doesn't need to touch
> RECORD (which would be generated by the install --root). ?In fact, since
> PREFIXES is strictly advisory, it can be generated by the install --root and
> left alone at installation time. ?And, if it's generated without the --root
> prefix included, RPMs would have a correct PREFIXES file without further
> manipulation. ?Plus, the bdist_wininst .exe installer can easily write a
> PREFIXES file, since it has to know the prefixes in order to do an
> installation in the first place.
>
> (We still need to hash out an initial list of prefixes, of course.)

Instead of building this list of prefixes from scratch, let's use the
ones that are *actually* used
when the install command is called, have a look at the list I've suggested :

http://mail.python.org/pipermail/python-dev/2009-July/090514.html

And let's drop other absolute paths,

From p.f.moore at gmail.com  Wed Jul  8 18:22:52 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 8 Jul 2009 17:22:52 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090708151647.9E0703A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org> <4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
Message-ID: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>

2009/7/8 P.J. Eby <pje at telecommunity.com>:
> At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:
>>
>> Who's going to use the APIs based around the RECORD file? Anyone?
>
> The distutils uninstall facility, for starters. ?easy_install and pip also
> will, eventually.

Is pip getting this via setuptools, or will it be independently
changing its code? (Who's the pip contact who could confirm this
directly?)

> In all three cases, the use will be in order to avoid overwriting files
> belonging to a system package manager, and in order to uninstall/upgrade
> their own files.

To uninstall/upgrade their own files counts as "an installer-specific
detail" in the sense that Eric stated it.

To avoid overwriting files belonging to a system package manager: That
sounds like we're looking at a use case where people use a mixture of
tools to install packages into the system Python. Personally, I never
do this (I use bdist_wininst exclusively). The cases I can imagine
are:

- no OS-specific package available, and the user can't build his own,
so he uses setup.py install.
- user using easy_install as well as the OS package

In case (1), I'd imagine the user is careful enough (given that he's
doing everything manually) to avoid installing something that's
already present.
Case (2) is setuptools-related (see below for setuptools-related comments).

> (I can't personally commit that pip will need this; but it's currently the
> only installer besides the distutils that installs things in this format,
> and would therefore need this to prevent overwriting things.)

Not the only one - MAL mentioned his (internal use?) packaging process.

> Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES
> appropriately for both development and actual installations.

If no-one else is using them, that's an internal setuptools detail
again. Or it's for interoperability between setuptools and distutils
(and other packagers don't benefit unless they are using PEP 376
APIs).

>> Seriously - who is actually going to USE any of the APIs being
>> proposed in PEP 376? If no-one's interested, we should stop right now.
>> Do any of the people from the summit read this list? If not, can
>> anyone contact them and get some feedback?
>
> Package management people aren't likely to be doing anything with this stuff
> directly; if they need a list of installed files they're probably using
> --root or --record at package build time, not needing to sniff stuff and
> installation time, and they already have installation facilities. ?Their
> only real need in relation to this will be to set an option to generate an
> INSTALLER file that says you can't uninstall their packages manually,
> thereby preventing the package from being uninstalled by the distutils
> utility, or any other well-behaved PEP 376-based package management tool.

So if I follow this, there's a use for this in making sure the new
distutils uninstall option doesn't break something it shouldn't, but
as the uninstall uses RECORD, it won't be (accidentally) usable on
distributions that didn't follow PEP 376 and create a RECORD file.

There's also a use for this in terms of setuptools/easy_install
interoperability.

I'm now going to take of my "PEP supporter" hat, and put on my "End
user" hat. Please understand that the following is *not* unbiased,
whereas the rest of what I've said *is* intended to be reasonably
impartial.

If the only driver for this PEP is setuptools, then I'm -1 on it.

Unless someone working on a packaging tool *other* than setuptools (or
setuptools-derived projects) speaks up and says "I have code of my own
which uses distutils, and I would benefit as follows from PEP 376 and
I will be changing my code to conform to PEP 376" I think the PEP
should be rejected, and the energy put into some other PEP which will
benefit the wider (non-setuptools) community.

OK, end of the opinion bit. I'll go back to trying to keep things
impartial in future mails.

Paul.

PS Note - I like the idea of the PEP in theory. It's the practical
benefits that I'm trying to establish.

From eric at trueblade.com  Wed Jul  8 19:58:30 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 13:58:30 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090708150844.EC3843A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>	<4A5477D6.1080708@gmail.com>	<loom.20090708T104939-364@post.gmane.org>	<4A547E20.5030606@gmail.com>	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>	<loom.20090708T121713-308@post.gmane.org>	<4A54984D.4020305@trueblade.com>
	<20090708150844.EC3843A407B@sparrow.telecommunity.com>
Message-ID: <4A54DE46.4030404@trueblade.com>

P.J. Eby wrote:
> At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
>> I agree with this. For RPM's, there's a whole other database of what 
>> files were installed. Is it really the intent that a file will be 
>> managed by multiple different installers? That I can install with RPM 
>> but remove with some python-installer (or other) tool? That way lies 
>> madness. In fact, I see RECORD as an installer-specific detail that 
>> doesn't need to be standardized at all.
> 
> This is a misunderstanding.  The purpose is to let an *installer* (like 
> easy_install) know that these files belong to some other installer, and 
> not allow overwriting them.  That's why there's also an INSTALLER file.

I really don't see this use case. Supporting multiple installers for the 
same file (or even just trying to prevent two installers from writing 
the same file)? Wouldn't you be better off just saying an installer 
can't overwrite any existing file?

Likewise, I don't see a use case for installing with one installer and 
uninstalling with another.

Put those two together, and the mechanism that an installer uses to 
record what files it installed is a private implementation detail.

Eric.


From Scott.Daniels at Acm.Org  Wed Jul  8 20:10:52 2009
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Wed, 08 Jul 2009 11:10:52 -0700
Subject: [Python-Dev] Current patch process question
Message-ID: <h32n1n$3uc$1@ger.gmane.org>

I am confused by the current hydra-like development.  I've recently
submitted a change (issue6422 -- a feature request) which is currently
moving through the process.  I am not particularly asking for special
attention on that issue, but I am confused about the process I should be
using (in the face of two trunks).

There seem to be two places to plausibly introduce a change: py3k and
trunk.  Should I submit changes for trunk, py3k, or both?  I tried
asking in the initial feature request, but nobody replied, and I finally
decided that I'd go for patching trunk in the interim.

For myself, I'd like to know if I should add a patch file for the py3k
branch, or will that simply make more work for someone else?  Certainly
it will be easier for me to do this sooner rather than later.

For the wider group, how should I have known the answer to these
questions?  Is there some page I overlooked?  Is there a better forum
in which to ask this question?

--Scott David Daniels
Scott.Daniels at Acm.Org


From solipsis at pitrou.net  Wed Jul  8 20:12:41 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 8 Jul 2009 18:12:41 +0000 (UTC)
Subject: [Python-Dev] Current patch process question
References: <h32n1n$3uc$1@ger.gmane.org>
Message-ID: <loom.20090708T181015-483@post.gmane.org>


Hi,

> There seem to be two places to plausibly introduce a change: py3k and
> trunk.  Should I submit changes for trunk, py3k, or both?  I tried
> asking in the initial feature request, but nobody replied, and I finally
> decided that I'd go for patching trunk in the interim.

Patches are usually first checked into trunk, and then svnmerge'd to py3k.
So it makes more sense to submit your patch against trunk.

> For myself, I'd like to know if I should add a patch file for the py3k
> branch, or will that simply make more work for someone else?

If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take
care of it.
If, on the other hand, porting your patch requires many changes, or delicate
ones, providing a patch against py3k can make the committer's life easier.

Regards

Antoine.



From pje at telecommunity.com  Wed Jul  8 20:33:02 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 14:33:02 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.co
 m>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
Message-ID: <20090708183001.0A8A53A407B@sparrow.telecommunity.com>

At 05:22 PM 7/8/2009 +0100, Paul Moore wrote:
>2009/7/8 P.J. Eby <pje at telecommunity.com>:
> > At 03:57 PM 7/8/2009 +0100, Paul Moore wrote:
> >>
> >> Who's going to use the APIs based around the RECORD file? Anyone?
> >
> > The distutils uninstall facility, for starters.  easy_install and pip also
> > will, eventually.
>
>Is pip getting this via setuptools, or will it be independently
>changing its code? (Who's the pip contact who could confirm this
>directly?)

It would likely be changing its code directly, with respect to 
supporting overwrite-avoidance and uninstall-before-upgrade 
scenarios.  (Generation support would likely come from setuptools, 
dependent on who implemented first.)  Ian Bicking is pip's author.


>In case (1), I'd imagine the user is careful enough (given that he's
>doing everything manually) to avoid installing something that's
>already present.

Obviously, you've never tried doing this yourself.  ;-)

My guess would actually be that in Unixy environments these days, 
people either use virtualenv or buildouts to manage such 
things.  Buildout handles uninstall-before-update (for eggs, anyway, 
I'm not sure about other things) and virtualenv makes it easy to 
create throwaway Python installs.


>So if I follow this, there's a use for this in making sure the new
>distutils uninstall option doesn't break something it shouldn't, but
>as the uninstall uses RECORD, it won't be (accidentally) usable on
>distributions that didn't follow PEP 376 and create a RECORD file.

That's an interesting point.  But for non-distutils tools, it'd be 
nice to be able to find out whether another package manager is 
administering some particular file.  For that matter, the distutils 
themselves should (ultimately) support uninstall-before-upgrade and 
avoid overwriting another package manager's files.


>If the only driver for this PEP is setuptools, then I'm -1 on it.

If it were being driven by setuptools, I'd have just implemented it 
myself and presented it as a fait accompli.  I can't speak to Tarek's 
motives, but I assume that, as stated in the PEP, the primary driver 
is supporting the distutils being able to uninstall things, and 
secondarily to allow other tools to be built on top of the API.


>a packaging tool *other* than setuptools (or setuptools-derived projects)

Is there really such a thing?  ;-)

AFAIK, every published tool for managing Python projects is either 
distutils-based or setuptools-based.  (Things like scons and pymake 
and various other project build tools don't seem to fall under 
"packaging tool" in this sense.)


From pje at telecommunity.com  Wed Jul  8 20:42:32 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 14:42:32 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A54DE46.4030404@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>
	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>
	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>
	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>
	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>
	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<20090708150844.EC3843A407B@sparrow.telecommunity.com>
	<4A54DE46.4030404@trueblade.com>
Message-ID: <20090708183930.E138B3A407B@sparrow.telecommunity.com>

At 01:58 PM 7/8/2009 -0400, Eric Smith wrote:
>I really don't see this use case. Supporting multiple installers for 
>the same file (or even just trying to prevent two installers from 
>writing the same file)? Wouldn't you be better off just saying an 
>installer can't overwrite any existing file?
>
>Likewise, I don't see a use case for installing with one installer 
>and uninstalling with another.
>
>Put those two together, and the mechanism that an installer uses to 
>record what files it installed is a private implementation detail.

You'll have to ask Tarek about that.  As I've said in another email, 
if I were doing this for setuptools I'd just go and do it; it's 
already on my backlog of things to do.  The point of the PEP is to 
provide an uninstall facility for the distutils, and to expose those 
records via an API so that other tools can be built on top of it.

Whether anything other than distutils-based tools will actually use 
the format directly is entirely irrelevant to the PEP's merits, since 
bdist_* tools and platform-specific distutils installers may need to 
read or write the format.  Hence, a PEP is appropriate to describe a 
format and API for the distutils family of tools to operate on.

The specific proposed format is based on an existing de facto 
standard for installed package metadata, so that the many existing 
packaging tools based on that other format and existing APIs can 
easily read or write this data, and so that the API can also be used 
as a way to detect the presence of already-installed distributions of 
software -- even ones installed other ways.


From martin at v.loewis.de  Wed Jul  8 21:41:10 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Jul 2009 21:41:10 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A5477D6.1080708@gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<4A530DFD.40802@egenix.com>	<79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com>	<87k52kpv5l.fsf@benfinney.id.au>	<79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com>	<Pine.LNX.4.64.0907070837180.6306@kimball.webabinitio.net>	<94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com>	<20090707202851.7DBB93A410E@sparrow.telecommunity.com>	<94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com>	<20090708050608.AC2C63A407B@sparrow.telecommunity.com>	<94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com>
	<4A5477D6.1080708@gmail.com>
Message-ID: <4A54F656.8040006@v.loewis.de>

>> Hehe very nice, thanks for the explanation :)
>>
>> (thanks to the others too)
> 
> I think is exchange actually a good reminder that the bar for acceptance
> of PEP 376 shouldn't be "Addresses every possible packaging issue we can
> come up with". That would be setting expectations far too high because
> packaging cross-platform is so messy.

OTOH, it *is* expected that the PEP explicitly lists aspects that it
didn't resolve, so that any reader can find out whether something was
discussed and rejected, or not discussed.

Regards,
Martin



From martin at v.loewis.de  Wed Jul  8 21:44:26 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Jul 2009 21:44:26 +0200
Subject: [Python-Dev] Current patch process question
In-Reply-To: <loom.20090708T181015-483@post.gmane.org>
References: <h32n1n$3uc$1@ger.gmane.org>
	<loom.20090708T181015-483@post.gmane.org>
Message-ID: <4A54F71A.90906@v.loewis.de>

>> There seem to be two places to plausibly introduce a change: py3k and
>> trunk.  Should I submit changes for trunk, py3k, or both?  I tried
>> asking in the initial feature request, but nobody replied, and I finally
>> decided that I'd go for patching trunk in the interim.
> 
> Patches are usually first checked into trunk, and then svnmerge'd to py3k.
> So it makes more sense to submit your patch against trunk.
> 
>> For myself, I'd like to know if I should add a patch file for the py3k
>> branch, or will that simply make more work for someone else?
> 
> If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take
> care of it.
> If, on the other hand, porting your patch requires many changes, or delicate
> ones, providing a patch against py3k can make the committer's life easier.

I usually abstain from "me too" messages, but only this time:
as Antoine says.

Regards,
Martin

From Scott.Daniels at Acm.Org  Wed Jul  8 21:57:14 2009
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Wed, 08 Jul 2009 12:57:14 -0700
Subject: [Python-Dev] Current patch process question
In-Reply-To: <4A54F71A.90906@v.loewis.de>
References: <h32n1n$3uc$1@ger.gmane.org>	<loom.20090708T181015-483@post.gmane.org>
	<4A54F71A.90906@v.loewis.de>
Message-ID: <h32t94$noo$1@ger.gmane.org>

Martin v. L?wis wrote:
>>> There seem to be two places to plausibly introduce a change: py3k and
>>> trunk.  Should I submit changes for trunk, py3k, or both?  ....
>> Patches are usually first checked into trunk, and then svnmerge'd to py3k.
>> So it makes more sense to submit your patch against trunk.
>>
>>> For myself, I'd like to know if I should add a patch file for the py3k
>>> branch, or will that simply make more work for someone else?
>> If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take
>> care of it.
>> If, on the other hand, porting your patch requires many changes, or delicate
>> ones, providing a patch against py3k can make the committer's life easier.
> 
> I usually abstain from "me too" messages, but only this time:
> as Antoine says.

Thanks to both of you for the quick response.  In this case, the port
should be dirt simple, so I'll consider myself done until comments
come back on the patch.

--Scott David Daniels
Scott.Daniels at Acm.Org


From p.f.moore at gmail.com  Wed Jul  8 23:56:15 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 8 Jul 2009 22:56:15 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090708183001.0A8A53A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
Message-ID: <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>

2009/7/8 P.J. Eby <pje at telecommunity.com>:
> If it were being driven by setuptools, I'd have just implemented it myself
> and presented it as a fait accompli. ?I can't speak to Tarek's motives, but
> I assume that, as stated in the PEP, the primary driver is supporting the
> distutils being able to uninstall things, and secondarily to allow other
> tools to be built on top of the API.

My understanding is that all of the various distutils PEPs were driven
by the "packaging summit" ay PyCon. The struggle here seems to be to
find *anyone* from that summit who will now comment on the discussion
:-(

>> a packaging tool *other* than setuptools (or setuptools-derived projects)
>
> Is there really such a thing? ?;-)

I wonder... :-)

> AFAIK, every published tool for managing Python projects is either
> distutils-based or setuptools-based. ?(Things like scons and pymake and
> various other project build tools don't seem to fall under "packaging tool"
> in this sense.)

The big driver seems to be things that take distutils packages and
build Linux packages (debs, rpms, ...) from them. Unfortunately, I
know nothing about these tools, and no-one seems to be speaking up on
the subject.

Maybe this was all thrashed out on the distutils SIG, and consensus
reached there (I only skim that list). If so, maybe this list is the
wrong one to be discussing changes to the PEP on.

Paul.

From eric at trueblade.com  Thu Jul  9 00:07:21 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 18:07:21 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>	
	<loom.20090708T121713-308@post.gmane.org>	
	<4A54984D.4020305@trueblade.com>	
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>	
	<4A54AF4F.9080800@trueblade.com>	
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>	
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>	
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>	
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
Message-ID: <4A551899.1080105@trueblade.com>

Paul Moore wrote:
> 2009/7/8 P.J. Eby <pje at telecommunity.com>:
>> If it were being driven by setuptools, I'd have just implemented it myself
>> and presented it as a fait accompli.  I can't speak to Tarek's motives, but
>> I assume that, as stated in the PEP, the primary driver is supporting the
>> distutils being able to uninstall things, and secondarily to allow other
>> tools to be built on top of the API.
> 
> My understanding is that all of the various distutils PEPs were driven
> by the "packaging summit" ay PyCon. The struggle here seems to be to
> find *anyone* from that summit who will now comment on the discussion
> :-(

I was there, and I've been commenting!

There might have been more discussion after the language summit and the 
one open space event I went to. But the focus as I recall was static 
metadata and version specification. When I originally brought up static 
metadata at the summit, I meant metadata describing the sources in the 
distribution, so that we can get rid of setup.py's. From that metadata, 
I want to be able to generate .debs, .rpms, .eggs, etc.

But I think we've veered into metadata that describes what has been 
installed. I don't think that's so useful. As I've said, this is private 
to the installers. If 2 installers want to communicate with each other 
about what they've installed, then they can agree on that data. I just 
don't find it generally useful for all installers, and therefore not 
useful for distutils.

I'd like to get back to the metadata that describes the source files. 
That's where the real value lies, in my opinion. I'll try and work on a 
post to distutils-sig explaining my thinking.



From p.f.moore at gmail.com  Thu Jul  9 01:07:29 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 9 Jul 2009 00:07:29 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A551899.1080105@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
Message-ID: <79990c6b0907081607s10cfd89dy43cd11b78881f5b9@mail.gmail.com>

2009/7/8 Eric Smith <eric at trueblade.com>:
> I was there, and I've been commenting!

Sorry, I hadn't realised that. Thanks for the correction.

> There might have been more discussion after the language summit and the one
> open space event I went to. But the focus as I recall was static metadata
> and version specification. When I originally brought up static metadata at
> the summit, I meant metadata describing the sources in the distribution, so
> that we can get rid of setup.py's. From that metadata, I want to be able to
> generate .debs, .rpms, .eggs, etc.
>
> But I think we've veered into metadata that describes what has been
> installed. I don't think that's so useful. As I've said, this is private to
> the installers. If 2 installers want to communicate with each other about
> what they've installed, then they can agree on that data. I just don't find
> it generally useful for all installers, and therefore not useful for
> distutils.
>
> I'd like to get back to the metadata that describes the source files. That's
> where the real value lies, in my opinion. I'll try and work on a post to
> distutils-sig explaining my thinking.

OK, that helps a lot. I see how your postings fit into things a little
better now.

Paul.

From cournape at gmail.com  Thu Jul  9 01:42:01 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 9 Jul 2009 08:42:01 +0900
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A551899.1080105@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
Message-ID: <5b8d13220907081642i4c8f4af2u8a59cfb332379cf6@mail.gmail.com>

On Thu, Jul 9, 2009 at 7:07 AM, Eric Smith<eric at trueblade.com> wrote:
> Paul Moore wrote:
>>
>> 2009/7/8 P.J. Eby <pje at telecommunity.com>:
>>>
>>> If it were being driven by setuptools, I'd have just implemented it
>>> myself
>>> and presented it as a fait accompli. ?I can't speak to Tarek's motives,
>>> but
>>> I assume that, as stated in the PEP, the primary driver is supporting the
>>> distutils being able to uninstall things, and secondarily to allow other
>>> tools to be built on top of the API.
>>
>> My understanding is that all of the various distutils PEPs were driven
>> by the "packaging summit" ay PyCon. The struggle here seems to be to
>> find *anyone* from that summit who will now comment on the discussion
>> :-(
>
> I was there, and I've been commenting!
>
> There might have been more discussion after the language summit and the one
> open space event I went to. But the focus as I recall was static metadata
> and version specification. When I originally brought up static metadata at
> the summit, I meant metadata describing the sources in the distribution, so
> that we can get rid of setup.py's. From that metadata, I want to be able to
> generate .debs, .rpms, .eggs, etc.

I agree wholeheartedly. Getting rid of setup.py for most packages
should be a goal IMHO. Most packages don't need anything fancy, and
static metadata are so much easier to use compared to
setup.py/distutils for 3rd party interop.

There was a discussion about how to describe/find the list of files to
form a distribution (for the different sdist/bdist_* commands), but no
agreement was reached. Some people strongly defend the setuptools
feature to get the list of files from the source control system, in
particular.

http://mail.python.org/pipermail/distutils-sig/2009-April/011226.html

David

From stephen at xemacs.org  Thu Jul  9 02:38:46 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 09 Jul 2009 09:38:46 +0900
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A551899.1080105@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
Message-ID: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>

Eric Smith writes:

 > But I think we've veered into metadata that describes what has been 
 > installed. I don't think that's so useful. As I've said, this is private 
 > to the installers. If 2 installers want to communicate with each other 
 > about what they've installed, then they can agree on that data. I just 
 > don't find it generally useful for all installers, and therefore not 
 > useful for distutils.

ISTM that the problem that it solves is uninstall in the absence of
the original installer.  Am I to understand that you don't think that
use case is important?  Or that there's another way to do this?

From steve at pearwood.info  Thu Jul  9 02:51:02 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 9 Jul 2009 10:51:02 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A551899.1080105@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
Message-ID: <200907091051.02623.steve@pearwood.info>

On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote:
> But I think we've veered into metadata that describes what has been
> installed. I don't think that's so useful. As I've said, this is
> private to the installers. If 2 installers want to communicate with
> each other about what they've installed, then they can agree on that
> data. I just don't find it generally useful for all installers, and
> therefore not useful for distutils.

But doesn't this metadata give any two installers a common language to 
use to communicate, instead of having every pair of installers create 
their own private communication method?

Personally, I like to be able to look at a package and say "What did 
that install?" Or contrary-wise, look at a file and say "What package 
installed that?" There are few things worse than discovering a bunch of 
mysterious executable files on your system that you don't remember 
installing, and then spending a few paranoid hours trying to determine 
whether you've stumbled across a root kit or virus or whether they have 
a legitimate reason to be there.



-- 
Steven D'Aprano

From eric at trueblade.com  Thu Jul  9 03:03:38 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 21:03:38 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>	<loom.20090708T121713-308@post.gmane.org>	<4A54984D.4020305@trueblade.com>	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>	<4A54AF4F.9080800@trueblade.com>	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>	<20090708151647.9E0703A407B@sparrow.telecommunity.com>	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4A5541EA.3020503@trueblade.com>

Stephen J. Turnbull wrote:
> Eric Smith writes:
> 
>  > But I think we've veered into metadata that describes what has been 
>  > installed. I don't think that's so useful. As I've said, this is private 
>  > to the installers. If 2 installers want to communicate with each other 
>  > about what they've installed, then they can agree on that data. I just 
>  > don't find it generally useful for all installers, and therefore not 
>  > useful for distutils.
> 
> ISTM that the problem that it solves is uninstall in the absence of
> the original installer.  Am I to understand that you don't think that
> use case is important?  Or that there's another way to do this?

Yes, I think that's a use case that doesn't need to be supported by this 
PEP. It's an issue for a single installer, or a group of installers. 
It's not something that _all_ installers need to care about.

From pje at telecommunity.com  Thu Jul  9 03:07:48 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 21:07:48 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20090709010447.2A86D3A407B@sparrow.telecommunity.com>

At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote:
>Eric Smith writes:
>
>  > But I think we've veered into metadata that describes what has been
>  > installed. I don't think that's so useful. As I've said, this is private
>  > to the installers. If 2 installers want to communicate with each other
>  > about what they've installed, then they can agree on that data. I just
>  > don't find it generally useful for all installers, and therefore not
>  > useful for distutils.
>
>ISTM that the problem that it solves is uninstall in the absence of
>the original installer.

Or uninstall where the installer is "setup.py install", actually.


>   Am I to understand that you don't think that
>use case is important?  Or that there's another way to do this?


From SridharR at activestate.com  Thu Jul  9 03:08:31 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 08 Jul 2009 18:08:31 -0700
Subject: [Python-Dev] PEP 376 - get_egginfo_files
In-Reply-To: <79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com>
References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com>
	<20090705182953.90F7D3A4099@sparrow.telecommunity.com>
	<79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com>
Message-ID: <op.uwrz4hu2brrvlq@double>

On Sun, 05 Jul 2009 11:46:58 -0700, Paul Moore <p.f.moore at gmail.com> wrote:

> 2009/7/5 P.J. Eby <pje at telecommunity.com>:
>> At 05:26 PM 7/5/2009 +0100, Paul Moore wrote:
>>>
>>> def get_distribution(name):
>>>    for d in get_distributions():
>>>        if d.name == name:
>>>            return d
>>>    return None
>>
>> Btw, this is broken code anyway, because it's not handling
>> case-insensitivity or name canonicalization.  (I've mentioned these  
>> issue
>> previously on the distutils-sig.)
> Fair point. (Although I don't recall your distutils-sig posting, so
> I'm not sure what you mean by "name canonicalisation").

Correct me if I'm wrong - I believe he was referring to the `safe_name`  
function:

   http://mail.python.org/pipermail/distutils-sig/2005-September/005120.html

-srid

From SridharR at activestate.com  Thu Jul  9 03:42:43 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 08 Jul 2009 18:42:43 -0700
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org> <4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
Message-ID: <op.uwr1phe9brrvlq@double>

On Wed, 08 Jul 2009 09:22:52 -0700, Paul Moore <p.f.moore at gmail.com> wrote:

> If the only driver for this PEP is setuptools, then I'm -1 on it.
> Unless someone working on a packaging tool *other* than setuptools (or
> setuptools-derived projects) speaks up and says "I have code of my own
> which uses distutils, and I would benefit as follows from PEP 376 and
> I will be changing my code to conform to PEP 376" I think the PEP
> should be rejected, and the energy put into some other PEP which will
> benefit the wider (non-setuptools) community.

Other than easy_install/pip, there is also PyPM which is being developed  
at ActiveState. PyPM is the Python package manager much like what ppm is  
for ActivePerl.

We build packages using 'setup.py install --root', but when it comes  
installation on client .. there is a separate database of installed  
packages that has the following information for each package installed:

    - metadata (name, version, author, etc..)
    - installed_files (list of files installed on client)

So "pypm remove Foo" will simply remove all the 'installed_files' of Foo  
and remove Foo from the package database. The package database is  
currently an sqlite-based one.

PEP 376 definitely will benefit PyPM besides benefiting setuptools/pip. If  
all package managers rely on common API to 1) retrieve metadata of  
installed packages 2) write metadata into site-packages), then it is very  
possible that one package manager does not mess up with the installation  
of others. For example:

   $ pip install IPython
   $ pypm install ipython

Without PEP 376, pypm will overwrite pip's installation. But with PEP 376,  
as pip writes the RECORD file, pypm will reuse it (via the API) and  
co-operate with it.

Eventually, I'd like PEP 376 to support system packagers too. So for  
example, if you did "apt-get install python-pyqt4", then running "pip  
install python-pyqt4" should return without installing anything .. as  
RECORD will be part of the .deb previously installed. As for generating  
the RECORD file, I vote for generating it during install time (w/ absolute  
paths).

-srid

From SridharR at activestate.com  Thu Jul  9 03:51:25 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 08 Jul 2009 18:51:25 -0700
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
Message-ID: <op.uwr13zlrbrrvlq@double>

Is there any reason why RECORD file can't be generated at runtime? Also,  
why should the RECORD file be generated at all by bdist* commands?

A .deb file contains "data.tar.gz" that is simply extracted over "/".  
There is no need for RECORD inside a .deb file because it is implicit in  
the contents on data.tar.gz.

-srid

On Tue, 07 Jul 2009 13:30:05 -0700, P.J. Eby <pje at telecommunity.com> wrote:

> Just an idea...  suppose that instead of using "real" absolute paths in  
> the RECORD file for non-local files (scripts, data, etc) we changed the  
> format to include a "prefix" field, containing something like LIBDIR,  
> SHARE, SCRIPTS, etc., ala bdist_wininst internals?
>
> Also, we could include a separate (optional) PREFIXES file defining what  
> those locations translated to at install time.  Dumb bdists would still  
> have their build paths there (or delete the file before packaging, or  
> use real paths instead of the --root prefixed versions).
>
> Upside: relocation can be detected and handled, RECORD remains  
> cross-platform, and bdists are ok.
>
> Downside: more complex API required to read/manipulate paths and delete  
> files, since you need to be able to check that you have the right  
> prefixes, and may have to ask the user for a prefix you don't recognize,  
> if the default in PREFIXES doesn't match.
>
> Thoughts?
>
> (Also, as a special case, any file that's actually installed to LIBDIR  
> or a subdirectory thereof (even if it's technically a "data" file or  
> script), will probably need to be designated under LIBDIR prefix to  
> prevent runtime breakage in the event any system package maintainers are  
> tempted to use RECORD files as a way of forcing a HFS conformance.   
> These are installation prefixes, *not* content types.)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:  
> http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com

From pje at telecommunity.com  Thu Jul  9 04:12:39 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 22:12:39 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <op.uwr13zlrbrrvlq@double>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
Message-ID: <20090709020938.5C0A83A407B@sparrow.telecommunity.com>

At 06:51 PM 7/8/2009 -0700, Sridhar Ratnakumar wrote:
>Is there any reason why RECORD file can't be generated at runtime?

Applications can be relocatable, yet require plugins and libraries to 
be upgraded, installed, uninstalled, etc.

(Also, RECORD is cross-platform to allow cross-platform 
installation/testing of cross-platform packages; that's why I propose 
keeping the prefix information separate.)


>  Also,
>why should the RECORD file be generated at all by bdist* commands?

bdist commands that use "install --root" will get it as a side 
effect, so they don't have to do anything special.  They needn't even 
generate the PREFIXES, if PREFIXES is also generated by "install 
--root", and omits the root from the prefixes written.  In other 
words, most bdist tools wouldn't change.  bdist_wininst and bdist_msi 
might need to regenerate PREFIXES at actual installation time, but 
AFAIK those are the only ones.  (And the spec would allow for the 
possibility that a PREFIXES file can be incorrect.)


From eric at trueblade.com  Thu Jul  9 04:55:14 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 22:55:14 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <200907091051.02623.steve@pearwood.info>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>
	<200907091051.02623.steve@pearwood.info>
Message-ID: <4A555C12.7000205@trueblade.com>

Steven D'Aprano wrote:
> On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote:
>> But I think we've veered into metadata that describes what has been
>> installed. I don't think that's so useful. As I've said, this is
>> private to the installers. If 2 installers want to communicate with
>> each other about what they've installed, then they can agree on that
>> data. I just don't find it generally useful for all installers, and
>> therefore not useful for distutils.
> 
> But doesn't this metadata give any two installers a common language to 
> use to communicate, instead of having every pair of installers create 
> their own private communication method?

I really don't get this use case of multiple installers trying to 
install the same package. There's just no way that running "yum install 
twisted" and "apt-get install twisted" and "pip install twisted" are 
going to coexist with each other. The best they can do is say "a file 
I'm trying to install already exists". Why try for anything else?
	
> Personally, I like to be able to look at a package and say "What did 
> that install?" Or contrary-wise, look at a file and say "What package 
> installed that?"

I completely support that. All of the system installers I know of 
support this (at least on Linux; I'm familiar with the msi database 
layout (as exposed by Orca) on Windows, but I've never looked to see if 
they record the results of installing an msi). I just see this as an 
issue within each installer. Why would the installers need to 
communicate this with each other?

> There are few things worse than discovering a bunch of 
> mysterious executable files on your system that you don't remember 
> installing, and then spending a few paranoid hours trying to determine 
> whether you've stumbled across a root kit or virus or whether they have 
> a legitimate reason to be there.

We can't solve the problem of "here's an arbitrary file on the system, 
where did it come from?". There are many, many installers outside of our 
control that might have installed something. Ruby has installers, Perl 
has installers, I'm sure there are hundreds of such installers. The user 
might have even used wget to install something.


From eric at trueblade.com  Thu Jul  9 05:20:59 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 23:20:59 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090709010447.2A86D3A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>	<loom.20090708T121713-308@post.gmane.org>	<4A54984D.4020305@trueblade.com>	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>	<4A54AF4F.9080800@trueblade.com>	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>	<20090708151647.9E0703A407B@sparrow.telecommunity.com>	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
Message-ID: <4A55621B.5070904@trueblade.com>

P.J. Eby wrote:
>> ISTM that the problem that it solves is uninstall in the absence of
>> the original installer.
> 
> Or uninstall where the installer is "setup.py install", actually.

I think we need to move away from "setup.py install". It's the 
antithesis of static metadata. setup.py needs to go away.

If we have a static metadata file called, say "setup.info", then I'd 
like to see:
"bdist_rpm setup.info" produce a .rpm
"bdist_msi setup.info" produce a .msi
"bdist_deb setup.info" produce a .deb
"bdist_egg setup.info" produce a .egg
and so on.

There might be a library (and I call dibs on the name "distlib" :) that 
provides support routines to parse setup.info, but there's no framework 
involved. And no need for a plugin system.

There might be support routines used to build setup.info. For those who 
think it's reasonable that the list of source files comes from your 
version control system, build a small command that produces setup.info 
from your vcs. If you need the version number to come from some other 
source, write a small command that reads that and sticks it into a 
setup.info. But the source distribution would be defined by a (possibly 
generated) setup.info.

The metadata in setup.info would be extensible. If setup.info contains 
information that's needed to build a .rpm that's not needed for a .deb, 
then that's fine. We can define the minimum information needed and an 
extension mechanism. As time goes by, some of the extensions might 
become standardized.

Then for each binary format, there's an installer (or more than one!), 
and it might handle dependency detection and automatic fetching, or it 
might not. Some of these exist (apt-get, yum, msi), some might need some 
tweaking (setuptools, pip) and others might need to be written from 
scratch. But the installer, its database of what it has installed, and 
how it handles uninstalls (if it does) is really not the issue.

It's the conflation of "build a package" with "install a package" that 
distutils and setuptools has given us (via setup.py) that leads to us 
never making progress. Let's just focus on "describe a package" and let 
the packagers handle "build a package" and let the installers handle 
"install and uninstall a package". Then the packagers (and their 
associated installers) can evolve independently of each other.

From eric at trueblade.com  Thu Jul  9 05:28:53 2009
From: eric at trueblade.com (Eric Smith)
Date: Wed, 08 Jul 2009 23:28:53 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <op.uwr1phe9brrvlq@double>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double>
Message-ID: <4A5563F5.3020707@trueblade.com>

> Eventually, I'd like PEP 376 to support system packagers too. So for 
> example, if you did "apt-get install python-pyqt4", then running "pip 
> install python-pyqt4" should return without installing anything .. as 
> RECORD will be part of the .deb previously installed. As for generating 
> the RECORD file, I vote for generating it during install time (w/ 
> absolute paths).

I think we should explicitly not support this. What if pip and apt-get 
(or rpm, or others) install the same package in different places because 
of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no 
way we're ever going to get this right, and it's not worth complicating 
our lives over it.

Seriously: Is there some real world use case I'm missing? Does any 
existing install system support this?

From pje at telecommunity.com  Thu Jul  9 05:38:53 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 23:38:53 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A55621B.5070904@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
Message-ID: <20090709033551.AA35D3A407B@sparrow.telecommunity.com>

At 11:20 PM 7/8/2009 -0400, Eric Smith wrote:
>P.J. Eby wrote:
>>>ISTM that the problem that it solves is uninstall in the absence of
>>>the original installer.
>>Or uninstall where the installer is "setup.py install", actually.
>
>I think we need to move away from "setup.py install". It's the 
>antithesis of static metadata.

Please note that that's entirely out of scope for the PEP at hand.

That being said, the rest of your proposal is strikingly similar to a 
proposal I previously floated on the distutils-sig for a concept 
called BUILDS.  The main difference is that I suggested that the spec 
should include a standard interface for running build operations that 
would produce the manifest (equivalent to your setup.info), and that 
distutils and setuptools should provide setup.py commands to generate 
said manifest, to allow for a seamless transition.

There was very little comment on the proposal, perhaps because it 
involves a lot of work that most people are sane enough not to sign 
up for.  ;-)

(Or more to the point, it's the sort of thing that never gets off the 
ground for design by consensus.  The best way to make something like 
your proposal to happen is to go off and build it, and get it to take 
over.  If it can't win substantial market share on its own merits, it 
probably doesn't deserve to be blessed as a standard.)


From pje at telecommunity.com  Thu Jul  9 05:40:53 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 08 Jul 2009 23:40:53 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A5563F5.3020707@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double> <4A5563F5.3020707@trueblade.com>
Message-ID: <20090709033751.638213A407B@sparrow.telecommunity.com>

At 11:28 PM 7/8/2009 -0400, Eric Smith wrote:
>>Eventually, I'd like PEP 376 to support system packagers too. So 
>>for example, if you did "apt-get install python-pyqt4", then 
>>running "pip install python-pyqt4" should return without installing 
>>anything .. as RECORD will be part of the .deb previously 
>>installed. As for generating the RECORD file, I vote for generating 
>>it during install time (w/ absolute paths).
>
>I think we should explicitly not support this. What if pip and 
>apt-get (or rpm, or others) install the same package in different 
>places because of system conventions (/usr vs. /usr/local vs. /opt, 
>say)? There's no way we're ever going to get this right, and it's 
>not worth complicating our lives over it.
>
>Seriously: Is there some real world use case I'm missing? Does any 
>existing install system support this?

The use case described above is supported right now by easy_install 
and pip; it doesn't require a RECORD file, though.  An .egg-info file 
or an .egg-info directory with a PKG-INFO is enough to prevent 
duplicate installation.

I don't know how pip handles file overwrites; easy_install OTOH never 
overwrites anything that's not in an .egg or a .pth.


From glyph at divmod.com  Thu Jul  9 05:46:41 2009
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 09 Jul 2009 03:46:41 -0000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A555C12.7000205@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<200907091051.02623.steve@pearwood.info>
	<4A555C12.7000205@trueblade.com>
Message-ID: <20090709034641.27019.110061093.divmod.xquotient.141@weber.divmod.com>

On 02:55 am, eric at trueblade.com wrote:
>I really don't get this use case of multiple installers trying to 
>install the same package. There's just no way that running "yum install 
>twisted" and "apt-get install twisted" and "pip install twisted" are 
>going to coexist with each other. The best they can do is say "a file 
>I'm trying to install already exists". Why try for anything else?

Here are some use-cases.

You do "apt-get install twisted", or, rather, Twisted comes preinstalled 
on your OS.  Later, you do "pip install 
py_awesome_network_thing_system", which depends on twisted.

You've configured pip to install files into ~/.local, though, and it's 
not sure if Twisted is installed or not.  So thanks to automatic 
dependency resolution, it downloads and installs another copy.  But 
that's not what you want: you want it to use the system-installed 
Twisted (and zope.interface, and pycrypto, and pytz, etc etc) and just 
install PANTS into your .local environment.  No files will conflict, but 
it would be helpful for the package installation tool to have an API to 
inspect both the system package installation and any other places where 
packages have been installed in order to make a decision about how it 
needs to satisfy its dependencies.

If you're not as interested as I am in per-user installation of 
packages, then consider the fact that Ubuntu stores Python packages in 
/usr/lib/pythonX.Y/dist-packages now.  Python package installers should 
be able to query that before putting things into site-packages. (which I 
*believe* is now reserved for non-OS-installed packages, but it's very 
hard to find documentation on the motivation behind debian's 
manipulations of distutils.  In any event, I digress.)

You're correct that worrying about 'yum install twisted' and 'apt-get 
install twisted' is way out of scope for any Python installation system; 
unfortunately, yum and apt are likely as not to have giant, opaque 
makefiles or shell scripts attached to these packages for obscure 
linux-y reasons that (hopefully) we don't need to care about, but would 
fight with each other if you tried to combine them, totally out of scope 
with distutils.

One hopes that regardless of whatever else they're doing, these packages 
*are* providing correct installation metadata.  (Unfortunately, I can 
bet you a nickel they will screw it up in the first few python packages 
that include it; see, for example, 
<https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/361865>.  This 
spec should try to gracefully address what to do if system packagers 
have done something unusual or wrong.)

So, the filesystem and the OS package database and the Python package 
"database" (such as it is) are definitely three distinct things, but 
they have some overlap, and there should be APIs for querying each.  If 
for no other reason so you know when Python packages shouldn't stomp on 
OS packages, and vice versa.

From glyph at divmod.com  Thu Jul  9 05:52:43 2009
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 09 Jul 2009 03:52:43 -0000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A5563F5.3020707@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T104939-364@post.gmane.org>
	<4A547E20.5030606@gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double> <4A5563F5.3020707@trueblade.com>
Message-ID: <20090709035243.27019.863612690.divmod.xquotient.150@weber.divmod.com>

On 03:28 am, eric at trueblade.com wrote:
>>Eventually, I'd like PEP 376 to support system packagers too. So for 
>>example, if you did "apt-get install python-pyqt4", then running "pip 
>>install python-pyqt4" should return without installing anything .. as 
>>RECORD will be part of the .deb previously installed. As for 
>>generating the RECORD file, I vote for generating it during install 
>>time (w/ absolute paths).
>
>I think we should explicitly not support this. What if pip and apt-get 
>(or rpm, or others) install the same package in different places 
>because of system conventions (/usr vs. /usr/local vs. /opt, say)? 
>There's no way we're ever going to get this right, and it's not worth 
>complicating our lives over it.

As I understand it, the point of the PEP is just to *provide the 
information* that the package installers need to get it right, not to 
actually implement the interaction with system package managers.
>Seriously: Is there some real world use case I'm missing? Does any 
>existing install system support this?

System package managers already provide .egg-info metadata, so this is 
hardly unprecedented.  The antecedent of your "this" is somewhat 
ambiguous, so I'm not sure if any package manager provides the exact 
feature that you have in mind.  If you're talking about different 
package managers installing stuff on the same system, you can look a bit 
farther afield: 'alien' allows you to install RPM and debian packages on 
the same system, with some reconciliation of dependencies and other 
metadata.

From p.f.moore at gmail.com  Thu Jul  9 09:18:18 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 9 Jul 2009 08:18:18 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A55621B.5070904@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
Message-ID: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com>

2009/7/9 Eric Smith <eric at trueblade.com>:
> P.J. Eby wrote:
>>>
>>> ISTM that the problem that it solves is uninstall in the absence of
>>> the original installer.
>>
>> Or uninstall where the installer is "setup.py install", actually.
>
> I think we need to move away from "setup.py install". It's the antithesis of
> static metadata. setup.py needs to go away.
>
> If we have a static metadata file called, say "setup.info", then I'd like to
> see:
> "bdist_rpm setup.info" produce a .rpm
> "bdist_msi setup.info" produce a .msi
> "bdist_deb setup.info" produce a .deb
> "bdist_egg setup.info" produce a .egg
> and so on.
>
> There might be a library (and I call dibs on the name "distlib" :) that
> provides support routines to parse setup.info, but there's no framework
> involved. And no need for a plugin system.

+1. Now who's going to design & write it?

> It's the conflation of "build a package" with "install a package" that
> distutils and setuptools has given us (via setup.py) that leads to us never
> making progress. Let's just focus on "describe a package" and let the
> packagers handle "build a package" and let the installers handle "install
> and uninstall a package". Then the packagers (and their associated
> installers) can evolve independently of each other.

There still needs to be some sort of package build system. I do *not*
want to see a situation where users can't easily build their own
packages. Not all systems have all Python packages packaged up by
"official" system packagers...

Paul

From ziade.tarek at gmail.com  Thu Jul  9 09:53:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 9 Jul 2009 09:53:25 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A551899.1080105@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
Message-ID: <94bdd2610907090053sf80135gd87b8e60453ccb5f@mail.gmail.com>

On Thu, Jul 9, 2009 at 12:07 AM, Eric Smith<eric at trueblade.com> wrote:
> Paul Moore wrote:
>>
>> 2009/7/8 P.J. Eby <pje at telecommunity.com>:
>>>
>>> If it were being driven by setuptools, I'd have just implemented it
>>> myself
>>> and presented it as a fait accompli. ?I can't speak to Tarek's motives,
>>> but
>>> I assume that, as stated in the PEP, the primary driver is supporting the
>>> distutils being able to uninstall things, and secondarily to allow other
>>> tools to be built on top of the API.
>>
>> My understanding is that all of the various distutils PEPs were driven
>> by the "packaging summit" ay PyCon. The struggle here seems to be to
>> find *anyone* from that summit who will now comment on the discussion
>> :-(
>
> I was there, and I've been commenting!

Yes, so Ian Bicking was, and Jim Fulton, and we worked from there on
Distutils-SIG.

To answer to Paul's question on my motivations for PEP 376, Phillip
gave the answers
I was going to give.

But as Eric says, that was one topic out of others.

>
> There might have been more discussion after the language summit and the one
> open space event I went to. But the focus as I recall was static metadata
> and version specification. When I originally brought up static metadata at
> the summit, I meant metadata describing the sources in the distribution, so
> that we can get rid of setup.py's. From that metadata, I want to be able to
> generate .debs, .rpms, .eggs, etc.
>
> But I think we've veered into metadata that describes what has been
> installed. I don't think that's so useful. As I've said, this is private to
> the installers. If 2 installers want to communicate with each other about
> what they've installed, then they can agree on that data. I just don't find
> it generally useful for all installers, and therefore not useful for
> distutils.
>
> I'd like to get back to the metadata that describes the source files. That's
> where the real value lies, in my opinion. I'll try and work on a post to
> distutils-sig explaining my thinking.

Yes please do so, and have a look at Tres and Matthias notes from the summit

http://wiki.python.org/moin/Distutils/StaticMetadata

but that's another topic, that will deserve a PEP on its own

Tarek

From ziade.tarek at gmail.com  Thu Jul  9 09:57:53 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 9 Jul 2009 09:57:53 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090709010447.2A86D3A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
Message-ID: <94bdd2610907090057o3fb6dd1ch55df608de23435de@mail.gmail.com>

On Thu, Jul 9, 2009 at 3:07 AM, P.J. Eby<pje at telecommunity.com> wrote:
> At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote:
>>
>> Eric Smith writes:
>>
>> ?> But I think we've veered into metadata that describes what has been
>> ?> installed. I don't think that's so useful. As I've said, this is
>> private
>> ?> to the installers. If 2 installers want to communicate with each other
>> ?> about what they've installed, then they can agree on that data. I just
>> ?> don't find it generally useful for all installers, and therefore not
>> ?> useful for distutils.
>>
>> ISTM that the problem that it solves is uninstall in the absence of
>> the original installer.
>
> Or uninstall where the installer is "setup.py install", actually.
>

Right, and for instance, Fedora packagers are using it to re-package eggs

http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools

with the --root option, so they will get a RECORD file and will be
able to set a value for
the INSTALLER one,

From ziade.tarek at gmail.com  Thu Jul  9 10:15:13 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 9 Jul 2009 10:15:13 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
	<79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com>
Message-ID: <94bdd2610907090115h2d4340fajb1309e9c1a9abf8b@mail.gmail.com>

On Thu, Jul 9, 2009 at 9:18 AM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/9 Eric Smith <eric at trueblade.com>:
>> P.J. Eby wrote:
>>>>
>>>> ISTM that the problem that it solves is uninstall in the absence of
>>>> the original installer.
>>>
>>> Or uninstall where the installer is "setup.py install", actually.
>>
>> I think we need to move away from "setup.py install". It's the antithesis of
>> static metadata. setup.py needs to go away.
>>
>> If we have a static metadata file called, say "setup.info", then I'd like to
>> see:
>> "bdist_rpm setup.info" produce a .rpm
>> "bdist_msi setup.info" produce a .msi
>> "bdist_deb setup.info" produce a .deb
>> "bdist_egg setup.info" produce a .egg
>> and so on.
>>
>> There might be a library (and I call dibs on the name "distlib" :) that
>> provides support routines to parse setup.info, but there's no framework
>> involved. And no need for a plugin system.
>
> +1. Now who's going to design & write it?

Some work has been started on that at the summit too on that, under
the "static metadata" name.
(see the wiki)

Can we move that discussion to distutils-SIG and focus on PEP 376 in
this thread ?
I am scared to see the discussion going in too many directions,

Regards
Tarek

From ziade.tarek at gmail.com  Thu Jul  9 10:22:19 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 9 Jul 2009 10:22:19 +0200
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <op.uwr1phe9brrvlq@double>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double>
Message-ID: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>

On Thu, Jul 9, 2009 at 3:42 AM, Sridhar
Ratnakumar<SridharR at activestate.com> wrote:
> Other than easy_install/pip, there is also PyPM which is being developed at
> ActiveState. PyPM is the Python package manager much like what ppm is for
> ActivePerl.
>

Great ! besides the RECORD file, anything remark on the PEP from a
PyPM point of view ?
Once Paul has finished to work on the PEP 302 part of the prototype,
maybe we could try it out
on PyPM to see how it fits ?

> Eventually, I'd like PEP 376 to support system packagers too. So for
> example, if you did "apt-get install python-pyqt4", then running "pip
> install python-pyqt4" should return without installing anything .. as RECORD
> will be part of the .deb previously installed. As for generating the RECORD
> file, I vote for generating it during install time (w/ absolute paths).

That should be possible for all "python setup.py install"-based systems
e.g. stating that if python-pyqt4 is already installed and if the
INSTALLER file contains
another name than "pip", pip should not try to do anything with it.

From p.f.moore at gmail.com  Thu Jul  9 10:32:52 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 9 Jul 2009 09:32:52 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double>
	<94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>
Message-ID: <79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com>

2009/7/9 Tarek Ziad? <ziade.tarek at gmail.com>:
> On Thu, Jul 9, 2009 at 3:42 AM, Sridhar
> Ratnakumar<SridharR at activestate.com> wrote:
>> Other than easy_install/pip, there is also PyPM which is being developed at
>> ActiveState. PyPM is the Python package manager much like what ppm is for
>> ActivePerl.
>>
>
> Great ! besides the RECORD file, anything remark on the PEP from a
> PyPM point of view ?
> Once Paul has finished to work on the PEP 302 part of the prototype,
> maybe we could try it out
> on PyPM to see how it fits ?

Note: the "pkgutil2" implementation I pushed to Tarek's bitbucket
repository a day or so ago includes PEP 302 support (using a few new
finder methods, as proposed). It's ready to go now. It's as compliant
with the PEP as possible given the outstanding questions (I've
implemented some things on the basis of what I thought was consensus
here, even though the PEP has not changed yet to reflect that - using
"metadata" rather than "egginfo" in method names, and removing the
local= parameter from some of the methods, being the 2 I can recall
immediately).

I'd like to add a test case using a non-standard PEP 302 importer, but
that's a small detail.

There are bound to be holes in the implementation, especially around
the areas of the open questions that started this thread. I'd
appreciate feedback, but I'm not expecting to make code changes until
the PEP is changed - it's too much like trying to hit a moving target
at the moment!

Paul.

PS I've added distutils-sig back into the cc list. Please keep "PEP
376" in the subject of any subsequent threads on that list if you want
my input, as I don't follow all the traffic on there.

From ncoghlan at gmail.com  Thu Jul  9 10:37:47 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Jul 2009 18:37:47 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A555C12.7000205@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>	<200907091051.02623.steve@pearwood.info>
	<4A555C12.7000205@trueblade.com>
Message-ID: <4A55AC5B.5020405@gmail.com>

Eric Smith wrote:
> I really don't get this use case of multiple installers trying to
> install the same package. There's just no way that running "yum install
> twisted" and "apt-get install twisted" and "pip install twisted" are
> going to coexist with each other. The best they can do is say "a file
> I'm trying to install already exists". Why try for anything else?

If that use case doesn't click for you, just focus on the
distutils.uninstall use case. At the moment, there is no automatic
method for uninstalling a distutils installed package. This PEP is
mainly about fixing that situation.

However, if we're going to go to all the effort of having an index of
distutils-installed packages and the metadata needed to uninstall them,
it makes sense to have an official API for querying that datastore. The
fact that that API then provides support for "Did someone else install
this already?" is really a sideshow.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Jul  9 10:50:12 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Jul 2009 18:50:12 +1000
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <20090709020938.5C0A83A407B@sparrow.telecommunity.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
Message-ID: <4A55AF44.4090508@gmail.com>

P.J. Eby wrote:
>>  Also,
>> why should the RECORD file be generated at all by bdist* commands?
> 
> bdist commands that use "install --root" will get it as a side effect,
> so they don't have to do anything special.  They needn't even generate
> the PREFIXES, if PREFIXES is also generated by "install --root", and
> omits the root from the prefixes written.  In other words, most bdist
> tools wouldn't change.  bdist_wininst and bdist_msi might need to
> regenerate PREFIXES at actual installation time, but AFAIK those are the
> only ones.  (And the spec would allow for the possibility that a
> PREFIXES file can be incorrect.)

I haven't seen any suggestions for handling non-relative paths that are
cleaner than this. The distutils based list that Tarek posted also gives
a good "starter set" of supported entries in the PREFIXES file:

PURELIB=<path>
PLATLIB=<path>
HEADERS=<path>
SCRIPTS=<path>
DATA=<path>

The question then would be whether relative paths should be handled
directly in the RECORD file (as in the current PEP) or by writing a "."
into PREFIXES to indicate relative paths.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From cournape at gmail.com  Thu Jul  9 11:26:19 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 9 Jul 2009 18:26:19 +0900
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
	<79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com>
Message-ID: <5b8d13220907090226r466f28d3y9af85848fe4c4f7a@mail.gmail.com>

On Thu, Jul 9, 2009 at 4:18 PM, Paul Moore<p.f.moore at gmail.com> wrote:

>>
>> There might be a library (and I call dibs on the name "distlib" :) that
>> provides support routines to parse setup.info, but there's no framework
>> involved. And no need for a plugin system.
>
> +1. Now who's going to design & write it?

I started a new thread on distutils-sig ("setup.py needs to go away")
to avoid jeopardizing this thread. I added the context as well as my
own suggestions for such a design.

David

From p.f.moore at gmail.com  Thu Jul  9 15:45:22 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 9 Jul 2009 14:45:22 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double>
	<94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>
	<79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com>
Message-ID: <79990c6b0907090645q7468f0dj4b32e89f70306022@mail.gmail.com>

2009/7/9 Paul Moore <p.f.moore at gmail.com>:
> I'd like to add a test case using a non-standard PEP 302 importer, but
> that's a small detail.

I've just pushed a PEP 302 importer test case.

The level of boilerplate needed is a bit painful, but it's fine as a
prototype. When we get consensus on some of the remaining details,
I'll tidy up the implementation and streamline the importer
requirements at that time.

Paul.

From eric at trueblade.com  Thu Jul  9 15:53:56 2009
From: eric at trueblade.com (Eric Smith)
Date: Thu, 09 Jul 2009 09:53:56 -0400
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <20090709033551.AA35D3A407B@sparrow.telecommunity.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>	<loom.20090708T121713-308@post.gmane.org>	<4A54984D.4020305@trueblade.com>	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>	<4A54AF4F.9080800@trueblade.com>	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>	<20090708151647.9E0703A407B@sparrow.telecommunity.com>	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>	<4A55621B.5070904@trueblade.com>
	<20090709033551.AA35D3A407B@sparrow.telecommunity.com>
Message-ID: <4A55F674.6090200@trueblade.com>

P.J. Eby wrote:
> At 11:20 PM 7/8/2009 -0400, Eric Smith wrote:
>> P.J. Eby wrote:
>>>> ISTM that the problem that it solves is uninstall in the absence of
>>>> the original installer.
>>> Or uninstall where the installer is "setup.py install", actually.
>>
>> I think we need to move away from "setup.py install". It's the 
>> antithesis of static metadata.
> 
> Please note that that's entirely out of scope for the PEP at hand.

Yes. I'm just trying to point out that the information in the PEP is 
applicable only to the set of installers that want to work together in 
some integrated way. It doesn't apply to all installers, and I think 
that's the bigger problem.

But I grant you that the rest of my mail is veering off-topic, and 
there's a more appropriate thread now on distutils-sig.

Eric.


From p.f.moore at gmail.com  Thu Jul  9 16:22:45 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 9 Jul 2009 15:22:45 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A55F674.6090200@trueblade.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
	<20090709033551.AA35D3A407B@sparrow.telecommunity.com>
	<4A55F674.6090200@trueblade.com>
Message-ID: <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com>

2009/7/9 Eric Smith <eric at trueblade.com>:
> Yes. I'm just trying to point out that the information in the PEP is
> applicable only to the set of installers that want to work together in some
> integrated way. It doesn't apply to all installers, and I think that's the
> bigger problem.

Interoperability standards are only as useful as the number of
applications that conform to them. The set of installers that will
*not* provide PEP 376 compatible metadata is an important measure of
the usefulness of the PEP. Some metadata (the INSTALLER file and the
proposed PREFIXES file) must be created by the installer for the PEP
376 APIs to work.

Simple cases (relocatable packages with all package data maintained
inside the package directory) will work fine even without installer
co-operation, as long as the installer copies the whole structure
unchanged. But:

- some cases are not simple, and it's not clear to me how useful
"nearly always accurate" data will be
- I understand that some packagers deliberately pick apart Python
distributions to conform to OS filesystem layout standards

Paul.

From tseaver at palladion.com  Thu Jul  9 20:46:18 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 09 Jul 2009 14:46:18 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <4A55AF44.4090508@gmail.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>	<op.uwr13zlrbrrvlq@double>	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com>
Message-ID: <h35dtr$sa9$2@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nick Coghlan wrote:
> P.J. Eby wrote:
>>>  Also,
>>> why should the RECORD file be generated at all by bdist* commands?
>> bdist commands that use "install --root" will get it as a side effect,
>> so they don't have to do anything special.  They needn't even generate
>> the PREFIXES, if PREFIXES is also generated by "install --root", and
>> omits the root from the prefixes written.  In other words, most bdist
>> tools wouldn't change.  bdist_wininst and bdist_msi might need to
>> regenerate PREFIXES at actual installation time, but AFAIK those are the
>> only ones.  (And the spec would allow for the possibility that a
>> PREFIXES file can be incorrect.)
> 
> I haven't seen any suggestions for handling non-relative paths that are
> cleaner than this. The distutils based list that Tarek posted also gives
> a good "starter set" of supported entries in the PREFIXES file:
> 
> PURELIB=<path>
> PLATLIB=<path>
> HEADERS=<path>
> SCRIPTS=<path>
> DATA=<path>
> 
> The question then would be whether relative paths should be handled
> directly in the RECORD file (as in the current PEP) or by writing a "."
> into PREFIXES to indicate relative paths.

Per discussion with Matthias Klose, it would be useful to split out the
following prefixes for "special" kinds of data:

- - docs
- - i18n / locales


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKVjr6+gerLs4ltQ4RAqwTAKCoohHwkbON3Gg6gmJ4AmZUcPSRdQCgyeZi
nBJVZQziPYnWN0cI3B2i4A0=
=bSHL
-----END PGP SIGNATURE-----


From pje at telecommunity.com  Thu Jul  9 21:09:33 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 09 Jul 2009 15:09:33 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <h35dtr$sa9$2@ger.gmane.org>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
Message-ID: <20090709190631.CE40E3A407B@sparrow.telecommunity.com>

At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Nick Coghlan wrote:
> > P.J. Eby wrote:
> >>>  Also,
> >>> why should the RECORD file be generated at all by bdist* commands?
> >> bdist commands that use "install --root" will get it as a side effect,
> >> so they don't have to do anything special.  They needn't even generate
> >> the PREFIXES, if PREFIXES is also generated by "install --root", and
> >> omits the root from the prefixes written.  In other words, most bdist
> >> tools wouldn't change.  bdist_wininst and bdist_msi might need to
> >> regenerate PREFIXES at actual installation time, but AFAIK those are the
> >> only ones.  (And the spec would allow for the possibility that a
> >> PREFIXES file can be incorrect.)
> >
> > I haven't seen any suggestions for handling non-relative paths that are
> > cleaner than this. The distutils based list that Tarek posted also gives
> > a good "starter set" of supported entries in the PREFIXES file:
> >
> > PURELIB=<path>
> > PLATLIB=<path>
> > HEADERS=<path>
> > SCRIPTS=<path>
> > DATA=<path>
> >
> > The question then would be whether relative paths should be handled
> > directly in the RECORD file (as in the current PEP) or by writing a "."
> > into PREFIXES to indicate relative paths.
>
>Per discussion with Matthias Klose, it would be useful to split out the
>following prefixes for "special" kinds of data:
>
>- - docs
>- - i18n / locales

Unfortunately, the distutils don't currently have a way to specify 
these.  I think we'll need to use only the ones that are distutils 
install targets at first, but of course the format is extensible.

OTOH, one supposed that at least the 'data_files' setup argument 
could be enhanced to allow a data type flag, that would then be used 
as a prefix name for those files, with a fallback to the DATA prefix 
if there isn't one specified.  But that's something that will require 
tagging support in the install_data command.  As it is, refactoring 
the distutils to track prefixes for the RECORD file is probably going 
to be...  challenging.  ;-)


From ncoghlan at gmail.com  Thu Jul  9 21:53:13 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jul 2009 05:53:13 +1000
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <h35dtr$sa9$2@ger.gmane.org>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>	<op.uwr13zlrbrrvlq@double>	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>	<4A55AF44.4090508@gmail.com>
	<h35dtr$sa9$2@ger.gmane.org>
Message-ID: <4A564AA9.3000305@gmail.com>

Tres Seaver wrote:
> Nick Coghlan wrote:
>> The question then would be whether relative paths should be handled
>> directly in the RECORD file (as in the current PEP) or by writing a "."
>> into PREFIXES to indicate relative paths.
> 
> Per discussion with Matthias Klose, it would be useful to split out the
> following prefixes for "special" kinds of data:
> 
> - docs
> - i18n / locales

There are probably all sorts of other useful distinctions which could be
made. However, that would be a topic for a different PEP - as Tarek
pointed out, improving distutils is such a huge can of worms that trying
to tackle all the issues in a single PEP would most likely just result
in developer burnout rather than improvement.

The basic PREFIXES discussion is at least somewhat relevant to PEP 376,
since the distutils uninstaller needs *some* way for the bdist_*
installers to tell it where the non-relative files ended up. Since we
need a mechanism for it, it might as well be one that should help with
the system package builders needs in the future.

Actually adding new categories, while I agree it would be useful, isn't
something that is needed to support uninstallation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Jul  9 22:00:19 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jul 2009 06:00:19 +1000
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>	<20090708183001.0A8A53A407B@sparrow.telecommunity.com>	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>	<4A551899.1080105@trueblade.com>	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>	<4A55621B.5070904@trueblade.com>	<20090709033551.AA35D3A407B@sparrow.telecommunity.com>	<4A55F674.6090200@trueblade.com>
	<79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com>
Message-ID: <4A564C53.4080102@gmail.com>

Paul Moore wrote:
> - some cases are not simple, and it's not clear to me how useful
> "nearly always accurate" data will be

Since the accuracy can always be checked against the filesystem, I think
the metadata is useful even if not 100% reliable. Applications that need
the extra assurance (such as the distutils uninstaller itself) can check
the filesytem and complain when it differs from what it is claimed in
the package metadata.

One thing that did occur to me based on this - do we want the format to
support designation of files (such as config files) that *shouldn't* be
uninstalled along with everything else? Or are we happy with not
mentioning the file in RECORD at all as the means of supporting that use
case?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fdrake at acm.org  Thu Jul  9 22:00:41 2009
From: fdrake at acm.org (Fred Drake)
Date: Thu, 09 Jul 2009 16:00:41 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <20090709190631.CE40E3A407B@sparrow.telecommunity.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
Message-ID: <4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org>

At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
> - - docs
> - - i18n / locales


On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote:
> Unfortunately, the distutils don't currently have a way to specify  
> these.


There actually is a way to identify documentation files via setup.cfg,  
but I don't know if the RPM packagers (or others) use that information.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From david.lyon at preisshare.net  Fri Jul 10 01:29:32 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Thu, 09 Jul 2009 19:29:32 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <4A564AA9.3000305@gmail.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>	<op.uwr13zlrbrrvlq@double>	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>	<4A55AF44.4090508@gmail.com>
	<h35dtr$sa9$2@ger.gmane.org> <4A564AA9.3000305@gmail.com>
Message-ID: <52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net>

On Fri, 10 Jul 2009 05:53:13 +1000, Nick Coghlan <ncoghlan at gmail.com>
wrote:
> There are probably all sorts of other useful distinctions which could be
> made. However, that would be a topic for a different PEP - as Tarek
> pointed out, improving distutils is such a huge can of worms that trying
> to tackle all the issues in a single PEP would most likely just result
> in developer burnout rather than improvement.

All it requires is a few word changes to the documentation to tell
programmers to put your documentation in "docs"...

If that is going to burn out the existing developers, it might
suggest that it is time to get some fresh developers onboard....

Thats not even a programmatic fix...

Regards

David



From david.lyon at preisshare.net  Fri Jul 10 01:22:57 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Thu, 09 Jul 2009 19:22:57 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
	<4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org>
Message-ID: <d0243786a3d51fd95469b1fc4664e768@preisshare.net>

On Thu, 09 Jul 2009 16:00:41 -0400, Fred Drake <fdrake at acm.org> wrote:
> At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
>> - - docs
>> - - i18n / locales
> 
> 
> On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote:
>> Unfortunately, the distutils don't currently have a way to specify  
>> these.
> 
> 
> There actually is a way to identify documentation files via setup.cfg,  
> but I don't know if the RPM packagers (or others) use that information.

Hi,

Perphaps the RPM packagers dont use it but the Python Package Manager
GUI (http://sourceforge.net/projects/pythonpkgmgr) does.

With that, if there is any package documentation in a docs directory
the user can click on it and display it.

Regards

David

From ncoghlan at gmail.com  Fri Jul 10 03:48:48 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jul 2009 11:48:48 +1000
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>	<op.uwr13zlrbrrvlq@double>	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>	<4A55AF44.4090508@gmail.com>
	<h35dtr$sa9$2@ger.gmane.org> <4A564AA9.3000305@gmail.com>
	<52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net>
Message-ID: <4A569E00.4080509@gmail.com>

David Lyon wrote:
> All it requires is a few word changes to the documentation to tell
> programmers to put your documentation in "docs"...

There is no current distutils setting to indicate that a particular data
file is actually documentation rather than something else - hence such a
thing *isn't* just a matter of updating the documentation. If you're
actually saying that "hey, that metadata shouldn't be in the code, it
should just be a file naming convention" then that *in itself* is a
design decision that needs to be discussed and agreed via a PEP (a
'standard' that nobody follows or has even agreed to follow is a useless
standard).

> If that is going to burn out the existing developers, it might
> suggest that it is time to get some fresh developers onboard....

No, tackling *all* of the distutils issues (supporting uninstallation,
moving from setup.py to static metadata for simple cases, providing more
fine-grained *programmatic* file categorisation, anything else I've
missed) at once would probably burn out anyone that tried to do it.

The coding in all this really isn't that hard. It's the gathering of
feedback, assessment of use cases and the constant balancing act between
retaining enough flexibility to allow for future extensions without a
complete rebuild of the design without making the current design so
complex as to be unusable.

If it was just a matter of throwing some code in the standard library or
some words in the documentation and declaring "this is how it shall be
from now on" it would have been done years ago :P

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From tjreedy at udel.edu  Fri Jul 10 05:34:58 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 09 Jul 2009 23:34:58 -0400
Subject: [Python-Dev] Search error on tracker?
Message-ID: <h36ct2$al1$1@ger.gmane.org>

Before posting http://bugs.python.org/issue6453
I searched the tracker for 'bool TypeError' to avoid making a duplicate 
report. A few irrelevant hits were return but not the important one
http://bugs.python.org/issue6428
which was the one I needed to find.

Going back and searching titles for 'TypeError', cut and pasted from the 
title of my report, I get 6 hits but neither of the two above.

Is there some sort of delay in updating the search indexes?

Terry Jan Reedy


From martin at v.loewis.de  Fri Jul 10 06:44:11 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Jul 2009 06:44:11 +0200
Subject: [Python-Dev] Search error on tracker?
In-Reply-To: <h36ct2$al1$1@ger.gmane.org>
References: <h36ct2$al1$1@ger.gmane.org>
Message-ID: <4A56C71B.5060006@v.loewis.de>

Terry Reedy wrote:
> Before posting http://bugs.python.org/issue6453
> I searched the tracker for 'bool TypeError' to avoid making a duplicate
> report. A few irrelevant hits were return but not the important one
> http://bugs.python.org/issue6428
> which was the one I needed to find.

When I search for 'bool TypeError', #6428 shows up. Are you sure you had
been searching for closed reports as well?

> Going back and searching titles for 'TypeError', cut and pasted from the
> title of my report, I get 6 hits but neither of the two above.

Again, did you search for closed reports also?

> Is there some sort of delay in updating the search indexes?

No, it's immediate.

Regards,
Martin

From p.f.moore at gmail.com  Fri Jul 10 10:56:36 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 10 Jul 2009 09:56:36 +0100
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <4A564C53.4080102@gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
	<20090709033551.AA35D3A407B@sparrow.telecommunity.com>
	<4A55F674.6090200@trueblade.com>
	<79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com>
	<4A564C53.4080102@gmail.com>
Message-ID: <79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com>

2009/7/9 Nick Coghlan <ncoghlan at gmail.com>:
> Paul Moore wrote:
>> - some cases are not simple, and it's not clear to me how useful
>> "nearly always accurate" data will be
>
> Since the accuracy can always be checked against the filesystem, I think
> the metadata is useful even if not 100% reliable. Applications that need
> the extra assurance (such as the distutils uninstaller itself) can check
> the filesytem and complain when it differs from what it is claimed in
> the package metadata.

The inaccuracy I was thinking of was that if distribution X's RECORD
doesn't mention file Y, the implication is that X doesn't "own" Y.
There's no way of checking this against the filesystem - RECORD is the
only place that says that X owns Y.

I was thinking in terms of "does anyone else mind if I
delete/overwrite this file?" queries, not uninstallation.

> One thing that did occur to me based on this - do we want the format to
> support designation of files (such as config files) that *shouldn't* be
> uninstalled along with everything else? Or are we happy with not
> mentioning the file in RECORD at all as the means of supporting that use
> case?

No. In my view, the uninstall feature is a relatively minor issue -
people who want uninstall facilities generally use system packages (or
complain that easy_install doesn't support uninstallation :-)). They
aren't using setup.py install. (Of course, my experience is minimal in
this regard, so maybe I'm wrong).

Remember that the PEP says "Distutils will provide a **very basic**
uninstall function" (my emphasis).

Unless I'm misunderstanding, no-one is intending to use the new
distutils uninstall feature in their own package manager
(bdist_wininst, RPM, apt, ...)

Paul.

From ziade.tarek at gmail.com  Fri Jul 10 15:31:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 10 Jul 2009 15:31:10 +0200
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <20090709190631.CE40E3A407B@sparrow.telecommunity.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
Message-ID: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com>

On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby<pje at telecommunity.com> wrote:
> At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Nick Coghlan wrote:
>> > P.J. Eby wrote:
>> >>> ?Also,
>> >>> why should the RECORD file be generated at all by bdist* commands?
>> >> bdist commands that use "install --root" will get it as a side effect,
>> >> so they don't have to do anything special. ?They needn't even generate
>> >> the PREFIXES, if PREFIXES is also generated by "install --root", and
>> >> omits the root from the prefixes written. ?In other words, most bdist
>> >> tools wouldn't change. ?bdist_wininst and bdist_msi might need to
>> >> regenerate PREFIXES at actual installation time, but AFAIK those are
>> >> the
>> >> only ones. ?(And the spec would allow for the possibility that a
>> >> PREFIXES file can be incorrect.)
>> >
>> > I haven't seen any suggestions for handling non-relative paths that are
>> > cleaner than this. The distutils based list that Tarek posted also gives
>> > a good "starter set" of supported entries in the PREFIXES file:
>> >
>> > PURELIB=<path>
>> > PLATLIB=<path>
>> > HEADERS=<path>
>> > SCRIPTS=<path>
>> > DATA=<path>
>> >
>> > The question then would be whether relative paths should be handled
>> > directly in the RECORD file (as in the current PEP) or by writing a "."
>> > into PREFIXES to indicate relative paths.

I like the idea of a separated PREFIXES file, then using variables in
the RECORD file.
While this doesn't change anything code-wise, we can keep that way two
csv-browsable files.

>>
>> Per discussion with Matthias Klose, it would be useful to split out the
>> following prefixes for "special" kinds of data:
>>
>> - - docs
>> - - i18n / locales
>
> Unfortunately, the distutils don't currently have a way to specify these. ?I
> think we'll need to use only the ones that are distutils install targets at
> first, but of course the format is extensible.
>
> OTOH, one supposed that at least the 'data_files' setup argument could be
> enhanced to allow a data type flag, that would then be used as a prefix name
> for those files, with a fallback to the DATA prefix if there isn't one
> specified. ?But that's something that will require tagging support in the
> install_data command. ?As it is, refactoring the distutils to track prefixes
> for the RECORD file is probably going to be... ?challenging. ?;-)

yes, if we want to provide extensibility, a new prefix that is used by
a developer in its distribution
will have to be provided for all platforms, so distutils can build the
proper RECORD file
by querying the path that corresponds to the platform it runs on.

In other words, if a data type flag is provided in data_files, the
list of absolute paths
per key in distutils.command.install.INSTALL_SCHEMES  should be
provided with it.

This could be done with a new 'extra_prefixes' argument for example,
provided with the 'data_files',
that would let distutils extend INSTALL_SCHEMES on the fly

setup(
    ...
    data_files={'i18n': ['en.po', 'fr.po']},
    extra_prefixes={'i18n': {'unix_prefix': '/somewhere',
                                         'mac': '/Application/right/here'}
 ...
  )

But then data_files could maybe be left alone (that's specific to
'data' after all)
and a new option created.

In any case that would be a great solution to avoid 'wild absolute
path' and be sure
that the developer thinks about where each fil of his distrbution
goes, depending
on the plaftorm.

While debian packagers and al might not use it to re-package python
distributions in debian
packages, I am sure they will like it very much because it helps
identifiying what the developer
wanted to do with these "data" files.

From asmodai at in-nomine.org  Fri Jul 10 15:58:28 2009
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Fri, 10 Jul 2009 15:58:28 +0200
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
	<94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com>
Message-ID: <20090710135828.GS65921@nexus.in-nomine.org>

-On [20090710 15:32], Tarek Ziad? (ziade.tarek at gmail.com) wrote:
>setup(
>    ...
>    data_files={'i18n': ['en.po', 'fr.po']},
>    extra_prefixes={'i18n': {'unix_prefix': '/somewhere',
>                                         'mac': '/Application/right/here'}
> ...
>  )

As an aside:

Where i18n above would sooner be called 'locale' rather than i18n, since
i18n is only providing software functionality for supporting localization.
So in that sense L10n would be better, but locale is quite a defacto
standard in many projects.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
In every colour there's the Light...

From status at bugs.python.org  Fri Jul 10 18:07:30 2009
From: status at bugs.python.org (Python tracker)
Date: Fri, 10 Jul 2009 18:07:30 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20090710160730.01D44785FE@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (07/03/09 - 07/10/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2245 open (+32) / 16035 closed (+17) / 18280 total (+49)

Open issues with patches:   894

Average duration of open issues: 660 days.
Median duration of open issues: 414 days.

Open Issues Breakdown
   open  2215 (+32)
pending    30 ( +0)

Issues Created Or Reopened (52)
_______________________________

multiprocessing: BaseManager.from_address documented but doesn't 07/09/09
       http://bugs.python.org/issue3518    reopened jnoller                       
                                                                               

logging.basicConfig(level='DEBUG', ... and setLevel("DEBUG") res 07/06/09
       http://bugs.python.org/issue6314    reopened r.david.murray                
       patch                                                                   

distutils compiler switch ignored                                07/08/09
       http://bugs.python.org/issue6377    reopened tarek                         
       patch                                                                   

Dictionaries should support __add__                              07/04/09
CLOSED http://bugs.python.org/issue6410    reopened ezio.melotti                  
                                                                               

3.1 not functional                                               07/03/09
CLOSED http://bugs.python.org/issue6411    created  element.effect                
                                                                               

Titlecase as defined in Unicode Case Mappings not followed       07/03/09
       http://bugs.python.org/issue6412    created  christoph                     
       patch                                                                   

incorrect log level in distutlis.dist for announce               07/04/09
CLOSED http://bugs.python.org/issue6413    created  tarek                         
                                                                               

struct module : processor endianness descriptions misleading     07/04/09
       http://bugs.python.org/issue6414    created  kmag                          
       easy                                                                    

warnings.warn segfaults on bad formatted string                  07/04/09
       http://bugs.python.org/issue6415    created  wiesenth                      
       patch                                                                   

Failed to compile selectmodule.c on windows (trunk)              07/04/09
CLOSED http://bugs.python.org/issue6416    created  ocean-city                    
       easy                                                                    

multiprocessing Process examples: print and getppid              07/04/09
       http://bugs.python.org/issue6417    created  mnewman                       
                                                                               

unittest.TestProgram change breaks nose                          07/04/09
       http://bugs.python.org/issue6418    created  jpellerin                     
       patch                                                                   

Broken test_kqueue.py on OpenBSD                                 07/04/09
       http://bugs.python.org/issue6419    created  henry.precheur                
                                                                               

Fix warning in nismodule.c on OpenBSD                            07/04/09
CLOSED http://bugs.python.org/issue6420    created  henry.precheur                
       patch                                                                   

errors in docs re module initialization vs self arg to functions 07/04/09
       http://bugs.python.org/issue6421    created  aleax                         
       easy                                                                    

timeit called from within Python should allow autoranging        07/05/09
       http://bugs.python.org/issue6422    created  scott_daniels                 
       patch                                                                   

The cgi docs should advertize using "in" instead of "has_key"    07/05/09
       http://bugs.python.org/issue6423    created  cito                          
       patch                                                                   

Idle3 crashing when using completion                             07/05/09
CLOSED http://bugs.python.org/issue6424    created  tcourbon                      
                                                                               

imaplib.IMAP4.fetch() is missing documentation for message_set p 07/06/09
CLOSED http://bugs.python.org/issue6425    created  Sjoerder                      
                                                                               

imaplib.IMAP4 "command illegal in this state" is unhelpful error 07/06/09
       http://bugs.python.org/issue6426    created  Sjoerder                      
       easy                                                                    

Rename float*.[ch] to double.[ch]                                07/06/09
CLOSED http://bugs.python.org/issue6427    created  jackdied                      
                                                                               

TypeError: __bool__ should return bool or int, returned int      07/06/09
CLOSED http://bugs.python.org/issue6428    created  SilentGhost                   
       patch                                                                   

2to3: fix_future conflicts with fix_print                        07/06/09
       http://bugs.python.org/issue6429    created  alexandre.vassalotti          
                                                                               

array.array falsely advertises support for 'w' in documentation  07/06/09
       http://bugs.python.org/issue6430    created  alexandre.vassalotti          
                                                                               

Fraction fails equality test with a user-defined type            07/07/09
       http://bugs.python.org/issue6431    created  casevh                        
       patch                                                                   

ImageTk.PhotoImage                                               07/07/09
CLOSED http://bugs.python.org/issue6432    created  jpmieville                    
                                                                               

multiprocessing: pool.map hangs on empty list                    07/07/09
       http://bugs.python.org/issue6433    created  ede                           
       patch                                                                   

buffer overflow in Zipfile when wrinting more than 2gig file     07/07/09
       http://bugs.python.org/issue6434    created  segfault42                    
                                                                               

logging: cache the traceback text on formatter, instead of recor 07/08/09
CLOSED http://bugs.python.org/issue6435    created  srid                          
                                                                               

trace module doesn't seem to produce .cover files for Py3 (but d 07/08/09
       http://bugs.python.org/issue6436    created  mark                          
                                                                               

Personal invitation from Manas                                   07/08/09
CLOSED http://bugs.python.org/issue6437    created  gravitywarrior1               
                                                                               

cygwincompiler regular expressions broken                        07/08/09
       http://bugs.python.org/issue6438    created  tarek                         
                                                                               

Demo/embed/demo.c use of PySys_SetArgv() is invalid              07/08/09
       http://bugs.python.org/issue6439    created  ideasman42                    
                                                                               

2to3: convert deprecated string.maketrans to str.maketrans       07/08/09
CLOSED http://bugs.python.org/issue6440    created  ede                           
                                                                               

Tkinter cannot find *64 bit* Tcl/Tk on Mac OS X                  07/08/09
       http://bugs.python.org/issue6441    created  keflavich                     
                                                                               

Backwards __contains__/has_key in xml/sax/xmlreader.py           07/08/09
       http://bugs.python.org/issue6442    created  MostAwesomeDude               
       patch                                                                   

_winreg.QueryValueEx confused on double null terminated strings  07/09/09
       http://bugs.python.org/issue6443    created  AndrewZiem                    
                                                                               

multiline exception logging via syslog handler                   07/09/09
       http://bugs.python.org/issue6444    created  LwarX                         
                                                                               

Add check parameter to subprocess.Popen.communicate              07/09/09
       http://bugs.python.org/issue6445    created  OG7                           
                                                                               

import_spam() in extending python can fail without setting the e 07/09/09
       http://bugs.python.org/issue6446    created  jak                           
                                                                               

PATCH: typo (English) in subprocess module                       07/09/09
       http://bugs.python.org/issue6447    created  Yinon                         
       patch                                                                   

imp.find_module() -- be explicity that 'path' must be a list     07/09/09
       http://bugs.python.org/issue6448    created  jordanb                       
                                                                               

Improve/update python.org/dev/                                   07/09/09
       http://bugs.python.org/issue6449    created  ezio.melotti                  
       easy                                                                    

normpath() sometimes maps unicode to str                         07/09/09
       http://bugs.python.org/issue6450    created  sandberg                      
                                                                               

2to3 adds spare brackets to print                                07/09/09
CLOSED http://bugs.python.org/issue6451    created  bazooka                       
                                                                               

urllib2.Request() will not work with long authorization headers  07/09/09
       http://bugs.python.org/issue6452    created  jamal                         
                                                                               

Improve bool TypeError message                                   07/10/09
CLOSED http://bugs.python.org/issue6453    reopened tjreedy                       
       easy                                                                    

Add "example" keyword argument to optparse constructor           07/09/09
       http://bugs.python.org/issue6454    created  waylonis                      
                                                                               

Lib/distutils/tests/test_build_ext.py fails on windows           07/10/09
CLOSED http://bugs.python.org/issue6455    created  ocean-city                    
       patch                                                                   

locale.D_* and .T_* are int, not string                          07/10/09
CLOSED http://bugs.python.org/issue6456    created  dauerbaustelle                
                                                                               

subprocess.Popen.communicate can lose data from output/error str 07/10/09
       http://bugs.python.org/issue6457    created  dwalczak                      
                                                                               

With logging.config.fileConfig can't create logger without handl 07/10/09
       http://bugs.python.org/issue6458    created  sean                          
                                                                               



Issues Now Closed (40)
______________________

xdrlib fails to detect overflow (struct bug?)                     586 days
       http://bugs.python.org/issue1523    marketdickinson               
                                                                               

python2.6 -3 gives "warning: callable() not supported in 3.x" on  509 days
       http://bugs.python.org/issue2060    ezio.melotti                  
                                                                               

operator.{isCallable,sequenceIncludes} needs a fixer              474 days
       http://bugs.python.org/issue2370    alexandre.vassalotti          
       patch, 26backport, needs review                                         

Regression for executing packages                                 432 days
       http://bugs.python.org/issue2751    ncoghlan                      
                                                                               

truncation of text in tables in Library Reference PDF             407 days
       http://bugs.python.org/issue2977    georg.brandl                  
                                                                               

datetime.time does not support arithmetic                         370 days
       http://bugs.python.org/issue3250    lemburg                       
       patch                                                                   

os.chdir() et al: is the path str or bytes?                       298 days
       http://bugs.python.org/issue3810    r.david.murray                
                                                                               

pydoc in web server mode tails at initial request                 277 days
       http://bugs.python.org/issue4005    alexandre.vassalotti          
       patch, needs review                                                     

Tkinter cannot find Tcl/Tk on Mac OS X                            279 days
       http://bugs.python.org/issue4017    ronaldoussoren                
                                                                               

Partial function application 'from the right'                     150 days
       http://bugs.python.org/issue5191    amaury.forgeotdarc            
       patch                                                                   

Remove deprecated features from struct module                     116 days
       http://bugs.python.org/issue5463    marketdickinson               
       patch                                                                   

SocketServer: TypeError: waitpid() takes no keyword arguments      76 days
       http://bugs.python.org/issue5814    r.david.murray                
                                                                               

format(1234.5, '.4') gives misleading result                       70 days
       http://bugs.python.org/issue5864    eric.smith                    
       patch                                                                   

Windows Installer Error 1722 when opting for compilation at inst   57 days
       http://bugs.python.org/issue5960    keldonin                      
                                                                               

Py3.1 pdb doesn't deal well with syntax errors                     17 days
       http://bugs.python.org/issue6323    amaury.forgeotdarc            
       patch, needs review                                                     

logging: ValueError: I/O operation on closed file                  16 days
       http://bugs.python.org/issue6333    hpk                           
                                                                               

"L" integer suffix in Python 3.1 tutorial                           5 days
       http://bugs.python.org/issue6353    ezio.melotti                  
       patch                                                                   

README typo                                                         2 days
       http://bugs.python.org/issue6398    ezio.melotti                  
       patch                                                                   

Error reporting by logging.config.fileConfig()                      7 days
       http://bugs.python.org/issue6399    vsajip                        
                                                                               

Dictionaries should support __add__                                 2 days
       http://bugs.python.org/issue6410    exarkun                       
                                                                               

3.1 not functional                                                  0 days
       http://bugs.python.org/issue6411    element.effect                
                                                                               

incorrect log level in distutlis.dist for announce                  0 days
       http://bugs.python.org/issue6413    tarek                         
                                                                               

Failed to compile selectmodule.c on windows (trunk)                 5 days
       http://bugs.python.org/issue6416    amaury.forgeotdarc            
       easy                                                                    

Fix warning in nismodule.c on OpenBSD                               2 days
       http://bugs.python.org/issue6420    amaury.forgeotdarc            
       patch                                                                   

Idle3 crashing when using completion                                0 days
       http://bugs.python.org/issue6424    amaury.forgeotdarc            
                                                                               

imaplib.IMAP4.fetch() is missing documentation for message_set p    2 days
       http://bugs.python.org/issue6425    Sjoerder                      
                                                                               

Rename float*.[ch] to double.[ch]                                   0 days
       http://bugs.python.org/issue6427    loewis                        
                                                                               

TypeError: __bool__ should return bool or int, returned int         0 days
       http://bugs.python.org/issue6428    amaury.forgeotdarc            
       patch                                                                   

ImageTk.PhotoImage                                                  0 days
       http://bugs.python.org/issue6432    jpmieville                    
                                                                               

logging: cache the traceback text on formatter, instead of recor    2 days
       http://bugs.python.org/issue6435    vsajip                        
                                                                               

Personal invitation from Manas                                      0 days
       http://bugs.python.org/issue6437    marketdickinson               
                                                                               

2to3: convert deprecated string.maketrans to str.maketrans          0 days
       http://bugs.python.org/issue6440    r.david.murray                
                                                                               

2to3 adds spare brackets to print                                   0 days
       http://bugs.python.org/issue6451    r.david.murray                
                                                                               

Improve bool TypeError message                                      0 days
       http://bugs.python.org/issue6453    amaury.forgeotdarc            
       easy                                                                    

Lib/distutils/tests/test_build_ext.py fails on windows              0 days
       http://bugs.python.org/issue6455    tarek                         
       patch                                                                   

locale.D_* and .T_* are int, not string                             0 days
       http://bugs.python.org/issue6456    amaury.forgeotdarc            
                                                                               

win32 os.path.normpath not correct for leading slash cases       2371 days
       http://bugs.python.org/issue665336  amaury.forgeotdarc            
                                                                               

ntpath.expandvars doesn't expand Windows-style variables.        2091 days
       http://bugs.python.org/issue824371  amaury.forgeotdarc            
       easy                                                                    

struct.pack raises TypeError where it used to convert            1073 days
       http://bugs.python.org/issue1530559 marketdickinson               
       patch                                                                   

struct.pack("I", "foo"); struct.pack("L", "foo") should fail      745 days
       http://bugs.python.org/issue1741130 marketdickinson               
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 23 distutils compiler switch ignored                                  2 days
open    http://bugs.python.org/issue6377   

 18 logging: ValueError: I/O operation on closed file                 16 days
closed  http://bugs.python.org/issue6333   

 14 Dictionaries should support __add__                                2 days
closed  http://bugs.python.org/issue6410   

 12 platform.version() don't work as expected in Vista in portugues  357 days
open    http://bugs.python.org/issue3410   

 10 OS X: python3 from python-3.1.dmg crashes at startup               9 days
open    http://bugs.python.org/issue6393   

  9 logging: cache the traceback text on formatter, instead of reco    2 days
closed  http://bugs.python.org/issue6435   

  7 _winreg.QueryValueEx confused on double null terminated strings    2 days
open    http://bugs.python.org/issue6443   

  7 test_socketserver fails on trunk in test_ForkingTCPServer         11 days
open    http://bugs.python.org/issue6382   

  6 3.1 not functional                                                 0 days
closed  http://bugs.python.org/issue6411   

  5 Fraction fails equality test with a user-defined type              3 days
open    http://bugs.python.org/issue6431   




From pje at telecommunity.com  Fri Jul 10 20:17:40 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 10 Jul 2009 14:17:40 -0400
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.co
 m>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
	<94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com>
Message-ID: <20090710181438.E1CFE3A4752@sparrow.telecommunity.com>

At 03:31 PM 7/10/2009 +0200, Tarek Ziad? wrote:
>On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby<pje at telecommunity.com> wrote:
> > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
> >>
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Nick Coghlan wrote:
> >> > P.J. Eby wrote:
> >> >>>  Also,
> >> >>> why should the RECORD file be generated at all by bdist* commands?
> >> >> bdist commands that use "install --root" will get it as a side effect,
> >> >> so they don't have to do anything special.  They needn't even generate
> >> >> the PREFIXES, if PREFIXES is also generated by "install --root", and
> >> >> omits the root from the prefixes written.  In other words, most bdist
> >> >> tools wouldn't change.  bdist_wininst and bdist_msi might need to
> >> >> regenerate PREFIXES at actual installation time, but AFAIK those are
> >> >> the
> >> >> only ones.  (And the spec would allow for the possibility that a
> >> >> PREFIXES file can be incorrect.)
> >> >
> >> > I haven't seen any suggestions for handling non-relative paths that are
> >> > cleaner than this. The distutils based list that Tarek posted also gives
> >> > a good "starter set" of supported entries in the PREFIXES file:
> >> >
> >> > PURELIB=<path>
> >> > PLATLIB=<path>
> >> > HEADERS=<path>
> >> > SCRIPTS=<path>
> >> > DATA=<path>
> >> >
> >> > The question then would be whether relative paths should be handled
> >> > directly in the RECORD file (as in the current PEP) or by writing a "."
> >> > into PREFIXES to indicate relative paths.
>
>I like the idea of a separated PREFIXES file, then using variables in
>the RECORD file.

Please note that my proposal intentionally omitted "variables", but 
instead used a separate column to designate the prefix.  This allows 
some guarantees regarding path sanity that could otherwise be abused, 
like combining prefixes or inserting them in the middle of paths.  In 
this way, the formal conversion of a path would be 
os.path.join(PREFIX, *relative_path.split('/')), rather than a 
variable expansion.


From tjreedy at udel.edu  Fri Jul 10 20:26:43 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 10 Jul 2009 14:26:43 -0400
Subject: [Python-Dev] Search error on tracker?
In-Reply-To: <4A56C71B.5060006@v.loewis.de>
References: <h36ct2$al1$1@ger.gmane.org> <4A56C71B.5060006@v.loewis.de>
Message-ID: <h38152$umg$1@ger.gmane.org>

Martin v. L?wis wrote:
> Terry Reedy wrote:
>> Before posting http://bugs.python.org/issue6453
>> I searched the tracker for 'bool TypeError' to avoid making a duplicate
>> report. A few irrelevant hits were return but not the important one
>> http://bugs.python.org/issue6428
>> which was the one I needed to find.
> 
> When I search for 'bool TypeError', #6428 shows up. Are you sure you had
> been searching for closed reports as well?

Un no. Thanks. Perhaps that field should default to 'don't care' like 
all the others. Sorry for the noise.

Terry


From martin at v.loewis.de  Sat Jul 11 00:50:29 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 11 Jul 2009 00:50:29 +0200
Subject: [Python-Dev] Mercurial: tag generation incorrect
Message-ID: <4A57C5B5.5070000@v.loewis.de>

According to

http://hg.python.org/cpython/tags?revcount=50

the tag r301 belongs to revision be32850b093f. However, this
is really r69557, which was not tagged. be32850b093f is listed
as having a child revision, 52b0a279fec6, and ISTM that *this*
should be the revision that got tagged.

Regards,
Martin


From hfuerstenau at gmx.net  Sat Jul 11 07:06:22 2009
From: hfuerstenau at gmx.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=)
Date: Sat, 11 Jul 2009 07:06:22 +0200
Subject: [Python-Dev] Mercurial: tag generation incorrect
In-Reply-To: <4A57C5B5.5070000@v.loewis.de>
References: <4A57C5B5.5070000@v.loewis.de>
Message-ID: <4A581DCE.4050307@gmx.net>

> be32850b093f is listed
> as having a child revision, 52b0a279fec6, and ISTM that *this*
> should be the revision that got tagged.

I think the tag is correct. Note that the concept of tagging is
different in Mercurial, where a tag can only refer to a revision
previous to the one where it is inserted in .hgtags. If I understand
correctly, all relevant tagging revisions from SVN are replaced by
Mercurial revisions setting tags, which then refer to their immediate
predecessors.

- Hagen

From ziade.tarek at gmail.com  Sat Jul 11 12:28:57 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 11 Jul 2009 12:28:57 +0200
Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files
In-Reply-To: <20090710181438.E1CFE3A4752@sparrow.telecommunity.com>
References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com>
	<op.uwr13zlrbrrvlq@double>
	<20090709020938.5C0A83A407B@sparrow.telecommunity.com>
	<4A55AF44.4090508@gmail.com> <h35dtr$sa9$2@ger.gmane.org>
	<20090709190631.CE40E3A407B@sparrow.telecommunity.com>
	<94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com>
	<20090710181438.E1CFE3A4752@sparrow.telecommunity.com>
Message-ID: <94bdd2610907110328x5d0bd8c7w25f44460d55cc7de@mail.gmail.com>

On Fri, Jul 10, 2009 at 8:17 PM, P.J. Eby<pje at telecommunity.com> wrote:
> At 03:31 PM 7/10/2009 +0200, Tarek Ziad? wrote:
>>
>> On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby<pje at telecommunity.com> wrote:
>> > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
>> >>
>> >> -----BEGIN PGP SIGNED MESSAGE-----
>> >> Hash: SHA1
>> >>
>> >> Nick Coghlan wrote:
>> >> > P.J. Eby wrote:
>> >> >>> ?Also,
>> >> >>> why should the RECORD file be generated at all by bdist* commands?
>> >> >> bdist commands that use "install --root" will get it as a side
>> >> >> effect,
>> >> >> so they don't have to do anything special. ?They needn't even
>> >> >> generate
>> >> >> the PREFIXES, if PREFIXES is also generated by "install --root", and
>> >> >> omits the root from the prefixes written. ?In other words, most
>> >> >> bdist
>> >> >> tools wouldn't change. ?bdist_wininst and bdist_msi might need to
>> >> >> regenerate PREFIXES at actual installation time, but AFAIK those are
>> >> >> the
>> >> >> only ones. ?(And the spec would allow for the possibility that a
>> >> >> PREFIXES file can be incorrect.)
>> >> >
>> >> > I haven't seen any suggestions for handling non-relative paths that
>> >> > are
>> >> > cleaner than this. The distutils based list that Tarek posted also
>> >> > gives
>> >> > a good "starter set" of supported entries in the PREFIXES file:
>> >> >
>> >> > PURELIB=<path>
>> >> > PLATLIB=<path>
>> >> > HEADERS=<path>
>> >> > SCRIPTS=<path>
>> >> > DATA=<path>
>> >> >
>> >> > The question then would be whether relative paths should be handled
>> >> > directly in the RECORD file (as in the current PEP) or by writing a
>> >> > "."
>> >> > into PREFIXES to indicate relative paths.
>>
>> I like the idea of a separated PREFIXES file, then using variables in
>> the RECORD file.
>
> Please note that my proposal intentionally omitted "variables", but instead
> used a separate column to designate the prefix. ?This allows some guarantees
> regarding path sanity that could otherwise be abused, like combining
> prefixes or inserting them in the middle of paths. ?In this way, the formal
> conversion of a path would be os.path.join(PREFIX,
> *relative_path.split('/')), rather than a variable expansion.
>

Let's build a detailed example on real distributions then. I propose
docutils because it has scripts, a module and a package. Could you build its
files so we can see ?

I'll try to look for other real world examples.

Regards
Tarek




>



-- 
Tarek Ziad? | http://ziade.org

From solipsis at pitrou.net  Sun Jul 12 15:10:25 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 12 Jul 2009 13:10:25 +0000 (UTC)
Subject: [Python-Dev] hg mirror for 3.1 maintenance branch
Message-ID: <loom.20090712T130839-155@post.gmane.org>


Hi,

I've added the release31-maint branch to the temporary Mercurial mirrors:
http://code.python.org/hg

Regards

Antoine.



From erik at cq2.nl  Mon Jul 13 12:08:29 2009
From: erik at cq2.nl (Erik Groeneveld)
Date: Mon, 13 Jul 2009 12:08:29 +0200
Subject: [Python-Dev] Define metatype and a type that uses it
In-Reply-To: <e27efe130907080835p3dedf14fi7a8044f7c60b049d@mail.gmail.com>
References: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
	<e27efe130907080835p3dedf14fi7a8044f7c60b049d@mail.gmail.com>
Message-ID: <aaec99390907130308y39ac4e81v225bde3b1e706ee8@mail.gmail.com>

Amaury,

Thank you very much for your detailed explanation.  It helps to
understand it better, and it mostly works now.  There is one thing
however:

On Wed, Jul 8, 2009 at 17:35, Amaury Forgeot d'Arc<amauryfa at gmail.com> wrote:
> - Don't define a JObjectMeta struct, use JObjectType directly instead.
> An instance of the metatype is the type itself!
> - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base
> type (the correct value would be sizeof(PyHeapTypeObject), this is
> important in order to create derived classes in python)

I'd like to add a C pointer field to the metatype instance (the type).
So, contrary to your advice, I have defined:

typedef struct {
    PyHeapTypeObject x;
    void* p;
} JObjectMeta;

This seems the way to do it for objects, but for types, it doesn't
seem right, as the p member turns out to be overwritten unexpectedly
at runtime.

Reading Python's object.h file it turns out that PyHeapTypeObject
'extends' PyTypeObject, which in turn has a PyObject_VAR_HEAD init
macro.  So mustn't:

PyTypeObject JObjectMetaType = {
    PyObject_HEAD_INIT(NULL)
};

actually be:

PyTypeObject JObjectMetaType = {
    PyVarObject_HEAD_INIT(NULL, 1)
};

with:

JObjectMetaType.tp_basic_size = sizeof(JObjectMeta);
JObjectMetaType.tp_itemsize = sizeof(void*);

I tried this, but it doesn't keep my app from dumping core on an
overwritten 'p'.

My question basically is: how can I define a pointer for each type
created with this metatype, such as is intended by the JObjectMeta
struct?

Best regards
Erik Groeneveld

From amauryfa at gmail.com  Mon Jul 13 13:35:22 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 13 Jul 2009 13:35:22 +0200
Subject: [Python-Dev] Define metatype and a type that uses it
In-Reply-To: <aaec99390907130308y39ac4e81v225bde3b1e706ee8@mail.gmail.com>
References: <aaec99390907080543t18d94d21pe63886ad85273345@mail.gmail.com>
	<e27efe130907080835p3dedf14fi7a8044f7c60b049d@mail.gmail.com>
	<aaec99390907130308y39ac4e81v225bde3b1e706ee8@mail.gmail.com>
Message-ID: <e27efe130907130435w3e24375dh84ea10f76d4da3e4@mail.gmail.com>

Hi,

2009/7/13 Erik Groeneveld <erik at cq2.nl>:
> Amaury,
>
> Thank you very much for your detailed explanation. ?It helps to
> understand it better, and it mostly works now. ?There is one thing
> however:
>
> On Wed, Jul 8, 2009 at 17:35, Amaury Forgeot d'Arc<amauryfa at gmail.com> wrote:
>> - Don't define a JObjectMeta struct, use JObjectType directly instead.
>> An instance of the metatype is the type itself!
>> - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base
>> type (the correct value would be sizeof(PyHeapTypeObject), this is
>> important in order to create derived classes in python)
>
> I'd like to add a C pointer field to the metatype instance (the type).
> So, contrary to your advice, I have defined:
>
> typedef struct {
> ? ?PyHeapTypeObject x;
> ? ?void* p;
> } JObjectMeta;
>
> This seems the way to do it for objects, but for types, it doesn't
> seem right, as the p member turns out to be overwritten unexpectedly
> at runtime.
>
> Reading Python's object.h file it turns out that PyHeapTypeObject
> 'extends' PyTypeObject, which in turn has a PyObject_VAR_HEAD init
> macro. ?So mustn't:
>
> PyTypeObject JObjectMetaType = {
> ? ?PyObject_HEAD_INIT(NULL)
> };
>
> actually be:
>
> PyTypeObject JObjectMetaType = {
> ? ?PyVarObject_HEAD_INIT(NULL, 1)
> };
>
> with:
>
> JObjectMetaType.tp_basic_size = sizeof(JObjectMeta);
> JObjectMetaType.tp_itemsize = sizeof(void*);
>
> I tried this, but it doesn't keep my app from dumping core on an
> overwritten 'p'.
>
> My question basically is: how can I define a pointer for each type
> created with this metatype, such as is intended by the JObjectMeta
> struct?

The best is probably to store it in the class dictionary:
    PyObject_SetAttrString(self, "_javatype_", PyLong_FromVoidPtr(p));

-- 
Amaury Forgeot d'Arc

From fuzzyman at voidspace.org.uk  Mon Jul 13 16:48:19 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 13 Jul 2009 15:48:19 +0100
Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418)
In-Reply-To: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com>
References: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com>
Message-ID: <4A5B4933.2060801@voidspace.org.uk>

jason pellerin wrote:
> Bringing python-dev into the discussion at Barry's request. The
> summary is that a recent change to unittest.TestProgram breaks nose by
> moving self.testRunner initialization from it's old home in
> TestProgram.runTests to TestProgram.__init__. The very small patch
> attached to the ticket moves it back to runTests.
>
> Here's the ticket: http://bugs.python.org/issue6418
> And a link to the testing in python list discussion:
>
> http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html
>   

I can look at this - no reason for it not to be included in 2.6.3 and 
ported to trunk. Sorry for the late reply - I've been on holiday.

Michael

> JP (primary author of nose)
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From eric.pruitt at gmail.com  Mon Jul 13 17:46:43 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 10:46:43 -0500
Subject: [Python-Dev] Windows Toolchain
Message-ID: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>

Hello,

I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
Visual C++ Express Edition in order to test some modifications I made to the
_subprocess.c file as part of my Google Summer of Code proposal. I cannot
seem to figure out how to compile Python on Windows and could use some
guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
work when I attempt to open and no error message is given when clicked. When
opened from the Visual C++, it says "Solution folders are not supported in
this version of the application."

Any advice is greatly appreciated,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/79a8476b/attachment.htm>

From eric.pruitt at gmail.com  Mon Jul 13 17:53:07 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 10:53:07 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
Message-ID: <171e8a410907130853s61f29beew8b0ed96502b08e0b@mail.gmail.com>

Sorry, forgot to include my build log:

Command Lines:
http://pastebin.com/f25614b01

Output Window:
> Compiling...
> python.c
> Compiling resources...
> Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1
> Copyright (C) Microsoft Corporation.  All rights reserved.
> Linking...
> LINK : fatal error LNK1181: cannot open input file '.\python31_d.lib'
> Creating browse information file...
> Microsoft Browse Information Maintenance Utility Version 9.00.30729
> Copyright (C) Microsoft Corporation. All rights reserved.

On Mon, Jul 13, 2009 at 10:46, Eric Pruitt <eric.pruitt at gmail.com> wrote:

> Hello,
>
> I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
> Visual C++ Express Edition in order to test some modifications I made to the
> _subprocess.c file as part of my Google Summer of Code proposal. I cannot
> seem to figure out how to compile Python on Windows and could use some
> guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
> work when I attempt to open and no error message is given when clicked. When
> opened from the Visual C++, it says "Solution folders are not supported in
> this version of the application."
>
> Any advice is greatly appreciated,
> Eric
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/9237c969/attachment.htm>

From eric.pruitt at gmail.com  Mon Jul 13 18:32:49 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 11:32:49 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
Message-ID: <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>

I opened the solution, hit Ctrl+F5 and it began compiling but it fails with
this error:
LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'

How do I specify that "pythoncore" be built? When I follow your
instructions, right clicking on the "python" project in the "Solution
Explorer" panel, I get the exact same link error as above. If I hit F7, the
build log changes completely. The last lines of output with F7 are as
follows:

> 21>Build log was saved at "file://C:\Documents and
Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm"
> 21>select - 1 error(s), 0 warning(s)
> ========== Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
==========


On Mon, Jul 13, 2009 at 11:03, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> Hi,
>
> 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
> > Hello,
> >
> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> Microsoft
> > Visual C++ Express Edition in order to test some modifications I made to
> the
> > _subprocess.c file as part of my Google Summer of Code proposal. I cannot
> > seem to figure out how to compile Python on Windows and could use some
> > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
> > work when I attempt to open and no error message is given when clicked.
>
> Is Visual C++ correctly installed?
>
> > When
> > opened from the Visual C++, it says "Solution folders are not supported
> in
> > this version of the application."
>
> This is a warning from the Express Edition, which has absolutely no
> impact on the build. You can safely ignore it and continue.
>
> What steps did you perform exactly?  From your build log the
> "pythoncore" project was not built. Why?
>
> The following has always worked for me: In the "Solution explorer"
> panel, right-click on the "python" project, and click "Build". This
> should build the project and all its dependencies.
> Or just hit the F7 key and watch the whole solution build.
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/e0b0dbbe/attachment-0001.htm>

From fuzzyman at voidspace.org.uk  Mon Jul 13 18:37:57 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 13 Jul 2009 17:37:57 +0100
Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418)
In-Reply-To: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com>
References: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com>
Message-ID: <4A5B62E5.1090700@voidspace.org.uk>

jason pellerin wrote:
> Bringing python-dev into the discussion at Barry's request. The
> summary is that a recent change to unittest.TestProgram breaks nose by
> moving self.testRunner initialization from it's old home in
> TestProgram.runTests to TestProgram.__init__. The very small patch
> attached to the ticket moves it back to runTests.
>
> Here's the ticket: http://bugs.python.org/issue6418
> And a link to the testing in python list discussion:
>
> http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html
>
>   

This is fixed on the Python 2.6 maintenance branch (revision 73997) but 
needs fixing on trunk. No problem applying the fix but I'm struggling to 
test this without causing spurious extra output on stderr. The style of 
dependency injection used means that merely overriding sys.stderr does 
nothing and a custom stream can't be used as we want to test the default 
initialisation behavior.

Michael Foord

> JP (primary author of nose)
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From eric.pruitt at gmail.com  Mon Jul 13 19:00:34 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 12:00:34 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com> 
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> 
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
Message-ID: <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>

It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
Unix style but after fixing them, I still have the same problem.

On Mon, Jul 13, 2009 at 11:43, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
> > I opened the solution, hit Ctrl+F5 and it began compiling but it fails
> with
> > this error:
> > LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'
> >
> > How do I specify that "pythoncore" be built? When I follow your
> > instructions, right clicking on the "python" project in the "Solution
> > Explorer" panel, I get the exact same link error as above. If I hit F7,
> the
> > build log changes completely. The last lines of output with F7 are as
> > follows:
> >
> >> 21>Build log was saved at "file://C:\Documents and
> >>
> Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm"
> >> 21>select - 1 error(s), 0 warning(s)
> >> ========== Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
> >> ==========
> >
>
> Did you open the sub-project "python.vcproj" by mistake?
> You must open the solution file: "pcbuild.sln".
>
> And if pcbuild.sln does not load correctly, check that it has correct
> line endings: it is supposed to have DOS line endings.
>
> --
> Amaury Forgeot d'Arc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/96388832/attachment.htm>

From tlesher at gmail.com  Mon Jul 13 19:36:08 2009
From: tlesher at gmail.com (Tim Lesher)
Date: Mon, 13 Jul 2009 13:36:08 -0400
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
Message-ID: <9613db600907131036n12f92e07m89b36404a6e2476@mail.gmail.com>

On Mon, Jul 13, 2009 at 11:46, Eric Pruitt<eric.pruitt at gmail.com> wrote:
> Hello,
>
> I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
> Visual C++ Express Edition in order to test some modifications I made to the
> _subprocess.c file as part of my Google Summer of Code proposal. I cannot
> seem to figure out how to compile Python on Windows and could use some
> guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
> work when I attempt to open and no error message is given when clicked. When
> opened from the Visual C++, it says "Solution folders are not supported in
> this version of the application."

What version of Visual C++ Express are you using?  This sounds
suspiciously like a version issue.
-- 
Tim Lesher <tlesher at gmail.com>

From p.f.moore at gmail.com  Mon Jul 13 20:09:45 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 13 Jul 2009 19:09:45 +0100
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
Message-ID: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>

2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
> Unix style but after fixing them, I still have the same problem.

Where did you get your copy of the Python source from? If it's from
Subversion, I'm surprised the line endings are wrong. If it's
Mercurial, then this may be indicative of the still somewhat open
question of how line endings will be handled in Mercurial (which
doesn't have a server-side way of saying that a file should use
platform-native line endings).

If you still have the same problem after fixing the file, this may
indicate further damage, or maybe the fix isn't doing the right thing
as the file has already been opened (and mangled) by Visual Studio,

I'd suggest trying to reproduce your issue with a clean checkout from
Subversion.

It would also help if you could describe precisely the steps you took
to download the sources and build them (I appreciate that this may not
be easy now, after you've tried various things).

Paul.

From eric.pruitt at gmail.com  Mon Jul 13 20:21:42 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 13:21:42 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com> 
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> 
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com> 
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> 
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
Message-ID: <171e8a410907131121y354c5fbfn7cdbb993474760ff@mail.gmail.com>

The code I downloaded was the official Python 3.1 release tarball. I only
checked the line endings in the pcbuild.sln file so I am not sure about the
other files. I will attempt to build the Python 3000 SVN trunk shortly and
report back.

On Mon, Jul 13, 2009 at 13:09, Paul Moore <p.f.moore at gmail.com> wrote:

> 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
> > It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
> be
> > Unix style but after fixing them, I still have the same problem.
>
> Where did you get your copy of the Python source from? If it's from
> Subversion, I'm surprised the line endings are wrong. If it's
> Mercurial, then this may be indicative of the still somewhat open
> question of how line endings will be handled in Mercurial (which
> doesn't have a server-side way of saying that a file should use
> platform-native line endings).
>
> If you still have the same problem after fixing the file, this may
> indicate further damage, or maybe the fix isn't doing the right thing
> as the file has already been opened (and mangled) by Visual Studio,
>
> I'd suggest trying to reproduce your issue with a clean checkout from
> Subversion.
>
> It would also help if you could describe precisely the steps you took
> to download the sources and build them (I appreciate that this may not
> be easy now, after you've tried various things).
>
> Paul.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/aa689e1d/attachment-0001.htm>

From p.f.moore at gmail.com  Mon Jul 13 20:42:33 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 13 Jul 2009 19:42:33 +0100
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
Message-ID: <79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com>

2009/7/13 Paul Moore <p.f.moore at gmail.com>:
> 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
>> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
>> Unix style but after fixing them, I still have the same problem.
[...]
> I'd suggest trying to reproduce your issue with a clean checkout from
> Subversion.

FWIW, I did:

svn co http://svn.python.org/projects/python/tags/r31/ py31

Open pcbuild in Visual C++ 2008 Express. OK on the warning about
solution folders.

Right click "python" and choose "Build".

Works OK here. Also works if I download Python-3.1.tgz from python.org
and unpack it using (command line) bsdtar. I wonder - did you use
WinZip to unpack the archive? That has some sort of brain-damaged
"Smart" CRLF conversion for tar files, which may have screwed up your
files.

Paul.

From eric.pruitt at gmail.com  Mon Jul 13 22:14:46 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 13 Jul 2009 15:14:46 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com> 
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> 
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com> 
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> 
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> 
	<79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com>
Message-ID: <171e8a410907131314p328abe3h6ae86a61b07966b4@mail.gmail.com>

Well, I narrowed the culprit down to Windows DEP. I had been having trouble
with it for some time now, with it raising an error for nearly every program
I ran so I just disabled it completely and Python now compiles fine. Thank
you all for the assistance.

On Mon, Jul 13, 2009 at 13:42, Paul Moore <p.f.moore at gmail.com> wrote:

> 2009/7/13 Paul Moore <p.f.moore at gmail.com>:
> > 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
> >> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
> be
> >> Unix style but after fixing them, I still have the same problem.
> [...]
> > I'd suggest trying to reproduce your issue with a clean checkout from
> > Subversion.
>
> FWIW, I did:
>
> svn co http://svn.python.org/projects/python/tags/r31/ py31
>
> Open pcbuild in Visual C++ 2008 Express. OK on the warning about
> solution folders.
>
> Right click "python" and choose "Build".
>
> Works OK here. Also works if I download Python-3.1.tgz from python.org
> and unpack it using (command line) bsdtar. I wonder - did you use
> WinZip to unpack the archive? That has some sort of brain-damaged
> "Smart" CRLF conversion for tar files, which may have screwed up your
> files.
>
> Paul.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090713/0b0d8fe3/attachment.htm>

From SridharR at activestate.com  Mon Jul 13 22:40:25 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Mon, 13 Jul 2009 13:40:25 -0700
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com>
	<4A551899.1080105@trueblade.com>
	<87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20090709010447.2A86D3A407B@sparrow.telecommunity.com>
	<4A55621B.5070904@trueblade.com>
	<20090709033551.AA35D3A407B@sparrow.telecommunity.com>
	<4A55F674.6090200@trueblade.com>
	<79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com>
	<4A564C53.4080102@gmail.com>
	<79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com>
Message-ID: <op.uw0w1nivbrrvlq@double>

On Fri, 10 Jul 2009 01:56:36 -0700, Paul Moore <p.f.moore at gmail.com> wrote:

>> One thing that did occur to me based on this - do we want the format to
>> support designation of files (such as config files) that *shouldn't* be
>> uninstalled along with everything else? Or are we happy with not
>> mentioning the file in RECORD at all as the means of supporting that use
>> case?
> No. In my view, the uninstall feature is a relatively minor issue -
> people who want uninstall facilities generally use system packages (or
> complain that easy_install doesn't support uninstallation :-)). They
> aren't using setup.py install. (Of course, my experience is minimal in
> this regard, so maybe I'm wrong).
> Remember that the PEP says "Distutils will provide a **very basic**
> uninstall function" (my emphasis).
> Unless I'm misunderstanding, no-one is intending to use the new
> distutils uninstall feature in their own package manager
> (bdist_wininst, RPM, apt, ...)

Perhaps distutils can also provide an `uninstall` hook that application  
can use to do some pre-processing?

>> import distutils.installer
>> distutils.installer.register_uninstall_hoook(my_uninstall_hook)

I imagine that packages like pywin32 may have to register DLLs (install  
time) and unregister them (uninstall time).

Just a wild idea .. I understand that this may not be related to the  
current PEP. :-)

-srid

From ncoghlan at gmail.com  Mon Jul 13 23:04:34 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 14 Jul 2009 07:04:34 +1000
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
Message-ID: <4A5BA162.4010804@gmail.com>

Paul Moore wrote:
> 2009/7/13 Eric Pruitt <eric.pruitt at gmail.com>:
>> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
>> Unix style but after fixing them, I still have the same problem.
> 
> Where did you get your copy of the Python source from? If it's from
> Subversion, I'm surprised the line endings are wrong. If it's
> Mercurial, then this may be indicative of the still somewhat open
> question of how line endings will be handled in Mercurial (which
> doesn't have a server-side way of saying that a file should use
> platform-native line endings).

For the record, we only have SVN set to force Windows line endings for
the old VC6 project files (.dsp and .dsw). The newer versions of Visual
Studio can handle Unix line endings just fine so the .sln/.vcprops/etc
files are left with native line endings and hence end up with Unix line
endings in the source tarball.

Cheers,
Nick.

P.S. I must admit I've never really understood how Data Execution
Prevention is ever going to work in a world with interpreted languages
and just in time compilers... the line between data and code is fuzzier
than one might think for a lot of programs...

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Jul 13 23:09:07 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 14 Jul 2009 07:09:07 +1000
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <4A5BA162.4010804@gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
	<4A5BA162.4010804@gmail.com>
Message-ID: <4A5BA273.3010907@gmail.com>

Nick Coghlan wrote:
> For the record, we only have SVN set to force Windows line endings for
> the old VC6 project files (.dsp and .dsw). The newer versions of Visual
> Studio can handle Unix line endings just fine so the .sln/.vcprops/etc
> files are left with native line endings and hence end up with Unix line
> endings in the source tarball.

It would appear that Amaury just changed that. I'm not sure that was
actually necessary though (since nobody else has complained and the
original poster of this thread discovered the line endings weren't the
problem after all).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Mon Jul 13 23:24:01 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 13 Jul 2009 23:24:01 +0200
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
Message-ID: <4A5BA5F1.7080504@v.loewis.de>

> I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> Microsoft Visual C++ Express Edition in order to test some modifications
> I made to the _subprocess.c file as part of my Google Summer of Code
> proposal. 

As a posting guideline, please be careful to not post too many messages
in quick succession. If you run into a problem, try to solve it for
yourself. If you fail, consult your mentor and/or wait a couple of
hours. Then, when posting, summarize your findings and the alternatives
that you tried unsuccessfully.

If people learn that you post while being in the process of working
on a problem, they will learn that it is better to wait a day or two
before responding, in case you figure it out on your own.

Regards,
Martin

From amauryfa at gmail.com  Mon Jul 13 23:40:30 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 13 Jul 2009 23:40:30 +0200
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <4A5BA273.3010907@gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
	<4A5BA162.4010804@gmail.com> <4A5BA273.3010907@gmail.com>
Message-ID: <e27efe130907131440w4ee514do2f559e102bf7e709@mail.gmail.com>

2009/7/13 Nick Coghlan <ncoghlan at gmail.com>:
> Nick Coghlan wrote:
>> For the record, we only have SVN set to force Windows line endings for
>> the old VC6 project files (.dsp and .dsw). The newer versions of Visual
>> Studio can handle Unix line endings just fine so the .sln/.vcprops/etc
>> files are left with native line endings and hence end up with Unix line
>> endings in the source tarball.
>
> It would appear that Amaury just changed that. I'm not sure that was
> actually necessary though (since nobody else has complained and the
> original poster of this thread discovered the line endings weren't the
> problem after all).

Well, I was caught several times by this line ending problem.
And the first time, I did not even understand what was going on...

-- 
Amaury Forgeot d'Arc

From curt at hagenlocher.org  Mon Jul 13 23:50:48 2009
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Mon, 13 Jul 2009 14:50:48 -0700
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <4A5BA162.4010804@gmail.com>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
	<4A5BA162.4010804@gmail.com>
Message-ID: <d2155e360907131450t6ac2a55dh7417eeb33de35d25@mail.gmail.com>

On Mon, Jul 13, 2009 at 2:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> P.S. I must admit I've never really understood how Data Execution
> Prevention is ever going to work in a world with interpreted languages
> and just in time compilers... the line between data and code is fuzzier
> than one might think for a lot of programs...

The OS has to provide a mechanism to enable execution for a particular
region of memory. Under Windows, this is done by the VirtualProtect
function.

--
Curt Hagenlocher
curt at hagenlocher.org

From solipsis at pitrou.net  Mon Jul 13 23:55:11 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 13 Jul 2009 21:55:11 +0000 (UTC)
Subject: [Python-Dev] Windows Toolchain
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
	<4A5BA162.4010804@gmail.com>
	<d2155e360907131450t6ac2a55dh7417eeb33de35d25@mail.gmail.com>
Message-ID: <loom.20090713T215444-57@post.gmane.org>

Curt Hagenlocher <curt <at> hagenlocher.org> writes:
> 
> The OS has to provide a mechanism to enable execution for a particular
> region of memory. Under Windows, this is done by the VirtualProtect
> function.

More surprising is that Microsoft didn't whitelist their own toolchain.




From curt at hagenlocher.org  Tue Jul 14 00:07:45 2009
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Mon, 13 Jul 2009 15:07:45 -0700
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <loom.20090713T215444-57@post.gmane.org>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com>
	<e27efe130907130903y6faad024nf9eae5e8cef92eb0@mail.gmail.com>
	<171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com>
	<e27efe130907130943o3ffd6956u4286d489aa2781c1@mail.gmail.com>
	<171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com>
	<79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com>
	<4A5BA162.4010804@gmail.com>
	<d2155e360907131450t6ac2a55dh7417eeb33de35d25@mail.gmail.com>
	<loom.20090713T215444-57@post.gmane.org>
Message-ID: <d2155e360907131507k69e2d976w2c38c832e0da633a@mail.gmail.com>

On Mon, Jul 13, 2009 at 2:55 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Curt Hagenlocher <curt <at> hagenlocher.org> writes:
>>
>> The OS has to provide a mechanism to enable execution for a particular
>> region of memory. Under Windows, this is done by the VirtualProtect
>> function.
>
> More surprising is that Microsoft didn't whitelist their own toolchain.

But dev tools don't actually execute the code they create in-process;
they write the executable code out to disk and a new process is spun
up from that image. So there should be no need to disable DEP for
anything that the toolchain creates directly.

Given that this was affecting multiple programs, the most likely
culprit is either an anti-virus program or a virus -- though you
probably can't rule out some kind of glitch in the OS itself. :/

--
Curt Hagenlocher
curt at hagenlocher.org

From sridharr at activestate.com  Tue Jul 14 02:07:02 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 13 Jul 2009 17:07:02 -0700
Subject: [Python-Dev] PEP 376 - Open questions
In-Reply-To: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>
References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com>
	<79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com>
	<loom.20090708T121713-308@post.gmane.org>
	<4A54984D.4020305@trueblade.com>
	<79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com>
	<4A54AF4F.9080800@trueblade.com>
	<79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com>
	<20090708151647.9E0703A407B@sparrow.telecommunity.com>
	<79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com>
	<op.uwr1phe9brrvlq@double>
	<94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com>
Message-ID: <op.uw06l0gv1q4jlt@whymac.activestate.com>

On Thu, 09 Jul 2009 01:22:19 -0700, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

> On Thu, Jul 9, 2009 at 3:42 AM, Sridhar
> Ratnakumar<SridharR at activestate.com> wrote:
>> Other than easy_install/pip, there is also PyPM which is being  
>> developed at
>> ActiveState. PyPM is the Python package manager much like what ppm is  
>> for
>> ActivePerl.
>>
>
> Great ! besides the RECORD file, anything remark on the PEP from a
> PyPM point of view ?

I'll shortly send an email I composed regarding this.

> Once Paul has finished to work on the PEP 302 part of the prototype,
> maybe we could try it out
> on PyPM to see how it fits ?

I took a look at PEP 302 and Paul's new implemention in pkgutil2.py

   http://bitbucket.org/tarek/pep376/src/tip/pkgutil2.py

And yes, I'd happy to try the new pkgutil API on PyPM .. keeping  
interoperability with pip/easy_install installed packages in mind.

-srid





From sridharr at activestate.com  Tue Jul 14 02:12:32 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 13 Jul 2009 17:12:32 -0700
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
Message-ID: <op.uw06u6px1q4jlt@whymac.activestate.com>

Here are my comments regarding PEP 376 with respect to PyPM (the Python
package manager being developd at ActiveState)


Multiple versions: I understand that the PEP does not support
installation (thus uninstallation) of multiple versions of the same
package. Should this be explicitly mentioned in the PEP -- as
`get_distribution` API accepts only `name` argument, and not a `version`
argument?

> get_distribution(name) -> Distribution or None.
>Scans all elements in sys.path and looks for all directories ending
> with .egg-info. Returns a Distribution corresponding to the .egg-info
> directory that contains a PKG-INFO that matches name for the name
> metadata.
>Notice that there should be at most one result. The first result
> founded is returned. If the directory is not found, returns None.

Some packages have package names with mixed case. Example: ConfigObj
.. as registered in setup.py. However, other packages such as turbogears
specifies "configobj" (lowercase) in their install_requires.

Is `get_distribution(name)` supposed to handle mixed cases? Will it
match both 'ConfigObj' and 'configobj'?

> get_installed_files(local=False) -> iterator of (path, md5, size)

Will this also return the directories /created/ during the installation?
For example, will it also contain the entry "docutils" .. along with
"docutils/__init__.py"?

If not, how is the installer (pip, pypm, etc..) supposed to know which
directories to remove (docutils/) and which directories not to remove
(site-packages/, bin/, etc..)?

> The new version of PEP 345 (XXX work in progress) extends the Metadata
> standard and fullfills the requirements described in PEP 262, like the
> REQUIRES section.

Can you tell more about this?

I see that PEP 262 allows both distributions names ('docutils') and
modules/packages ('roman.py') in the 'Requires:' section. Is this how
the new PEP is going to adhere to? Or, is it going to adhere to PEP
345's way of allowing *only* modules/packages?

In practice, I noticed that packages usually specify distribution names
in their 'Requires:' file (or install_requires.txt in the case of
setuptools). Hence, PyPM *assumes* the install requirements to be
distribution name. But then .. most distributions have the same name as
their primary module/package.

Ok, so PEP 345 also specifies the 'Provides:' header. Does
easy_install/pip make use 'Provides:' at all when resolving
dependencies? For example, does 'pip install sphinx' go look for all
distributions that 'provides' the 'docutils' provision.. or does it
simply get the distribution named 'docutils'?

-srid

From fuzzyman at voidspace.org.uk  Tue Jul 14 03:15:22 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Tue, 14 Jul 2009 02:15:22 +0100
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
In-Reply-To: <op.uw06u6px1q4jlt@whymac.activestate.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
Message-ID: <E52F1994-E8C2-477B-83E8-7503D5BA769C@voidspace.org.uk>




--
http://www.ironpythoninaction.com

On 14 Jul 2009, at 01:12, "Sridhar Ratnakumar"  
<sridharr at activestate.com> wrote:

> Here are my comments regarding PEP 376 with respect to PyPM (the  
> Python
> package manager being developd at ActiveState)
>
>
> Multiple versions: I understand that the PEP does not support
> installation (thus uninstallation) of multiple versions of the same
> package. Should this be explicitly mentioned in the PEP -- as
> `get_distribution` API accepts only `name` argument, and not a  
> `version`
> argument?
>
>> get_distribution(name) -> Distribution or None.
>> Scans all elements in sys.path and looks for all directories ending
>> with .egg-info. Returns a Distribution corresponding to the .egg-info
>> directory that contains a PKG-INFO that matches name for the name
>> metadata.
>> Notice that there should be at most one result. The first result
>> founded is returned. If the directory is not found, returns None.
>
> Some packages have package names with mixed case. Example: ConfigObj
> .. as registered in setup.py. However, other packages such as  
> turbogears
> specifies "configobj" (lowercase) in their install_requires.
>
> Is `get_distribution(name)` supposed to handle mixed cases? Will it
> match both 'ConfigObj' and 'configobj'?
>

An abomination for which I am truly sorry - however to be precise I'm  
pretty sure the setup.py specifies configobj and it is only registered  
on PyPI with mixed case (which I don't believe I can change).

Michael




>> get_installed_files(local=False) -> iterator of (path, md5, size)
>
> Will this also return the directories /created/ during the  
> installation?
> For example, will it also contain the entry "docutils" .. along with
> "docutils/__init__.py"?
>
> If not, how is the installer (pip, pypm, etc..) supposed to know which
> directories to remove (docutils/) and which directories not to remove
> (site-packages/, bin/, etc..)?
>
>> The new version of PEP 345 (XXX work in progress) extends the  
>> Metadata
>> standard and fullfills the requirements described in PEP 262, like  
>> the
>> REQUIRES section.
>
> Can you tell more about this?
>
> I see that PEP 262 allows both distributions names ('docutils') and
> modules/packages ('roman.py') in the 'Requires:' section. Is this how
> the new PEP is going to adhere to? Or, is it going to adhere to PEP
> 345's way of allowing *only* modules/packages?
>
> In practice, I noticed that packages usually specify distribution  
> names
> in their 'Requires:' file (or install_requires.txt in the case of
> setuptools). Hence, PyPM *assumes* the install requirements to be
> distribution name. But then .. most distributions have the same name  
> as
> their primary module/package.
>
> Ok, so PEP 345 also specifies the 'Provides:' header. Does
> easy_install/pip make use 'Provides:' at all when resolving
> dependencies? For example, does 'pip install sphinx' go look for all
> distributions that 'provides' the 'docutils' provision.. or does it
> simply get the distribution named 'docutils'?
>
> -srid
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk

From eric.pruitt at gmail.com  Tue Jul 14 19:42:09 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Tue, 14 Jul 2009 12:42:09 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <4A5BA5F1.7080504@v.loewis.de>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<4A5BA5F1.7080504@v.loewis.de>
Message-ID: <171e8a410907141042y3c2ce93bwfc38b23db4ce02f@mail.gmail.com>

I will keep a grace period in mind when I am posting. In my defense,
however, I had been working on the problem for some time before posting this
and my messages in quick succession were in response to the suggestions
people offered. The problem has arisen again but I will work on it some more
and will also post a detailed post on my GSoC blog of the process I went
through when I encountered the issue.

On Mon, Jul 13, 2009 at 16:24, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> > Microsoft Visual C++ Express Edition in order to test some modifications
> > I made to the _subprocess.c file as part of my Google Summer of Code
> > proposal.
>
> As a posting guideline, please be careful to not post too many messages
> in quick succession. If you run into a problem, try to solve it for
> yourself. If you fail, consult your mentor and/or wait a couple of
> hours. Then, when posting, summarize your findings and the alternatives
> that you tried unsuccessfully.
>
> If people learn that you post while being in the process of working
> on a problem, they will learn that it is better to wait a day or two
> before responding, in case you figure it out on your own.
>
> Regards,
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090714/16cd577c/attachment.htm>

From dinov at microsoft.com  Tue Jul 14 22:16:10 2009
From: dinov at microsoft.com (Dino Viehland)
Date: Tue, 14 Jul 2009 20:16:10 +0000
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
Message-ID: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>

I'm updating IronPython to match CPython's behavior w/ for this issue:

http://bugs.python.org/issue1683368

One thing that I've noticed is that this doesn't seem to be respecting the deletion of attributes (on 2.6.2):

class x(object): pass

x().__init__(2,3,4) # throws - seems right

class x(object):
    def __new__(cls, *args):
            return object.__new__(cls)

x().__init__(2,3,4)  # doesn't throw - seems right

del x.__new__
x().__init__(2,3,4) # doesn't throw - I would expect this to throw.

Is this just a bug in CPython not updating whether __new__ has been defined?  Or is there something that makes this behavior expected which I'm just missing?



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090714/65cddec3/attachment.htm>

From benjamin at python.org  Tue Jul 14 23:11:07 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 14 Jul 2009 16:11:07 -0500
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
Message-ID: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>

2009/7/14 Dino Viehland <dinov at microsoft.com>:
> Is this just a bug in CPython not updating whether __new__ has been
> defined?? Or is there something that makes this behavior expected which I?m
> just missing?

There's a wonderful comment about this in typeobject.c:

/* You may wonder why object.__new__() only complains about arguments
   when object.__init__() is not overridden, and vice versa.

   Consider the use cases:

   1. When neither is overridden, we want to hear complaints about
      excess (i.e., any) arguments, since their presence could
      indicate there's a bug.

   2. When defining an Immutable type, we are likely to override only
      __new__(), since __init__() is called too late to initialize an
      Immutable object.  Since __new__() defines the signature for the
      type, it would be a pain to have to override __init__() just to
      stop it from complaining about excess arguments.

   3. When defining a Mutable type, we are likely to override only
      __init__().  So here the converse reasoning applies: we don't
      want to have to override __new__() just to stop it from
      complaining.

   4. When __init__() is overridden, and the subclass __init__() calls
      object.__init__(), the latter should complain about excess
      arguments; ditto for __new__().

   Use cases 2 and 3 make it unattractive to unconditionally check for
   excess arguments.  The best solution that addresses all four use
   cases is as follows: __init__() complains about excess arguments
   unless __new__() is overridden and __init__() is not overridden
   (IOW, if __init__() is overridden or __new__() is not overridden);
   symmetrically, __new__() complains about excess arguments unless
   __init__() is overridden and __new__() is not overridden
   (IOW, if __new__() is overridden or __init__() is not overridden).

   However, for backwards compatibility, this breaks too much code.
   Therefore, in 2.6, we'll *warn* about excess arguments when both
   methods are overridden; for all other cases we'll use the above
   rules.

*/



-- 
Regards,
Benjamin

From eric.pruitt at gmail.com  Tue Jul 14 23:14:58 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Tue, 14 Jul 2009 16:14:58 -0500
Subject: [Python-Dev] Windows Toolchain
In-Reply-To: <4A5BA5F1.7080504@v.loewis.de>
References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> 
	<4A5BA5F1.7080504@v.loewis.de>
Message-ID: <171e8a410907141414t49d3369v213a8021ec49efbc@mail.gmail.com>

The problem appears resolved again and I have two posts on the issue on my
blog located at http://subdev.blogspot.com/. I was missing an MSVC++
run-time DLL and re-installing Visual C++ Express fixed the problem. A bit
of a somewhat random note -- looking through some of the buildbot output for
Python 3.1, I noticed that at one point, the bot compilations had the same
issue; Google Cache<http://74.125.95.132/search?q=cache:oadoPcYTjW8J:www.python.org/dev/buildbot/3.x/x86%2520XP-4%25203.x/builds/606/step-compile/0/text+LINK+:+fatal+error+LNK1181:+cannot+open+input+file+%27.%5Cpython31_d.lib%27&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a>
.

On Mon, Jul 13, 2009 at 16:24, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> > Microsoft Visual C++ Express Edition in order to test some modifications
> > I made to the _subprocess.c file as part of my Google Summer of Code
> > proposal.
>
> As a posting guideline, please be careful to not post too many messages
> in quick succession. If you run into a problem, try to solve it for
> yourself. If you fail, consult your mentor and/or wait a couple of
> hours. Then, when posting, summarize your findings and the alternatives
> that you tried unsuccessfully.
>
> If people learn that you post while being in the process of working
> on a problem, they will learn that it is better to wait a day or two
> before responding, in case you figure it out on your own.
>
> Regards,
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090714/dbae6784/attachment-0001.htm>

From dinov at microsoft.com  Tue Jul 14 23:25:34 2009
From: dinov at microsoft.com (Dino Viehland)
Date: Tue, 14 Jul 2009 21:25:34 +0000
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
	<1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
Message-ID: <1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com>

Benjamin wrote:

> There's a wonderful comment about this in typeobject.c:
>

This is basically the same what I've gathered from the issue
description which was quite helpful.  But in this case we're
dealing with mutating the type object and changing whether
__new__ or __init__ exist at all at runtime - and
object.__new__/__init__ don't seem to be picking up on the
change.

I believe that the comments here w.r.t. mutability/immutability
are more about whether the instances are immutable and not
the type objects themselves - for example list vs tuple where
tuple has __new__ but no __init__.

Based upon the behavior I'm seeing it seems to me that the
presence of __new__ / __init__ must be getting cached somewhere
and the deletion isn't updating the cache and that's specifically
what struck me as odd here.




From pje at telecommunity.com  Wed Jul 15 06:25:02 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 00:25:02 -0400
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.co
 m>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
	<1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
Message-ID: <20090715042200.946A13A410E@sparrow.telecommunity.com>

At 04:11 PM 7/14/2009 -0500, Benjamin Peterson wrote:
>4. When __init__() is overridden, and the subclass __init__() 
>calls       object.__init__(), the latter should complain about 
>excess       arguments; ditto for __new__().

Actually, this rule is a PITA, because there's no good way to get rid 
of the warnings when you're trying to write MRO-agnostic mixins.

In other words, it negates many of the gains that were obtained by 
having new-style MROs, since you can no-longer write pass-through 
constructors that leave their arguments untouched.  Instead, every 
class must know ahead of time whether it will be the "last" class 
before 'object' -- thereby making it impossible to slip other mixins 
into the chain below them.

In effect, 2.6 forces you to have a common known base class *other* 
than 'object' in order to write co-operative classes.  :-(



From brett at python.org  Wed Jul 15 06:26:36 2009
From: brett at python.org (Brett Cannon)
Date: Tue, 14 Jul 2009 21:26:36 -0700
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?
Message-ID: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>

I implemented get_filename() as specified in PEP 302 for importlib's source
and bytecode loaders and I was starting to create the ABC for importlib.abc,
but then I realized that perhaps the loader should live in runpy instead of
importlib.
Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot.
But this addition to the PEP 302 protocols is very specific to runpy and is
not directly required to make imports work (the implementation was just a
reshuffling of pre-existing code from one method to another). I am tempted
to leave the ABC out of importlib and stop at implementing get_filename()
for importlib.abc.PyLoader and PyPycLoader.

Any opinions?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090714/431dbe04/attachment.htm>

From greg.ewing at canterbury.ac.nz  Wed Jul 15 09:29:27 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 15 Jul 2009 19:29:27 +1200
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <20090715042200.946A13A410E@sparrow.telecommunity.com>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
	<1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
	<20090715042200.946A13A410E@sparrow.telecommunity.com>
Message-ID: <4A5D8557.90703@canterbury.ac.nz>

P.J. Eby wrote:

> In effect, 2.6 forces you to have a common known base class *other* than 
> 'object' in order to write co-operative classes.  :-(

You have to do that anyway if you want to make cooperative
calls to any method *other* that __init__.

-- 
Greg

From ziade.tarek at gmail.com  Wed Jul 15 11:01:24 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 11:01:24 +0200
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
In-Reply-To: <op.uw06u6px1q4jlt@whymac.activestate.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
Message-ID: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>

On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
Ratnakumar<sridharr at activestate.com> wrote:
> Here are my comments regarding PEP 376 with respect to PyPM (the Python
> package manager being developd at ActiveState)
>
>
> Multiple versions: I understand that the PEP does not support
> installation (thus uninstallation) of multiple versions of the same
> package. Should this be explicitly mentioned in the PEP -- as
> `get_distribution` API accepts only `name` argument, and not a `version`
> argument?

That's another can of worms ;)

Before I answer here's a bit of background, i's a bit long but required, sorry

For people that don't want to read the rest, here's the idea :
multiple version support imho should
be introduced later, if they are to be introduced, by extending PEP
302 protocol.

The long explanantion now:

given a "foo" package, containing a "bar" module, multiple versions
support implies to do one of these:

1 - a custom PEP 302-like loader/importer that picks a version of
"foo" when the code imports the "bar" module.
  this works if the "foo" package is not directly available in
sys.path, and if the custom loader/importer is put in sys.meta_path
  for example.  If "foo 0.9" is located in /var/packages/foo/0.9 and
if "foo 1.0" is in  /var/packages/foo/1.0,
  The loader will select the right foo package to load and return with
through a loader that scans   /var/packages/foo/*

  To make it work it requires 2 things :
        a/ a version comparison system (see PEP 386) that will make
the loader pick the "latest" version by default
        b/ an API that will force the loader to pick one particular version

2 - changing the paths in sys.path to include the path containing the
right version, and let the existing importer/loader do the work.
That's what setuptools does with its multiple version system: an API
called "require" will let you change sys.path on the fly

           >>> from pkg_resources import require
           >>> require('docutils==0.4')     <--- looks for a docutils
egg distribution and adds it in the path

So if we support multiple versions in Python, (I'd love too). PEP 376
would need to be able to find the various versions
of each distribution, not by scanning sys.path but rather by scanning
a arbitrary collection of directories, then publishing the
right ones in sys.path (with a PEP 302 loader, or ala setuptools by
inserting them in sys.path)

In other words this would require changing the way the distributions
are stored. e.g. in self-contained eggs or in a brand-new
storage tree. (I am currently experimenting this with "virtual
site-packages" see http://bitbucket.org/tarek/vsp/src/tip/README.txt)

But as we said earlier, people might want to store their modules
anywhere (on a sql database, on Mars, etc.) and provide
a PEP 302-like loader for them. PJE has "eggs" but John Doe might want
to store its packages differently and provide an importer/loader for
them.

So each one of them provides a "package manager", which should composed of :

A- a loader/importer system
B- an installation system (that is easy_install -m for setuptools)
C- query APIs
D- a version comparison system
E- an uninstaller

So the real solution is to work with PEP 302 importers/loaders (A)
(e.g. "package managers")

Which means that PEP 302 need to be changed to become
'distribution-aware' as Paul said.
Because we would then be able to browse distributions (C) that are not
already loaded in sys.path, so work on two versions
of the same distribution.

but some open questions remains:

It also implies that each package manager provides its installer (B)
and a version comparison system (D)

I'm not sure about the way package installers could be declared. Plus,
how people would deal with several installers ?
For the version comparison system I am not sure either, but it would
require to have one global version comparison
system to rule them all otherwise some conflicts may occur.

So there's no plan to support multiple versions yet, because that
requires another PEP imho.

Since distutils is a package manager in some ways (it provides an
installer, and stores distributions that are made
available in sys.path) My feeling is that we need first to finish
what's missing to make it fully usable (eg query apis + uninstaller)

Then maybe think about extending PEP 302

>
>> get_distribution(name) -> Distribution or None.
>> Scans all elements in sys.path and looks for all directories ending
>> with .egg-info. Returns a Distribution corresponding to the .egg-info
>> directory that contains a PKG-INFO that matches name for the name
>> metadata.
>> Notice that there should be at most one result. The first result
>> founded is returned. If the directory is not found, returns None.
>
> Some packages have package names with mixed case. Example: ConfigObj
> .. as registered in setup.py. However, other packages such as turbogears
> specifies "configobj" (lowercase) in their install_requires.
>
> Is `get_distribution(name)` supposed to handle mixed cases? Will it
> match both 'ConfigObj' and 'configobj'?

As PJE said, we need normalization here yes.

Right now PyPI is case insensitive for its index:

http://pypi.python.org/simple/ConfigObj ==
http://pypi.python.org/simple/configobj

But in the meantime, IIRC, the XML-RPC apis are case sensitive, and so the
html browsing. easy_install is case insensitive though, because it
uses the index.

So we should be case-insensitive everywhere, so in PEP 376 too.

>
>> get_installed_files(local=False) -> iterator of (path, md5, size)
>
> Will this also return the directories /created/ during the installation?
> For example, will it also contain the entry "docutils" .. along with
> "docutils/__init__.py"?

I don't think it's necessary to add "docutils" if
"docutils/__init__.py" is present

But for empty directories added during installation we should add the I guess.

So, I'll add a note.

>
> If not, how is the installer (pip, pypm, etc..) supposed to know which
> directories to remove (docutils/) and which directories not to remove
> (site-packages/, bin/, etc..)?
>
>> The new version of PEP 345 (XXX work in progress) extends the Metadata
>> standard and fullfills the requirements described in PEP 262, like the
>> REQUIRES section.
>
> Can you tell more about this?
>
> I see that PEP 262 allows both distributions names ('docutils') and
> modules/packages ('roman.py') in the 'Requires:' section. Is this how
> the new PEP is going to adhere to? Or, is it going to adhere to PEP
> 345's way of allowing *only* modules/packages?

The plan is to add what setuptools called "installed_requires", so
you can tell which *distributions* should be installed, no matter if
they are composed
of a single module, or many packages.

>
> In practice, I noticed that packages usually specify distribution names
> in their 'Requires:' file (or install_requires.txt in the case of
> setuptools). Hence, PyPM *assumes* the install requirements to be
> distribution name. But then .. most distributions have the same name as
> their primary module/package.

That's it yes: it will be distribution aware. If a module or package
has the same name
than the distribution name, it will make no difference.

>
> Ok, so PEP 345 also specifies the 'Provides:' header. Does
> easy_install/pip make use 'Provides:' at all when resolving
> dependencies? For example, does 'pip install sphinx' go look for all
> distributions that 'provides' the 'docutils' provision.. or does it
> simply get the distribution named 'docutils'?

setuptools doesn't. I don't think pip does.

btw: is PyPM a public project ?

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From p.f.moore at gmail.com  Wed Jul 15 12:10:13 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 11:10:13 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
Message-ID: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>

2009/7/15 Tarek Ziad? <ziade.tarek at gmail.com>:
> On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
> Ratnakumar<sridharr at activestate.com> wrote:
>> Here are my comments regarding PEP 376 with respect to PyPM (the Python
>> package manager being developd at ActiveState)
>>
>>
>> Multiple versions: I understand that the PEP does not support
>> installation (thus uninstallation) of multiple versions of the same
>> package. Should this be explicitly mentioned in the PEP -- as
>> `get_distribution` API accepts only `name` argument, and not a `version`
>> argument?
>
> That's another can of worms ;)

:-)

> Before I answer here's a bit of background, i's a bit long but required, sorry
>
> For people that don't want to read the rest, here's the idea :
> multiple version support imho should
> be introduced later, if they are to be introduced, by extending PEP
> 302 protocol.

Disclaimer: I've only read the short version, so if some of this is
covered in the longer explanation, I apologise now.

-1.

In my view, multiple version support is not at all related to PEP 302
- or to core Python in general. The import statement has no concept of
versions, any version handling is done by explicit user manipulation
of sys.path.

PEP 302 is currently purely an import protocol. As such, it only cares
about locating the correct code to run to populate sys.modules['foo'].
Once the code has been located, there are a number of other details
that might be useful, hence the extensions like get_data,
get_filename, etc. But note that these are all *loader* extensions -
their starting point is an imported module.

The PEP 376 support I've just added is a *finder* extension, which is
working alongside the scanning of the container - but rather than
looking for modules, it's looking for distributions. Disappointingly
(for me) it turned out not to give much opportunity to share code -
the finder extensions could just as easily have been a completely
independent protocol.

PEP 376 support has added a requirement for 3 additional methods to
the existing 1 finder method in PEP 302. That's already a 300%
increase in complexity. I'm against adding any further complexity to
PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
*own* locator protocol and avoid any extra burden on PEP 302. I'm not
sure implementers of PEP 302 importers will even provide the current
PEP 376 extensions.

I propose that before the current prototype is turned into a final
(spec and) implementation, the PEP 302 extensions are extracted and
documented as an independent protocol, purely part of PEP 376. (This
*helps* implementers, as they can write support for, for example,
eggs, without needing to modify the existing egg importer). I'll
volunteer to do that work - but I won't start until the latest
iteration of questions and discussions has settled down and PEP 376
has achieved a stable form with the known issues addressed.

Of course, this is moving more and more towards saying that the design
of setuptools, with its generic means for locating distributions, etc
etc, is the right approach. We're reinventing the wheel here. But the
problem is that too many people dislike setuptools as it stands for it
to gain support. My understanding is that the current set of PEPs were
intended to be a stripped down, more generally acceptable subset of
setuptools. Let's keep them that way (and omit the complexities of
multi-version support).

If you want setuptools, you know where to get it...

Paul.

From fuzzyman at voidspace.org.uk  Wed Jul 15 12:17:38 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 15 Jul 2009 11:17:38 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
Message-ID: <4A5DACC2.5030609@voidspace.org.uk>

Paul Moore wrote:
> 2009/7/15 Tarek Ziad? <ziade.tarek at gmail.com>:
>   
>> On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
>> Ratnakumar<sridharr at activestate.com> wrote:
>>     
>>> Here are my comments regarding PEP 376 with respect to PyPM (the Python
>>> package manager being developd at ActiveState)
>>>
>>>
>>> Multiple versions: I understand that the PEP does not support
>>> installation (thus uninstallation) of multiple versions of the same
>>> package. Should this be explicitly mentioned in the PEP -- as
>>> `get_distribution` API accepts only `name` argument, and not a `version`
>>> argument?
>>>       
>> That's another can of worms ;)
>>     
>
> :-)
>
>   
>> Before I answer here's a bit of background, i's a bit long but required, sorry
>>
>> For people that don't want to read the rest, here's the idea :
>> multiple version support imho should
>> be introduced later, if they are to be introduced, by extending PEP
>> 302 protocol.
>>     
>
> Disclaimer: I've only read the short version, so if some of this is
> covered in the longer explanation, I apologise now.
>
> -1.
>   

I agree. People with versioning issues *should* be using virtualenv.

Michael Foord

> In my view, multiple version support is not at all related to PEP 302
> - or to core Python in general. The import statement has no concept of
> versions, any version handling is done by explicit user manipulation
> of sys.path.
>
> PEP 302 is currently purely an import protocol. As such, it only cares
> about locating the correct code to run to populate sys.modules['foo'].
> Once the code has been located, there are a number of other details
> that might be useful, hence the extensions like get_data,
> get_filename, etc. But note that these are all *loader* extensions -
> their starting point is an imported module.
>
> The PEP 376 support I've just added is a *finder* extension, which is
> working alongside the scanning of the container - but rather than
> looking for modules, it's looking for distributions. Disappointingly
> (for me) it turned out not to give much opportunity to share code -
> the finder extensions could just as easily have been a completely
> independent protocol.
>
> PEP 376 support has added a requirement for 3 additional methods to
> the existing 1 finder method in PEP 302. That's already a 300%
> increase in complexity. I'm against adding any further complexity to
> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
> *own* locator protocol and avoid any extra burden on PEP 302. I'm not
> sure implementers of PEP 302 importers will even provide the current
> PEP 376 extensions.
>
> I propose that before the current prototype is turned into a final
> (spec and) implementation, the PEP 302 extensions are extracted and
> documented as an independent protocol, purely part of PEP 376. (This
> *helps* implementers, as they can write support for, for example,
> eggs, without needing to modify the existing egg importer). I'll
> volunteer to do that work - but I won't start until the latest
> iteration of questions and discussions has settled down and PEP 376
> has achieved a stable form with the known issues addressed.
>
> Of course, this is moving more and more towards saying that the design
> of setuptools, with its generic means for locating distributions, etc
> etc, is the right approach. We're reinventing the wheel here. But the
> problem is that too many people dislike setuptools as it stands for it
> to gain support. My understanding is that the current set of PEPs were
> intended to be a stripped down, more generally acceptable subset of
> setuptools. Let's keep them that way (and omit the complexities of
> multi-version support).
>
> If you want setuptools, you know where to get it...
>
> Paul.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/


From solipsis at pitrou.net  Wed Jul 15 12:27:58 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 15 Jul 2009 10:27:58 +0000 (UTC)
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to
	runpy?
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
Message-ID: <loom.20090715T102404-716@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:
> 
> 
> I implemented get_filename() as specified in PEP 302 for importlib's source
and bytecode loaders and I was starting to create the ABC for importlib.abc,
but then I realized that perhaps the loader should live in runpy instead of
importlib.
> 
> Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot. But
this addition to the PEP 302 protocols is very specific to runpy and is not
directly required to make imports work (the implementation was just a
reshuffling of pre-existing code from one method to another). I am tempted to
leave the ABC out of importlib and stop at implementing get_filename() for
importlib.abc.PyLoader and PyPycLoader.

I am not sure when this discussion started. Are you replying to a 3 month-old
message of yours? :)

In any case, keeping all import-related ABCs in a single place sounds like a
good idea.
Alternatively, if the method is runpy specific, it doesn't have its place in an
ABC, does it?


Antoine.



From p.f.moore at gmail.com  Wed Jul 15 14:39:31 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 13:39:31 +0100
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to
	runpy?
In-Reply-To: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
Message-ID: <79990c6b0907150539p33def8aid70f64704bcece6d@mail.gmail.com>

2009/7/15 Brett Cannon <brett at python.org>:
> I implemented get_filename() as specified in PEP 302 for importlib's source
> and bytecode loaders and I was starting to create the ABC for importlib.abc,
> but then I realized that perhaps the loader should live in runpy instead of
> importlib.
> Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot.
> But this addition to the PEP 302 protocols is very specific to runpy and is
> not directly required to make imports work (the implementation was just a
> reshuffling of pre-existing code from one method to another). I am tempted
> to leave the ABC out of importlib and stop at implementing get_filename()
> for importlib.abc.PyLoader and PyPycLoader.
> Any opinions?

importlib.

In my view, all of the interfaces defined by PEP 302 should be
available from one place. And importlib is that place.

Paul.

From ncoghlan at gmail.com  Wed Jul 15 15:41:40 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 15 Jul 2009 23:41:40 +1000
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>	<1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
	<1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com>
Message-ID: <4A5DDC94.60509@gmail.com>

Dino Viehland wrote:
> Based upon the behavior I'm seeing it seems to me that the
> presence of __new__ / __init__ must be getting cached somewhere
> and the deletion isn't updating the cache and that's specifically
> what struck me as odd here.

Digging through typeobject.c, it isn't clear to me why this wouldn't be
getting picked up.

If all is working correctly:

1. When the class is created, tp_new and tp_init are copied from the
base class (see inherit_special and inherit_slots)

2. When object_new and object_init are called they compare the values
stored in the tp_new and tp_init slots with their own function addresses
to decide whether or not to raise the warning

3. When either of those is deleted, the update_slots call in
type_setattro should fix up the slot inheritance, and hence affect
future calls to __new__ and __init__

That said, the following comments in update_one_slot() are leading me to
wonder if Guido might owe Dino a beer:

{
        /* The __new__ wrapper is not a wrapper descriptor,
            so must be special-cased differently.
            If we don't do this, creating an instance will
            always use slot_tp_new which will look up
            __new__ in the MRO which will call tp_new_wrapper
            which will look through the base classes looking
            for a static base and call its tp_new (usually
            PyType_GenericNew), after performing various
            sanity checks and constructing a new argument
            list.  Cut all that nonsense short -- this speeds
            up instance creation tremendously. */
        specific = (void *)type->tp_new;
        /* XXX I'm not 100% sure that there isn't a hole
            in this reasoning that requires additional
            sanity checks.  I'll buy the first person to
            point out a bug in this reasoning a beer. */
}

I *think* this logic may be getting confused when update_one_slot() is
called immediately after the "del x.__new__" call has stuffed a NULL
into the tp_new slot. In the normal class creation case the tp_new
method will have been copied down from the parent class so the
conditions when update_one_slot() gets called are different (ditto for
when you actually *set* x.__new__ rather than delete it).

If nobody else gets anywhere with this I should have a chance to
actually play with it (rather than just reading code) tomorrow evening.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Jul 15 15:51:39 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 15 Jul 2009 23:51:39 +1000
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or
	to	runpy?
In-Reply-To: <loom.20090715T102404-716@post.gmane.org>
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
	<loom.20090715T102404-716@post.gmane.org>
Message-ID: <4A5DDEEB.4040700@gmail.com>

Antoine Pitrou wrote:
> I am not sure when this discussion started. Are you replying to a 3 month-old
> message of yours? :)

That depends on how you define the beginning of the discussion...

However, the fact that importlib doesn't implement the comparatively
recent get_filename() optional extension documented in PEP 302 came up
in one of the PEP 376 threads within the last week or so.

> In any case, keeping all import-related ABCs in a single place sounds like a
> good idea.
> Alternatively, if the method is runpy specific, it doesn't have its place in an
> ABC, does it?

While runpy is the only client in the standard library for the
get_filename() method, the method is still a PEP 302 extension. I
documented the extension in the PEP as loaders are the only things
reliably in a position to provide the filename details that runpy needs
to set __file__ and sys.argv[0] correctly and until importlib came along
PEP 302 itself was the only real documentation of that API.

Since importlib is now the "go-to" location for people that want to
write their own PEP 302 importers and loaders, I would say that it is
also the right place for the new ExecutionLoader ABC.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Wed Jul 15 16:00:27 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 16:00:27 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
Message-ID: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>

On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>
> Disclaimer: I've only read the short version, so if some of this is
> covered in the longer explanation, I apologise now.

Next time I won't put a short version ;)


>
> PEP 376 support has added a requirement for 3 additional methods to
> the existing 1 finder method in PEP 302. That's already a 300%
> increase in complexity. I'm against adding any further complexity to
> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
> *own* locator protocol and avoid any extra burden on PEP 302. I'm not
> sure implementers of PEP 302 importers will even provide the current
> PEP 376 extensions.
>
> I propose that before the current prototype is turned into a final
> (spec and) implementation, the PEP 302 extensions are extracted and
> documented as an independent protocol, purely part of PEP 376. (This
> *helps* implementers, as they can write support for, for example,
> eggs, without needing to modify the existing egg importer). I'll
> volunteer to do that work - but I won't start until the latest
> iteration of questions and discussions has settled down and PEP 376
> has achieved a stable form with the known issues addressed.

Sure that makes sense. I am all for having these 302 extensions
flipped on PEP 376
side, then think about the "locator" protocol.

I am lagging a bit in the discussions, I have 10 messages left to read or so,
but the known issues I've listed so far are about the RECORD file and
absolute paths,
I am waiting for PJE example on the syntax he proposed for prefixes,
on the docutils example.

> Of course, this is moving more and more towards saying that the design
> of setuptools, with its generic means for locating distributions, etc
> etc, is the right approach.
> We're reinventing the wheel here. But the
> problem is that too many people dislike setuptools as it stands for it
> to gain support.

I don't think it's about setuptools design. I think it's more likely
to be about the fact
that there's no way in Python to install two different versions of the
same distribution
without "hiding" one from each other, using setuptools, virtualenv or
zc.buildout.

"installing" a distribution in Python means that its activated
globally, whereas people
need it locally at the application level.

> My understanding is that the current set of PEPs were
> intended to be a stripped down, more generally acceptable subset of
> setuptools. Let's keep them that way (and omit the complexities of
> multi-version support).
>
> If you want setuptools, you know where to get it...

Sure, but let's not forget that the multiple-version issue is a global
issue OS packagers
also meet. (setuptools is not the problem) :

- application Foo uses docutils 0.4 and doesn't work with docutils 0.5
- application Bar uses docutils 0.5

if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
shipped with it.

So right now application developers must use tools to isolate their
dependencies from
the rest of the system, using virtualenv or zc.buildout, and ship the
dependencies with them.
So they are sure that their applications are not broken by the system.

This is not optimal of course, because you end up with several
occurences of the same
versions sometimes. And can be a nightmare for os packagers and maintainers.

And virtualenv and such tools are now required when you develop
applications (mid-size and large ones)
and the "good practice" is to avoid any installation of any
distributions in Python itself.

So basically "site-packages" is a distribution location that is
avoided by everyone because it doesn't
know how to handle multiple versions. If we had a multi-versions
support protocol, that would
help os packagers and application developers to be friends again imho ;)

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Wed Jul 15 16:03:08 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 16:03:08 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <4A5DACC2.5030609@voidspace.org.uk>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<4A5DACC2.5030609@voidspace.org.uk>
Message-ID: <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>

On Wed, Jul 15, 2009 at 12:17 PM, Michael
Foord<fuzzyman at voidspace.org.uk> wrote:
>> Disclaimer: I've only read the short version, so if some of this is
>> covered in the longer explanation, I apologise now.
>>
>> -1.
>>
>
> I agree. People with versioning issues *should* be using virtualenv.
>

Let's remove site-packages from Python then.

From peter.hanecak at alcatel-lucent.sk  Wed Jul 15 15:46:59 2009
From: peter.hanecak at alcatel-lucent.sk (Peter Hanecak)
Date: Wed, 15 Jul 2009 15:46:59 +0200
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds
	XML-RPC limits
Message-ID: <4A5DDDD3.4070900@alcatel-lucent.sk>

Hello,

when I'm trying to use 64-bit integer values with SimpleXMLRPCServer, 
I'm getting "OverflowError: int exceeds XML-RPC limits" error each time 
I use an integer with value greater or equal to 2^31.

I googled this:

	http://bugs.python.org/issue2985


So, my question is: In which Python release has been this fix distributed?

Thank you in advance for information.

Btw, I've made some dummy scripts to demonstrate the issue:

- server.py:
from SimpleXMLRPCServer import SimpleXMLRPCServer

def dummy(number):
   return number

server = SimpleXMLRPCServer(("localhost", 8000))
server.register_function(dummy, 'dummy')
server.serve_forever()

- client.py
import xmlrpclib

proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
print proxy.dummy(0x7FFFFFFF)
print proxy.dummy(0x80000000)


The output from client is:
localhost.localdomain - - [15/Jul/2009 15:24:12] "POST / HTTP/1.0" 200 -
2147483647
Traceback (most recent call last):
   File "./client.py", line 7, in <module>
     print proxy.dummy(0x80000000)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
     return self.__send(self.__name, args)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1483, in __request
     allow_none=self.__allow_none)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 1132, in dumps
     data = m.dumps(params)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 677, in dumps
     dump(v, write)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
     f(self, value, write)
   File "/usr/lib64/python2.6/xmlrpclib.py", line 710, in dump_int
     raise OverflowError, "int exceeds XML-RPC limits"
OverflowError: int exceeds XML-RPC limits


Have a nice day.

Peter

p.s.: I'm not subscribed to the list, se please keep me in CC when 
replying. Thank you.

-- 
Peter Hanecak

ePC Developer | Alcatel-Lucent
Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic
email: peter.hanecak at alcatel-lucent.com
phone: +421 (0)2 49264 857

From benjamin at python.org  Wed Jul 15 16:29:02 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 15 Jul 2009 09:29:02 -0500
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int
	exceeds XML-RPC limits
In-Reply-To: <4A5DDDD3.4070900@alcatel-lucent.sk>
References: <4A5DDDD3.4070900@alcatel-lucent.sk>
Message-ID: <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>

2009/7/15 Peter Hanecak <peter.hanecak at alcatel-lucent.sk>:
> So, my question is: In which Python release has been this fix distributed?

Python 2.6 and above.



-- 
Regards,
Benjamin

From rdmurray at bitdance.com  Wed Jul 15 16:39:44 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 15 Jul 2009 10:39:44 -0400 (EDT)
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int
 exceeds XML-RPC limits
In-Reply-To: <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>
References: <4A5DDDD3.4070900@alcatel-lucent.sk>
	<1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0907151037570.6306@kimball.webabinitio.net>

On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote:
> 2009/7/15 Peter Hanecak <peter.hanecak at alcatel-lucent.sk>:
>> So, my question is: In which Python release has been this fix distributed?
>
> Python 2.6 and above.

But it doesn't solve your problem, since the ticket says it only fixes
reading long ints, not writing them.

--David

From peter.hanecak at alcatel-lucent.sk  Wed Jul 15 16:42:20 2009
From: peter.hanecak at alcatel-lucent.sk (Peter Hanecak)
Date: Wed, 15 Jul 2009 16:42:20 +0200
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int
 exceeds XML-RPC limits
In-Reply-To: <Pine.LNX.4.64.0907151037570.6306@kimball.webabinitio.net>
References: <4A5DDDD3.4070900@alcatel-lucent.sk>
	<1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>
	<Pine.LNX.4.64.0907151037570.6306@kimball.webabinitio.net>
Message-ID: <4A5DEACC.5040502@alcatel-lucent.sk>

Hello,

thank you David and Benjamin for quick response.

So my subsequent question is: What can help me solve the "writing" part?

Sincerely

Peter

On 07/15/2009 04:39 PM, R. David Murray wrote:
> On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote:
>> 2009/7/15 Peter Hanecak <peter.hanecak at alcatel-lucent.sk>:
>>> So, my question is: In which Python release has been this fix
>>> distributed?
>>
>> Python 2.6 and above.
>
> But it doesn't solve your problem, since the ticket says it only fixes
> reading long ints, not writing them.
>
> --David

-- 
Peter Hanecak

ePC Developer | Alcatel-Lucent
Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic
email: peter.hanecak at alcatel-lucent.com
phone: +421 (0)2 49264 857

From ziade.tarek at gmail.com  Wed Jul 15 16:46:51 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 16:46:51 +0200
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int
	exceeds XML-RPC limits
In-Reply-To: <4A5DEACC.5040502@alcatel-lucent.sk>
References: <4A5DDDD3.4070900@alcatel-lucent.sk>
	<1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>
	<Pine.LNX.4.64.0907151037570.6306@kimball.webabinitio.net>
	<4A5DEACC.5040502@alcatel-lucent.sk>
Message-ID: <94bdd2610907150746v7b16021dk82c6a8331f9072b6@mail.gmail.com>

On Wed, Jul 15, 2009 at 4:42 PM, Peter
Hanecak<peter.hanecak at alcatel-lucent.sk> wrote:
> Hello,
>
> thank you David and Benjamin for quick response.
>
> So my subsequent question is: What can help me solve the "writing" part?

Use strings.

Send str(0x7FFFFFFF) from the client for example , and get back your
number on server side with a long(number)  call.

That's what I do.

From dirkjan at ochtman.nl  Wed Jul 15 16:46:59 2009
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Wed, 15 Jul 2009 16:46:59 +0200
Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int
	exceeds XML-RPC limits
In-Reply-To: <4A5DEACC.5040502@alcatel-lucent.sk>
References: <4A5DDDD3.4070900@alcatel-lucent.sk>
	<1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com>
	<Pine.LNX.4.64.0907151037570.6306@kimball.webabinitio.net>
	<4A5DEACC.5040502@alcatel-lucent.sk>
Message-ID: <ea2499da0907150746u422bf1b8jc29dbb0d60c83268@mail.gmail.com>

On Wed, Jul 15, 2009 at 16:42, Peter
Hanecak<peter.hanecak at alcatel-lucent.sk> wrote:
> So my subsequent question is: What can help me solve the "writing" part?

The XML-RPC protocol, as specified at [1], doesn't support integers
with more than 32 bits (in fact, the i4 alias can be used to make the
use of 4 bytes explicit). So, either don't use XML-RPC, or encode it
in some other type (e.g. a string or base64).

Cheers,

Dirkjan

[1] http://www.xmlrpc.com/spec

From p.f.moore at gmail.com  Wed Jul 15 17:14:30 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 16:14:30 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<4A5DACC2.5030609@voidspace.org.uk>
	<94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>
Message-ID: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>

2009/7/15 Tarek Ziad? <ziade.tarek at gmail.com>:
> On Wed, Jul 15, 2009 at 12:17 PM, Michael
> Foord<fuzzyman at voidspace.org.uk> wrote:
>>> Disclaimer: I've only read the short version, so if some of this is
>>> covered in the longer explanation, I apologise now.
>>>
>>> -1.
>>>
>>
>> I agree. People with versioning issues *should* be using virtualenv.
>>
>
> Let's remove site-packages from Python then.

If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then
maybe you're right.

For me, py2exe (and a clean virtual machine if I require an absolutely
pristine environment, I guess virtualenv would help here too) does
what I need for application packaging. But I'll freely admit that my
needs are minimal.

Bluntly, as Python stands, import and sys.path do not offer any core
support for multiple versions. Custom solutions can be built on top of
that - that's what setuptools does. But they are precisely that -
custom solutions, and should be supported as such, outside the core
(and stdlib).

If standard Python support for multi-version imports is required (it's
not by me, but I accept that some people want it), then it should be
designed in thoughout:

- how do I import version 0.7.1 of package foo, rather than 0.7.2?
- how do I use foo 0.8 at my interactive prompt, and import bar 0.2
which relies on foo 0.7.1?
- what happens if I import foo 2.0 (which requires baz 0.1) and bar
1.5 (which requires baz 0.2)?
- what does "import foo" (without a version number) mean? Is it
different if it's at the command line or in bar 0.5 (which explicitly
declares a dependency on foo 0.7 in its setup.py)? Does the answer to
that mean that imports always need to read dependency information?
- does your head hurt yet? I could go on...

Any PEP dealing with multi versions should address these issues. It's
a big area, and I have no interest in it myself, but I do have an
interest in avoiding partial solutions which only look at some of the
questions that might arise.

Look - I really, really don't mind if people use setuptools. Honest.
But I do mind if core python gets changed to support little bits of
what setuptools does, adding complexity to deal with issues that
setuptools handles, but without making it possible to avoid using
setuptools. Where's the benefit to anyone then?

Paul.

From p.f.moore at gmail.com  Wed Jul 15 17:21:12 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 16:21:12 +0100
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to
	runpy?
In-Reply-To: <4A5DDEEB.4040700@gmail.com>
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
	<loom.20090715T102404-716@post.gmane.org> <4A5DDEEB.4040700@gmail.com>
Message-ID: <79990c6b0907150821r6488759dr4cd49c382f2daf79@mail.gmail.com>

2009/7/15 Nick Coghlan <ncoghlan at gmail.com>:
> While runpy is the only client in the standard library for the
> get_filename() method, the method is still a PEP 302 extension. I
> documented the extension in the PEP as loaders are the only things
> reliably in a position to provide the filename details that runpy needs
> to set __file__ and sys.argv[0] correctly and until importlib came along
> PEP 302 itself was the only real documentation of that API.

Yes, sorry about that :-)

> Since importlib is now the "go-to" location for people that want to
> write their own PEP 302 importers and loaders, I would say that it is
> also the right place for the new ExecutionLoader ABC.

Agreed. (But I said that before...)

Paul.

From him at online.de  Wed Jul 15 17:16:41 2009
From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=)
Date: Wed, 15 Jul 2009 17:16:41 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
Message-ID: <4A5DF2D9.6050403@online.de>

Tarek Ziad? wrote:
> So basically "site-packages" is a distribution location that is
> avoided by everyone because it doesn't
> know how to handle multiple versions. 
I think you overrate the importance of having multiple versions of a 
package available
for the same python interpreter. If you have m different versions of n 
packages then
you could have n**m different combinations for an application so you need a
possiblilty to select one combination from n**m possible ones at application
startup time. Is this really worth it?

> If we had a multi-versions
> support protocol, that would
> help os packagers and application developers to be friends again imho ;)
>   
> Let's remove site-packages from Python then.
The _one_ site-packages folder stands for _one_ python interpreter. All 
the clever
efforts to provide a set of package versions at runtime to an 
application (that uses
the singleton python interpreter) do logically create a new python 
interpreter with
a site-packages folder that contains just the versions of the packages 
the application
needs, unfortunately by mucking with PYTHONPATH and <package>.pth, 
site.py etc
making it very difficult to understand what is happening for the joe 
average python developer.




From ziade.tarek at gmail.com  Wed Jul 15 17:51:13 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 17:51:13 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<4A5DACC2.5030609@voidspace.org.uk>
	<94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>
	<79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
Message-ID: <94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com>

On Wed, Jul 15, 2009 at 5:14 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/15 Tarek Ziad? <ziade.tarek at gmail.com>:
>> On Wed, Jul 15, 2009 at 12:17 PM, Michael
>> Foord<fuzzyman at voidspace.org.uk> wrote:
>>>> Disclaimer: I've only read the short version, so if some of this is
>>>> covered in the longer explanation, I apologise now.
>>>>
>>>> -1.
>>>>
>>>
>>> I agree. People with versioning issues *should* be using virtualenv.
>>>
>>
>> Let's remove site-packages from Python then.
>
> If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then
> maybe you're right.

They do offer a solution, but these solution are hard to maintain from
a OS packager point of view.

In any case I don't see any use case to have a "site-packages"
remaining in Python itself.

> If standard Python support for multi-version imports is required (it's
> not by me, but I accept that some people want it), then it should be
> designed in thoughout:
> [..]
>
> Any PEP dealing with multi versions should address these issues. It's
> a big area, and I have no interest in it myself, but I do have an
> interest in avoiding partial solutions which only look at some of the
> questions that might arise.
>
> Look - I really, really don't mind if people use setuptools. Honest.
> But I do mind if core python gets changed to support little bits of
> what setuptools does, adding complexity to deal with issues that
> setuptools handles, but without making it possible to avoid using
> setuptools. Where's the benefit to anyone then?

I totally agree. But I think that the boundary between what Python+stdlib
should provide feature-wise and what third party packages provides is
still fuzzy and should
be made clearer.

At some point, third party projects are trying hard to isolate dependencies
because they can't do it with the execution environment Python sets by
default when
you launch a script or an interpreter.

site.py loads site-packages and user site-packages at startup
basically, and you can
use PYTHONPATH / sys.path to add more, but that's partially shared by all apps.

To address this issued, a project like zc.buildout will create a local
repository of distributions, and will
change sys.path on the fly so the local repository is used.

virtualenv on its side creates an isolated Python installation for
your application,

So If the stdlib doesn't provide a standard protocol that goes further
than that,
people that have this need will continue to use zc.buildout or
virtualenv and install many times
the same libs on a system.

It's perfectly fine from an application developer PoV, but it make
site-packages obsoletes and
every application installed that way looks like a black box for os
packagers. They don't want that.

At the end, application developers have to care about the way their applications
dependencies are installed, were they should just give the list of
those dependencies,
and let any package manager project install them.

If these matters are not adressed by Python sdtlib, then we should
remove the loading of
site-packages at Python startup completely, and make it cristal clear
that it's not the core/stdlib
problem.

Otherwise we should think hard about how to let os packagers manage a
single repository
of Python distributions (and multiple versions) and how application
developers could use them from within
their applications. This protocol imho could be in the sdtlib even if
the implementation
is outside, but that's just me.

In any case, these issues can be postponed after PEP 376, because
that's a another (bigger) part of the puzzle.

Regards
Tarek

From cournape at gmail.com  Wed Jul 15 17:22:03 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 16 Jul 2009 00:22:03 +0900
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
Message-ID: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>

On Wed, Jul 15, 2009 at 11:00 PM, Tarek Ziad?<ziade.tarek at gmail.com> wrote:
> On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>>
>> Disclaimer: I've only read the short version, so if some of this is
>> covered in the longer explanation, I apologise now.
>
> Next time I won't put a short version ;)
>
>
>>
>> PEP 376 support has added a requirement for 3 additional methods to
>> the existing 1 finder method in PEP 302. That's already a 300%
>> increase in complexity. I'm against adding any further complexity to
>> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
>> *own* locator protocol and avoid any extra burden on PEP 302. I'm not
>> sure implementers of PEP 302 importers will even provide the current
>> PEP 376 extensions.
>>
>> I propose that before the current prototype is turned into a final
>> (spec and) implementation, the PEP 302 extensions are extracted and
>> documented as an independent protocol, purely part of PEP 376. (This
>> *helps* implementers, as they can write support for, for example,
>> eggs, without needing to modify the existing egg importer). I'll
>> volunteer to do that work - but I won't start until the latest
>> iteration of questions and discussions has settled down and PEP 376
>> has achieved a stable form with the known issues addressed.
>
> Sure that makes sense. I am all for having these 302 extensions
> flipped on PEP 376
> side, then think about the "locator" protocol.
>
> I am lagging a bit in the discussions, I have 10 messages left to read or so,
> but the known issues I've listed so far are about the RECORD file and
> absolute paths,
> I am waiting for PJE example on the syntax he proposed for prefixes,
> on the docutils example.
>
>> Of course, this is moving more and more towards saying that the design
>> of setuptools, with its generic means for locating distributions, etc
>> etc, is the right approach.
>> We're reinventing the wheel here. But the
>> problem is that too many people dislike setuptools as it stands for it
>> to gain support.
>
> I don't think it's about setuptools design. I think it's more likely
> to be about the fact
> that there's no way in Python to install two different versions of the
> same distribution
> without "hiding" one from each other, using setuptools, virtualenv or
> zc.buildout.
>
> "installing" a distribution in Python means that its activated
> globally, whereas people
> need it locally at the application level.
>
>> My understanding is that the current set of PEPs were
>> intended to be a stripped down, more generally acceptable subset of
>> setuptools. Let's keep them that way (and omit the complexities of
>> multi-version support).
>>
>> If you want setuptools, you know where to get it...
>
> Sure, but let's not forget that the multiple-version issue is a global
> issue OS packagers
> also meet. (setuptools is not the problem) :
>
> - application Foo uses docutils 0.4 and doesn't work with docutils 0.5
> - application Bar uses docutils 0.5
>
> if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
> shipped with it.

As was stated by Debian packagers on the distutils ML, the problem is
that docutils 0.5 breaks packages which work with docutils 0.4 in the
first place.

http://www.mail-archive.com/distutils-sig at python.org/msg05775.html

And current hacks to work around lack of explicit version handling for
module import is a maintenance burden:

http://www.mail-archive.com/distutils-sig at python.org/msg05742.html

setuptools has given the incentive to use versioning as a workaround
for API/ABI compatibility. That's the core problem, and most problems
brought by setuptools (sys.path and .pth hacks with the unreliability
which ensued) are consequences of this. I don't see how virtualenv
solves anything in that regard for deployment issues. I doubt using
things like virtualenv will make OS packagers happy.

David

From pje at telecommunity.com  Wed Jul 15 17:58:42 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 11:58:42 -0400
Subject: [Python-Dev] 2.6 object.__init__ & deling __new__
In-Reply-To: <4A5D8557.90703@canterbury.ac.nz>
References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com>
	<1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com>
	<20090715042200.946A13A410E@sparrow.telecommunity.com>
	<4A5D8557.90703@canterbury.ac.nz>
Message-ID: <20090715155538.A77C03A4099@sparrow.telecommunity.com>

At 07:29 PM 7/15/2009 +1200, Greg Ewing wrote:
>P.J. Eby wrote:
>
>>In effect, 2.6 forces you to have a common known base class *other* 
>>than 'object' in order to write co-operative classes.  :-(
>
>You have to do that anyway if you want to make cooperative
>calls to any method *other* that __init__.

I haven't found that to be an issue, actually, since there's usually 
a base where the "bottom" version of those methods live, and anything 
implementing those methods is going to inherit it via the mixin.  The 
problem is that for at least __init__ and __new__, 'object' *is* the 
"bottom" -- and there may be more than one family of mixins that need 
to call it.


From p.f.moore at gmail.com  Wed Jul 15 17:59:56 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 16:59:56 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
Message-ID: <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com>

2009/7/15 David Cournapeau <cournape at gmail.com>:
> As was stated by Debian packagers on the distutils ML, the problem is
> that docutils 0.5 breaks packages which work with docutils 0.4 in the
> first place.
>
> http://www.mail-archive.com/distutils-sig at python.org/msg05775.html
>
> And current hacks to work around lack of explicit version handling for
> module import is a maintenance burden:
>
> http://www.mail-archive.com/distutils-sig at python.org/msg05742.html
>
> setuptools has given the incentive to use versioning as a workaround
> for API/ABI compatibility. That's the core problem, and most problems
> brought by setuptools (sys.path and .pth hacks with the unreliability
> which ensued) are consequences of this. I don't see how virtualenv
> solves anything in that regard for deployment issues. I doubt using
> things like virtualenv will make OS packagers happy.

So, I think what you're saying is:

- The real issues is packages not maintaining backward compatibility (I agree)
- Setuptools is a workaround (I agree, at least it's *one* workaround)
- Virtualenv isn't a workaround (I don't know virtualenv, I'll take
your word for it)

Three points:

- When building *applications*, bundling everything (py2exe-style) is
an alternative workaround - universal on Windows, apparently uncommon
on Unix/Linux.
- When managing multiple packages in a "toolkit" style interactive
Python installation, I'm not aware of a good solution (other than
avoiding core that hits this issue in the first place).
- I do not believe that it's clear that sanctioning the setuptools
workaround as the "right" approach by building it into the Python
core/stdlib is the right thing to do.

Paul.

From ziade.tarek at gmail.com  Wed Jul 15 18:00:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 18:00:48 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <4A5DF2D9.6050403@online.de>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<4A5DF2D9.6050403@online.de>
Message-ID: <94bdd2610907150900j62a249fwcf88a188d71c5a51@mail.gmail.com>

On Wed, Jul 15, 2009 at 5:16 PM, Joachim K?nig<him at online.de> wrote:
> Tarek Ziad? wrote:
>>
>> So basically "site-packages" is a distribution location that is
>> avoided by everyone because it doesn't
>> know how to handle multiple versions.
>
> I think you overrate the importance of having multiple versions of a package
> available
> for the same python interpreter. If you have m different versions of n
> packages then
> you could have n**m different combinations for an application so you need a
> possiblilty to select one combination from n**m possible ones at application
> startup time. Is this really worth it?

When you create an application that uses several libs, and when you
create your distribution,
you end up pinning a version for each one of your dependency to avoid
any problems.

While it's workable in a small application to list the dependencies in
a text file, and let your users
install them manually, it's impossible in bigger applications.

Applications based on Zope for example have **hundreds** of dependencies
you need to have installed, in very specific versions.

If you install a Zope 2 app on your system, and a Zope 3 one, you have
good chances of
breaking them because the "zope.schema" distribution is incompatible.

So either each application ships its own collection of dependencies
and ignore site-packages (that's what zope-based
applications does with zc.buildout), either they have a way to pick
the right version of each package.


>
>> If we had a multi-versions
>> support protocol, that would
>> help os packagers and application developers to be friends again imho ;)
>> ?Let's remove site-packages from Python then.
>
> The _one_ site-packages folder stands for _one_ python interpreter. All the
> clever
> efforts to provide a set of package versions at runtime to an application
> (that uses
> the singleton python interpreter) do logically create a new python
> interpreter with
> a site-packages folder that contains just the versions of the packages the
> application
> needs, unfortunately by mucking with PYTHONPATH and <package>.pth, site.py
> etc
> making it very difficult to understand what is happening for the joe average
> python developer.

That's what we have nowadays. python packages installed in different places,
and scripts tweaking the path to launch an application with them.

From pje at telecommunity.com  Wed Jul 15 18:21:32 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 12:21:32 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.co
 m>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
Message-ID: <20090715161829.E45BE3A4116@sparrow.telecommunity.com>

At 11:10 AM 7/15/2009 +0100, Paul Moore wrote:
>I propose that before the current prototype is turned into a final
>(spec and) implementation, the PEP 302 extensions are extracted and
>documented as an independent protocol, purely part of PEP 376. (This
>*helps* implementers, as they can write support for, for example,
>eggs, without needing to modify the existing egg importer).

Btw, this is why setuptools chose to implement these things as 
adapters or generic functions in pkg_resources (and in the vestigial 
bits that were added to python 2.5's pkgutil).

So as you can see, trying to solve these particular problems tends to 
lead to reinventing setuptools or at least portions thereof.  ;-)



>But the
>problem is that too many people dislike setuptools as it stands for it
>to gain support. My understanding is that the current set of PEPs were
>intended to be a stripped down, more generally acceptable subset of
>setuptools. Let's keep them that way (and omit the complexities of
>multi-version support).

Even without multi-version support, the parts of PEP 376 that aren't 
about uninstallation are still reinventing chunks of 
pkg_resources.  Had pkg_resources been in the stdlib a couple years 
back (note that bugs and changes in it are still quite rare), the PEP 
376 bits for pkgutil could have been focused strictly on 
uninstallation, and just reused pkg_resources API for finding 
distributions, reading metadata, getting version info, etc. etc.

All that stuff is extremely stable code, very widely used for a very 
long time.  If politics is the only thing keeping it from being used, 
then that's a pretty sad statement about the community, given that 
pkg_resources is not to blame for 99% of what people complain about 
in relation to setuptools.  All pkg_resources does is find stuff (on 
request) and maybe add it to sys.path (on request), pull out data and 
metadata from distributions (on request), or set up namespace 
packages (on request).

So if politics demands that it be rejected by association with 
"setuptools", then just search-and-replace the API, PEP 8-ify it, 
call it something different, and lie to everyone about where it came 
from.  I won't tell if you won't.  ;-)


From pje at telecommunity.com  Wed Jul 15 18:27:11 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 12:27:11 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.co
 m>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<4A5DACC2.5030609@voidspace.org.uk>
	<94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>
	<79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
Message-ID: <20090715162407.AD8723A4116@sparrow.telecommunity.com>

At 04:14 PM 7/15/2009 +0100, Paul Moore wrote:
>Look - I really, really don't mind if people use setuptools. Honest.
>But I do mind if core python gets changed to support little bits of
>what setuptools does, adding complexity to deal with issues that
>setuptools handles, but without making it possible to avoid using
>setuptools.

I don't see how that's being proposed.  Nothing in PEP 376 requires 
setuptools or any portion thereof, AFAICT.


From pje at telecommunity.com  Wed Jul 15 18:30:39 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 12:30:39 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <4A5DF2D9.6050403@online.de>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<4A5DF2D9.6050403@online.de>
Message-ID: <20090715162736.B54D33A4116@sparrow.telecommunity.com>

At 05:16 PM 7/15/2009 +0200, Joachim K?nig wrote:
>f you have m different versions of n packages then
>you could have n**m different combinations for an application so you need a
>possiblilty to select one combination from n**m possible ones at application
>startup time. Is this really worth it?

Obviously yes, as neither buildout nor setuptools would exist otherwise.  ;-)
Nor would Fedora be packaging certain library versions as eggs 
specifically to get certain multi-version scenarios to work.

The specific solutions for handling n*m problems aren't fantastic, 
but they are clearly needed.

(Buildout, btw, actually hardwires the n*m choice at install time, 
which is probably better for production situations than setuptools' 
dynamic approach.)


From pje at telecommunity.com  Wed Jul 15 18:34:00 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 12:34:00 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com
 >
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
	<79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com>
Message-ID: <20090715163056.B34B63A4116@sparrow.telecommunity.com>

At 04:59 PM 7/15/2009 +0100, Paul Moore wrote:
>- Virtualenv isn't a workaround (I don't know virtualenv, I'll take
>your word for it)

It's not one for system package maintainers because it would 
effectively be managing multiple instances of 'python'.  Really not a 
suitable solution.


>- I do not believe that it's clear that sanctioning the setuptools
>workaround as the "right" approach by building it into the Python
>core/stdlib is the right thing to do.

I still don't understand how we're doing that.


From rdmurray at bitdance.com  Wed Jul 15 18:49:57 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 15 Jul 2009 12:49:57 -0400 (EDT)
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<4A5DACC2.5030609@voidspace.org.uk>
	<94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>
	<79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0907151235210.6306@kimball.webabinitio.net>

On Wed, 15 Jul 2009 at 16:14, Paul Moore wrote:
> Bluntly, as Python stands, import and sys.path do not offer any core
> support for multiple versions. Custom solutions can be built on top of
> that - that's what setuptools does. But they are precisely that -
> custom solutions, and should be supported as such, outside the core
> (and stdlib).
>
> If standard Python support for multi-version imports is required (it's
> not by me, but I accept that some people want it), then it should be
> designed in thoughout:

Isn't this problem directly analogous to the '.so version' (*) problem?
Can we learn anything from the solution to that problem?  Does not the
fact that a (standard) solution to that problem was required indicate
that a similar solution needs to be provided by core Python?  (And yes,
it's out of scope for PEP 376).

--David

(*) or, for our Windows users, DLL Hell and the Side By Side Component
Sharing solution...

From ziade.tarek at gmail.com  Wed Jul 15 19:07:34 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 15 Jul 2009 19:07:34 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <20090715161829.E45BE3A4116@sparrow.telecommunity.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<20090715161829.E45BE3A4116@sparrow.telecommunity.com>
Message-ID: <94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com>

2009/7/15 P.J. Eby <pje at telecommunity.com>:
[...]
> So if politics demands that it be rejected by association with "setuptools",
> then just search-and-replace the API, PEP 8-ify it, call it something
> different, and lie to everyone about where it came from. ?I won't tell if
> you won't. ?;-)

While setuptools did address some issues, we are now adressing in PEP 376,
it does a *lot* of other things and it's impossible to extract the
features we would
want to re-use. e.g. what I would call the good bits.

The prototype code we are building on bitbucket is clear, easy to
understand, read and adopt and already PEP8-fied (which is required
for sdtlib inclusion)

Reading and understanding pkg_resources requires a lot of time.

A side note on setuptools, I don't mean to be rude here or to tell you
what you should do,
really, but:

  While I am very glad that you are part of PEP 376 discussions, and
that you helped
  on almost every section of it, I really don't understand why you
keep on pushing setuptools
  *code* here and at distutils-SIG since you don't maintain it anymore
(the last commit is > 9 months)

  Part of the rejection of setuptools is because of that and because you don't
  bless anyone to maintain the project code base, or do releases,
neither to communicate clearly
  on what's its roadmap.

Regards
Tarek

From SridharR at activestate.com  Wed Jul 15 19:38:21 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 15 Jul 2009 10:38:21 -0700
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
Message-ID: <op.uw4dx7ttbrrvlq@double>

On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

>>> get_installed_files(local=False) -> iterator of (path, md5, size)
>>
>> Will this also return the directories /created/ during the installation?
>> For example, will it also contain the entry "docutils" .. along with
>> "docutils/__init__.py"?
>
> I don't think it's necessary to add "docutils" if
> "docutils/__init__.py" is present
>
> But for empty directories added during installation we should add the I  
> guess.
>
> So, I'll add a note.

It seems that you overlooked the below paragraph.

>> If not, how is the installer (pip, pypm, etc..) supposed to know which
>> directories to remove (docutils/) and which directories not to remove
>> (site-packages/, bin/, etc..)?

Quoting from the PEP:

[quote]'(...)uninstall uses the APIs described earlier and remove all  
unique files, as long as their hash didn't change. Then it removes empty  
directories left behind.'[endquote]

Let's assume that site-packages/ contained only one package 'Foo'. Will  
uninstall('Foo') remove the site-packages/ directory just because it  
turned out to be empty after removing 'Foo'? To explain, let's assume the  
RECORD of 'Foo' contains:

$ cat RECORD
Foo/__init__.py
Foo/bar/__init__.py
Foo/bar/test.py

and according to what you wrote in the PEP ("it removes empty directories  
left behind"):

$ python -m distutils.util.uninstall Foo
rm /.../site-packages/Foo/__init__.py
rm /.../site-packages/Foo/bar/__init__.py
rm /.../site-packages/Foo/bar/test.py
rm empty dir /.../site-packages/Foo/bar
rm empty dir /.../site-packages/Foo/
rm empty dir /.../site-packages/        # !!!!!

it also remove the site-packages directory!

Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do  
you see my point?

> btw: is PyPM a public project ?

If by 'public', you meant open source, then no.

-srid

From p.f.moore at gmail.com  Wed Jul 15 19:40:28 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 18:40:28 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <20090715161829.E45BE3A4116@sparrow.telecommunity.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<20090715161829.E45BE3A4116@sparrow.telecommunity.com>
Message-ID: <79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.com>

2009/7/15 P.J. Eby <pje at telecommunity.com>:
> At 11:10 AM 7/15/2009 +0100, Paul Moore wrote:
>>
>> I propose that before the current prototype is turned into a final
>> (spec and) implementation, the PEP 302 extensions are extracted and
>> documented as an independent protocol, purely part of PEP 376. (This
>> *helps* implementers, as they can write support for, for example,
>> eggs, without needing to modify the existing egg importer).
>
> Btw, this is why setuptools chose to implement these things as adapters or
> generic functions in pkg_resources (and in the vestigial bits that were
> added to python 2.5's pkgutil).
>
> So as you can see, trying to solve these particular problems tends to lead
> to reinventing setuptools or at least portions thereof. ?;-)

Yes, that's the impression I'm getting :-)

And yet you seem resistant to reading any message into the fact that
people find it easier to solve the problems from scratch than to
extract the relevant parts of the existing code. To be explicit - it's
too flipping hard to identify, extract and tidy up the appropriate
bits of the setuptools code! (Disclaimer: I haven't looked at the code
for some time - if the style has changed utterly in the last 2 years
or so, so that it no longer resembles the style of the pkgutil code
you contributed to the stdlib, I'll retract that criticism of the
setuptools codebase).

>> But the
>> problem is that too many people dislike setuptools as it stands for it
>> to gain support. My understanding is that the current set of PEPs were
>> intended to be a stripped down, more generally acceptable subset of
>> setuptools. Let's keep them that way (and omit the complexities of
>> multi-version support).
>
> Even without multi-version support, the parts of PEP 376 that aren't about
> uninstallation are still reinventing chunks of pkg_resources. ?Had
> pkg_resources been in the stdlib a couple years back (note that bugs and
> changes in it are still quite rare), the PEP 376 bits for pkgutil could have
> been focused strictly on uninstallation, and just reused pkg_resources API
> for finding distributions, reading metadata, getting version info, etc. etc.
>
> All that stuff is extremely stable code, very widely used for a very long
> time. ?If politics is the only thing keeping it from being used, then that's
> a pretty sad statement about the community, given that pkg_resources is not
> to blame for 99% of what people complain about in relation to setuptools.
> ?All pkg_resources does is find stuff (on request) and maybe add it to
> sys.path (on request), pull out data and metadata from distributions (on
> request), or set up namespace packages (on request).

I don't think it's quite "politics" as you describe it. Maybe
"marketing", if you want a simple tag for it.

The problem is that the setuptools "brand name" is associated with a
lot of baggage, that maybe it doesn't deserve, but equally no-one
seems interested in addressing. It's certainly not a technical
argument at this point. And yet you counter criticisms with technical
responses. But if I recall the history correctly, one of the initial
goals of setuptools was to become a de-facto standard precisely by
marketing and filling the ecological niche - a goal it has achieved
fantastically well. The fact that in doing so it alienated certain
groups is a more or less inevitable consequence.

My argument - which you may well disagree with - is that the
"setuptools ecosystem" (by which I mean
setuptools/easy_install/pkg_resources/eggs and the various subsystems
around these) is documented in a way which is:

- confusing to the casual user
- not at all modular (ie, it's impossible to easily pick out individual "bits")
- full of new concepts, so it's hard to find an easy place to start from

There's not much technically at issue here, it's all about presentation.

> So if politics demands that it be rejected by association with "setuptools",
> then just search-and-replace the API, PEP 8-ify it, call it something
> different, and lie to everyone about where it came from. ?I won't tell if
> you won't. ?;-)

Unless the documentation and code is cleaned up, it'll still be
obvious from the style where it came from - that's the problem. Who's
going to maintain the code within the standard library, and are they
happy with the style of the existing code - even with the serial
numbers filed off?

And of course, someone has to do the clean-up. It seems to me that the
fact that people are more inclined to reinvent the code than to try to
understand the existing codebase and pick out the relevant bits, says
something important about how easy it would be to maintain the
existing code within the Python core...

Paul.

From tjreedy at udel.edu  Wed Jul 15 19:43:25 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 15 Jul 2009 13:43:25 -0400
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
In-Reply-To: <94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>	<4A5DACC2.5030609@voidspace.org.uk>	<94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com>	<79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com>
	<94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com>
Message-ID: <h3l4fp$3fe$2@ger.gmane.org>

Tarek Ziad? wrote:

> In any case I don't see any use case to have a "site-packages"
> remaining in Python itself.

I have and am using it.  Where else would you have me put library 
packages meant to be accessible by any Python program?

Terry Jan Reedy


From SridharR at activestate.com  Wed Jul 15 19:52:09 2009
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Wed, 15 Jul 2009 10:52:09 -0700
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
Message-ID: <op.uw4ek7eubrrvlq@double>

On Wed, 15 Jul 2009 08:22:03 -0700, David Cournapeau <cournape at gmail.com>  
wrote:

>> if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
>> shipped with it.
> As was stated by Debian packagers on the distutils ML, the problem is
> that docutils 0.5 breaks packages which work with docutils 0.4 in the
> first place.

Thus I am -1 on multi-version support .. and would rather have the python  
developers make their packages backward compatible just like what Armin  
did with Jinja and Jinja2. Debian at the moment has only one package  
"python-docutils" which is 0.5. How is a debian application supposed to  
depend upon 0.4? With Jinja, there is no problem .. there are  
'python-jinja' and 'python-jinja2'. Similarly, CherryPy should have gone  
with renaming their package names to CherryPy2 and CherryPy3.

-srid


PS: Quoting from a fellow developer:

> > [...] it sounds like CherryPy 3.x is really an incompatible
> > module that just kept the old name. That is rather unfortunate, but can
> > sometimes happen.
>  I have never seen a Python package changing its name (when significantly
> changing API, design, etc..). The only exception that comes to my mind
> is Jinja2 (renamed from 'Jinja'):
>  [Armin] (...) Because we love backwards compatibility these changes will
> go into a package called "jinja2"
> <http://lucumr.pocoo.org/2008/4/13/jinja2-making-things-awesome>

Well, congrats to the Jinja team then!  The others will eventually learn...
Mixing incompatible APIs in a single namespace and using a non-standardized
version numbering system to keep things apart creates a world of pain!

> The challenge however is in compartmentalizing versions according to
> their major release numbers. Versioning in the Python world is already a
> mess which we are beginning to sort out:
> http://wiki.python.org/moin/Distutils/VersionComparison (PyPM relies on
> this standard and the ongoing implementation - verlib.py)

How embarrassing for a cult that prides itself on having only one way for
everything they do...   CPAN versions numbers are just as much if not
more of a mess, but at least you can argue that it is the price for there
being "more than one way to do it".

From pje at telecommunity.com  Wed Jul 15 20:12:44 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 14:12:44 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com
 >
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<20090715161829.E45BE3A4116@sparrow.telecommunity.com>
	<94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com>
Message-ID: <20090715180942.407DB3A403A@sparrow.telecommunity.com>

At 07:07 PM 7/15/2009 +0200, Tarek Ziad? wrote:
>   Part of the rejection of setuptools is because of that and 
> because you don't
>   bless anyone to maintain the project code base, or do releases,
>neither to communicate clearly
>   on what's its roadmap.

Jim Fulton and Ian Bicking have been officially blessed for *years*, 
and I've posted plenty about what's on its roadmap.


>I really don't understand why you keep on pushing setuptools *code*

pkg_resources is not setuptools; its stability and revision history 
are very different, as it addresses different problems.


From pje at telecommunity.com  Wed Jul 15 20:20:10 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 15 Jul 2009 14:20:10 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.co
 m>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<20090715161829.E45BE3A4116@sparrow.telecommunity.com>
	<79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.com>
Message-ID: <20090715181705.E22823A403A@sparrow.telecommunity.com>

At 06:40 PM 7/15/2009 +0100, Paul Moore wrote:
>And of course, someone has to do the clean-up. It seems to me that the
>fact that people are more inclined to reinvent the code than to try to
>understand the existing codebase and pick out the relevant bits, says
>something important about how easy it would be to maintain the
>existing code within the Python core...

That's normal for any code that contains "legacy" issues, which is 
why people always prefer rewriting code they don't understand: it's 
more fun to write than to read.  However, as Joel Spolsky has well 
explained, rewriting such code inevitably means that you must 
re-learn the lessons that were learned by the original author.

It seems like a waste, but then, I suppose those lessons must be 
relearned *some* way.  I just think it'd be better if, having 
re-learned most of the lessons by trying to rewrite, one could then 
go back and learn the rest from the code.  ;-)

That having been said, it's obviously a dead parrot... one which I 
will now cease attempting to revive.


From chrism at plope.com  Wed Jul 15 19:47:35 2009
From: chrism at plope.com (Chris McDonough)
Date: Wed, 15 Jul 2009 13:47:35 -0400
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <20090715163056.B34B63A4116@sparrow.telecommunity.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>	<79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com>
	<20090715163056.B34B63A4116@sparrow.telecommunity.com>
Message-ID: <4A5E1637.30109@plope.com>

I've been trying to follow this discussion now for weeks. The signal to noise 
ratio is pretty low.

I'd love to have an stdlib solution for distribution packaging and installation. 
  But I think we might as well pack it up and go home if the folks whom are 
contributing to the discussion "recreationally" (whom are not themselves 
implementers and potential implementers or spec writers or potential spec 
writers of packaging systems) continue to chime in on *every single issue*, 
contributing only stop energy.  It's just completely pointless.

- C


On 7/15/09 12:34 PM, P.J. Eby wrote:
> At 04:59 PM 7/15/2009 +0100, Paul Moore wrote:
>> - Virtualenv isn't a workaround (I don't know virtualenv, I'll take
>> your word for it)
>
> It's not one for system package maintainers because it would effectively
> be managing multiple instances of 'python'. Really not a suitable solution.
>
>
>> - I do not believe that it's clear that sanctioning the setuptools
>> workaround as the "right" approach by building it into the Python
>> core/stdlib is the right thing to do.
>
> I still don't understand how we're doing that.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/lists%40plope.com
>


From brett at python.org  Wed Jul 15 20:38:22 2009
From: brett at python.org (Brett Cannon)
Date: Wed, 15 Jul 2009 11:38:22 -0700
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to
	runpy?
In-Reply-To: <4A5DDEEB.4040700@gmail.com>
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com> 
	<loom.20090715T102404-716@post.gmane.org> <4A5DDEEB.4040700@gmail.com>
Message-ID: <bbaeab100907151138n7b9686beyf8e0ad5e202e7fa6@mail.gmail.com>

On Wed, Jul 15, 2009 at 06:51, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Antoine Pitrou wrote:
> > I am not sure when this discussion started. Are you replying to a 3
> month-old
> > message of yours? :)
>
> That depends on how you define the beginning of the discussion...
>

Especially since I was offlist three months ago. =)


>
> However, the fact that importlib doesn't implement the comparatively
> recent get_filename() optional extension documented in PEP 302 came up
> in one of the PEP 376 threads within the last week or so.
>

I have been ignoring the PEP 372 discussions since I came into it late and I
don't deal with packaging enough to want to influence the discussion, so I
missed that point being made.


>
> > In any case, keeping all import-related ABCs in a single place sounds
> like a
> > good idea.
> > Alternatively, if the method is runpy specific, it doesn't have its place
> in an
> > ABC, does it?
>
> While runpy is the only client in the standard library for the
> get_filename() method, the method is still a PEP 302 extension. I
> documented the extension in the PEP as loaders are the only things
> reliably in a position to provide the filename details that runpy needs
> to set __file__ and sys.argv[0] correctly and until importlib came along
> PEP 302 itself was the only real documentation of that API.
>
> Since importlib is now the "go-to" location for people that want to
> write their own PEP 302 importers and loaders, I would say that it is
> also the right place for the new ExecutionLoader ABC.


OK. I will add an ExecutionLoader ABC to importlib that simply subclasses
InspectLoader and adds the get_filename() abstract method. Thanks to
everyone for their feedback.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090715/03e686ad/attachment.htm>

From martin at v.loewis.de  Wed Jul 15 21:26:38 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 15 Jul 2009 21:26:38 +0200
Subject: [Python-Dev] Mercurial: tag generation incorrect
In-Reply-To: <4A581DCE.4050307@gmx.net>
References: <4A57C5B5.5070000@v.loewis.de> <4A581DCE.4050307@gmx.net>
Message-ID: <4A5E2D6E.6040200@v.loewis.de>

Hagen F?rstenau wrote:
>> be32850b093f is listed
>> as having a child revision, 52b0a279fec6, and ISTM that *this*
>> should be the revision that got tagged.
> 
> I think the tag is correct. Note that the concept of tagging is
> different in Mercurial, where a tag can only refer to a revision
> previous to the one where it is inserted in .hgtags. If I understand
> correctly, all relevant tagging revisions from SVN are replaced by
> Mercurial revisions setting tags, which then refer to their immediate
> predecessors.

Ah, ok. Thanks for the explanation.

Regards,
Martin

From ncoghlan at gmail.com  Wed Jul 15 23:56:46 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 16 Jul 2009 07:56:46 +1000
Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to
	runpy?
In-Reply-To: <bbaeab100907151138n7b9686beyf8e0ad5e202e7fa6@mail.gmail.com>
References: <bbaeab100907142126i63170a83ibe2730c7ef62cffa@mail.gmail.com>
	<loom.20090715T102404-716@post.gmane.org>
	<4A5DDEEB.4040700@gmail.com>
	<bbaeab100907151138n7b9686beyf8e0ad5e202e7fa6@mail.gmail.com>
Message-ID: <4A5E509E.4010900@gmail.com>

Brett Cannon wrote:
> On Wed, Jul 15, 2009 at 06:51, Nick Coghlan <ncoghlan at gmail.com
> <mailto:ncoghlan at gmail.com>> wrote:
>     However, the fact that importlib doesn't implement the comparatively
>     recent get_filename() optional extension documented in PEP 302 came up
>     in one of the PEP 376 threads within the last week or so.
> 
> I have been ignoring the PEP 372 discussions since I came into it late
> and I don't deal with packaging enough to want to influence the
> discussion, so I missed that point being made.

Since I'm pretty sure you were the one that mentioned it, it's possible
I just misremembered which thread it came up in. The various discussions
can blur together sometimes!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From p.f.moore at gmail.com  Thu Jul 16 00:27:00 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 15 Jul 2009 23:27:00 +0100
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
In-Reply-To: <4A5E1637.30109@plope.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>
	<79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com>
	<20090715163056.B34B63A4116@sparrow.telecommunity.com>
	<4A5E1637.30109@plope.com>
Message-ID: <79990c6b0907151527k1d8b8889j9adc7f4b1edc4e57@mail.gmail.com>

2009/7/15 Chris McDonough <chrism at plope.com>:
> I've been trying to follow this discussion now for weeks. The signal to
> noise ratio is pretty low.

I agree :-(

> I'd love to have an stdlib solution for distribution packaging and
> installation. ?But I think we might as well pack it up and go home if the
> folks whom are contributing to the discussion "recreationally" (whom are not
> themselves implementers and potential implementers or spec writers or
> potential spec writers of packaging systems) continue to chime in on *every
> single issue*, contributing only stop energy. ?It's just completely
> pointless.

I thought that as someone who is writing code for Tarek's PEP 376
prototype implementation, I counted as offering useful input. Thinking
further on what I've said, I'm no longer sure that's true.

I do think that at *some* point, the feedback from the non-packagers,
negative or irrelevant though it might be, needs to be requested and
considered. Maybe now isn't the right time, but it needs to be done at
some point.

Until then, I'll try to avoid adding to the noise.

Paul.

From david.lyon at preisshare.net  Thu Jul 16 01:39:40 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 15 Jul 2009 19:39:40 -0400
Subject: [Python-Dev] PEP 376 - from pythonpkgmgr point of view
In-Reply-To: <4A5E1637.30109@plope.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>	<5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com>	<79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com>
	<20090715163056.B34B63A4116@sparrow.telecommunity.com>
	<4A5E1637.30109@plope.com>
Message-ID: <7cc3a3a3ff2f5a17c81060e6155c88c2@preisshare.net>

On Wed, 15 Jul 2009 13:47:35 -0400, Chris McDonough <chrism at plope.com>
wrote:
> I've been trying to follow this discussion now for weeks. The signal to
> noise ratio is pretty low.

I'm -1 on that..

As a relative newcomer to python packaging I'm finding all these
discussions very informative. :-)

> I'd love to have an stdlib solution for distribution packaging and
> installation.

That's right.

I'd love to find out what the process is for submitting my pythonpkgmgr
project for consideration into the stdlib.

As it fills a huge gap that you've just identified that currently exists 
within the python that is being shipped today. That is, allowing users to 
easily install packages from pypi.

>   But I think we might as well pack it up and go home if the folks whom
are
> contributing to the discussion "recreationally" (whom are not themselves 
> implementers and potential implementers or spec writers or potential spec

> writers of packaging systems) continue to chime in on *every single
issue*,
> 
> contributing only stop energy.  It's just completely pointless.

We can all work in different ways..

I'm not writing PEPS and don't care too much about how the internals of
a package work. As long as there are APIs for such stuff I'm happy.

What I'm working on is to try to improve the user experience which
isn't so handled so easily by a design by commitee process. 

Packaging and Distutils, need to be looked at wholistically as
well. It's not just about the api's, but how well they work.

It's not noise.... it's just the process....

ps:

(http://peak.telecommunity.com/DevCenter/EasyInstall)

"Easy Install
...
Please share your experiences with us! If you encounter difficulty
installing a package, please contact us via the distutils mailing list.
(Note: please DO NOT send private email directly to the author of
setuptools; it will be discarded. "

So what are we to do if we find issues with setuptools other than to post
to distutils?

David



From ziade.tarek at gmail.com  Thu Jul 16 02:43:50 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 16 Jul 2009 02:43:50 +0200
Subject: [Python-Dev] PEP 376 - from PyPM's point of view
In-Reply-To: <op.uw4dx7ttbrrvlq@double>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<op.uw4dx7ttbrrvlq@double>
Message-ID: <94bdd2610907151743m619f91b9n9cc1dc31a86d2729@mail.gmail.com>

2009/7/15 Sridhar Ratnakumar <SridharR at activestate.com>:
> On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziad? <ziade.tarek at gmail.com>
> wrote:
>
>>>> get_installed_files(local=False) -> iterator of (path, md5, size)
>>>
>>> Will this also return the directories /created/ during the installation?
>>> For example, will it also contain the entry "docutils" .. along with
>>> "docutils/__init__.py"?
>>
>> I don't think it's necessary to add "docutils" if
>> "docutils/__init__.py" is present
>>
>> But for empty directories added during installation we should add the I
>> guess.
>>
>> So, I'll add a note.
>
> It seems that you overlooked the below paragraph.
>
>>> If not, how is the installer (pip, pypm, etc..) supposed to know which
>>> directories to remove (docutils/) and which directories not to remove
>>> (site-packages/, bin/, etc..)?
>
> Quoting from the PEP:
>
> [quote]'(...)uninstall uses the APIs described earlier and remove all unique
> files, as long as their hash didn't change. Then it removes empty
> directories left behind.'[endquote]
>
> Let's assume that site-packages/ contained only one package 'Foo'. Will
> uninstall('Foo') remove the site-packages/ directory just because it turned
> out to be empty after removing 'Foo'? To explain, let's assume the RECORD of
> 'Foo' contains:
>
> $ cat RECORD
> Foo/__init__.py
> Foo/bar/__init__.py
> Foo/bar/test.py
>
> and according to what you wrote in the PEP ("it removes empty directories
> left behind"):
>
> $ python -m distutils.util.uninstall Foo
> rm /.../site-packages/Foo/__init__.py
> rm /.../site-packages/Foo/bar/__init__.py
> rm /.../site-packages/Foo/bar/test.py
> rm empty dir /.../site-packages/Foo/bar
> rm empty dir /.../site-packages/Foo/
> rm empty dir /.../site-packages/ ? ? ? ?# !!!!!
>
> it also remove the site-packages directory!
>
> Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do
> you see my point?

I didn't mean that of course. While we can avoid your example for the
code, by removing only
packages that are fully emptied, and are alongside the egg-info directory,
we might not be able to do it properly for the data.

So let's add the directories as well

From him at online.de  Thu Jul 16 14:26:00 2009
From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=)
Date: Thu, 16 Jul 2009 14:26:00 +0200
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <20090715162736.B54D33A4116@sparrow.telecommunity.com>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>
	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>
	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>
	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>
	<4A5DF2D9.6050403@online.de>
	<20090715162736.B54D33A4116@sparrow.telecommunity.com>
Message-ID: <4A5F1C58.3030003@online.de>

P.J. Eby wrote:
> At 05:16 PM 7/15/2009 +0200, Joachim K?nig wrote:
>> f you have m different versions of n packages then
>> you could have n**m different combinations for an application so you 
>> need a
>> possiblilty to select one combination from n**m possible ones at 
>> application
>> startup time. Is this really worth it?
>
> Obviously yes, as neither buildout nor setuptools would exist 
> otherwise.  ;-)
> Nor would Fedora be packaging certain library versions as eggs 
> specifically to get certain multi-version scenarios to work.
>
> The specific solutions for handling n*m problems aren't fantastic, but 
> they are clearly needed.
I still do not see the need.

IMO the whole obfuscation comes from fact that all versions of all 
packages are installed into
one location where python automaticallly looks for packages and then 
with a lot of magic the
packages are hidden from the interpreter and only specific requested 
versions are made "visible"
to the interpreter at runtime.

Why do the package have to be installed there at the first place?

For an application it would be enough to have an additional directory on 
its PYTHONPATH where
the packages required for this application would be installed. So a 
package could be installed either
to the common directory ("site-packages") or an application specific 
directory (e.g. something like
"app-packages/<appname>/"). This approach has been used by Zope2 with 
its "private" lib/python
directory for years.

So one would have to set up the application specific packages before 
running the application, but the
whole clutter of uncounted versions of the same package in one directory 
could go away. The
"drawback" of this approach would be, that the same version of a package 
would have to be installed
multiple times if needed by different applications.





From thomas at python.org  Thu Jul 16 22:08:59 2009
From: thomas at python.org (Thomas Wouters)
Date: Thu, 16 Jul 2009 22:08:59 +0200
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <446CB69E.4090609@v.loewis.de>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>
	<ca471dc20605112148s8414d85j6c14bf2ed622cccd@mail.gmail.com>
	<44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de>
	<44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>
	<446CB69E.4090609@v.loewis.de>
Message-ID: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>

On Thu, May 18, 2006 at 20:02, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> Nick Coghlan wrote:
> > And if I understand it correctly, it falls under the category that
> > waiting for another thread while holding the import lock is a *really*
> > bad idea from a thread safety point of view.
> >
> > The thing with the import-after-fork deadlock is that you can trigger it
> > without even doing anything that's known not to be thread-safe.
>
> Right. With some googling, I found that one solution is pthread_atexit:
> a pthread_atexit handler is a triple (before, in_parent, in_child).
> You set it to (acquire, release, release). When somebody forks,
> the pthread library will first acquire the import lock in the thread
> that wants to fork. Then the fork occurs, and the import lock gets
> then released both in the parent and in the child.
>
> I would like to see this approach implemented, but I agree with you
> that a test case should be created first.
>

Picking up a rather old discussion... We encountered this bug at Google and
I'm now "incentivized" to fix it.

For a short recap: Python has an import lock that prevents more than one
thread from doing an import at any given time. However, unlike most of the
locks we have lying around, we don't clear that lock in the child after an
os.fork(). That means that doing an os.fork() during an import means the
child process can't do any other imports. It also means that doing an
os.fork() *while another thread is doing an import* means the child process
can't do any other imports.

Since this three-year-old discussion we've added a couple of
post-fork-cleanups to CPython (the TLS, the threading module's idea of
active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do
simply discard the memory for other locks held during fork (the GIL, see
Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in
Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock,
except when the platform is AIX. I don't see any particular reason why we
aren't doing the same thing to the import lock that we do to the other
locks, on all platforms. It's a quick fix for a real problem (see
http://bugs.python.org/issue1590864 and
http://bugs.python.org/issue1404925 for two bugreports that seem to be this
very issue.)

It also seems to me, since we have two or three (depending on how you count)
uses for it, that we should either add a way to free the old locks (to the
various threading implementations) or add a way to use pthread_atexit
portably -- or at least portably across systems with fork(). I don't think
we should wait with fixing the big issue (making threads and fork() being
unnecessarily flakey) until we have a good fix for the small issue (a tiny
bit of a memory leak after a fork() -- in the child process only.)
Especially since the good fix for the small issue might require
co-ordination between all the threading implementations we have and nobody
knows about.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090716/a11ca265/attachment.htm>

From ncoghlan at gmail.com  Thu Jul 16 23:07:22 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Jul 2009 07:07:22 +1000
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>	
	<ca471dc20605112148s8414d85j6c14bf2ed622cccd@mail.gmail.com>	
	<44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de>	
	<44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de>	
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>	
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>	
	<446CB69E.4090609@v.loewis.de>
	<9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
Message-ID: <4A5F968A.4060706@gmail.com>

Thomas Wouters wrote:
> Picking up a rather old discussion... We encountered this bug at Google
> and I'm now "incentivized" to fix it.
> 
> For a short recap: Python has an import lock that prevents more than one
> thread from doing an import at any given time. However, unlike most of
> the locks we have lying around, we don't clear that lock in the child
> after an os.fork(). That means that doing an os.fork() during an import
> means the child process can't do any other imports. It also means that
> doing an os.fork() *while another thread is doing an import* means the
> child process can't do any other imports.
> 
> Since this three-year-old discussion we've added a couple of
> post-fork-cleanups to CPython (the TLS, the threading module's idea of
> active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we
> already do simply discard the memory for other locks held during fork
> (the GIL, see Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in
> Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock,
> except when the platform is AIX. I don't see any particular reason why
> we aren't doing the same thing to the import lock that we do to the
> other locks, on all platforms. It's a quick fix for a real problem (see
> http://bugs.python.org/issue1590864 and
> http://bugs.python.org/issue1404925 for two bugreports that seem to be
> this very issue.)

+1. Leaving deadlock issues around that people can run into without
doing anything wrong in their application is unfriendly.

> It also seems to me, since we have two or three (depending on how you
> count) uses for it, that we should either add a way to free the old
> locks (to the various threading implementations) or add a way to use
> pthread_atexit portably -- or at least portably across systems with
> fork(). I don't think we should wait with fixing the big issue (making
> threads and fork() being unnecessarily flakey) until we have a good fix
> for the small issue (a tiny bit of a memory leak after a fork() -- in
> the child process only.) Especially since the good fix for the small
> issue might require co-ordination between all the threading
> implementations we have and nobody knows about.

One-off memory leaks for relatively rare operations like fork() don't
bother me all that much, so I agree that having fork() leak another lock
in the child process is preferable to it having a chance to mysteriously
deadlock.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dalcinl at gmail.com  Thu Jul 16 23:17:12 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Jul 2009 18:17:12 -0300
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <446CB69E.4090609@v.loewis.de>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>
	<ca471dc20605112148s8414d85j6c14bf2ed622cccd@mail.gmail.com>
	<44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de>
	<44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>
	<446CB69E.4090609@v.loewis.de>
Message-ID: <e7ba66e40907161417m21ecd184l416f541a35d8f539@mail.gmail.com>

On Thu, May 18, 2006 at 3:02 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>
> Right. With some googling, I found that one solution is pthread_atexit:
> a pthread_atexit handler is a triple (before, in_parent, in_child).
> You set it to (acquire, release, release).

Did you mean pthread_atfork() ?

> When somebody forks,
> the pthread library will first acquire the import lock in the thread
> that wants to fork. Then the fork occurs, and the import lock gets
> then released both in the parent and in the child.
>
> I would like to see this approach implemented, but I agree with you
> that a test case should be created first.
>


Whould some new C-API functions do the trick?

PyOS_AtForkPrepare() # acquire import lock
pid = fork()
if (pid == 0)
    PyOS_AtForkChild() # current PyOS_AfterFork() + release import lock
else
    PyOS_AtForkParent() # release import lock


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ncoghlan at gmail.com  Thu Jul 16 23:20:49 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Jul 2009 07:20:49 +1000
Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of  view
In-Reply-To: <4A5F1C58.3030003@online.de>
References: <op.uw06u6px1q4jlt@whymac.activestate.com>	<94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com>	<79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com>	<94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com>	<4A5DF2D9.6050403@online.de>	<20090715162736.B54D33A4116@sparrow.telecommunity.com>
	<4A5F1C58.3030003@online.de>
Message-ID: <4A5F99B1.6000006@gmail.com>

Joachim K?nig wrote:
> So one would have to set up the application specific packages before
> running the application, but the
> whole clutter of uncounted versions of the same package in one directory
> could go away. The
> "drawback" of this approach would be, that the same version of a package
> would have to be installed
> multiple times if needed by different applications.

While this is a very common practice in the Windows world, it is far
less common in the *nix world of vendor managed packaging systems.

As for why it can be a problem, it (bundling libraries with
applications) makes security vulnerability management a *lot* more
difficult for system administrators. If a bug is found in a key library
(e.g. openssl) a dependency based system just needs to update the single
shared copy of that library. In a bundling system, you first have to
work out which of your applications contain an instance of that library
and then see if the application vendors have provided a security patch.
If any one of them hasn't released a patch and you can't patch it
yourself, then you either have to stop using that application or else
accept remaining exposed to the vulnerability.

The bundling approach also leads to applications being much bigger than
they need to be. That isn't much of a problem for desktop or server
systems these days, but can still be an issue in the embedded world.

As far as the idea of making bundling easier goes, we already
implemented that in 2.6 and 3.0. It's the whole reason that zip files
and directories are directly executable now: the named zip file or
directory itself is automatically added to sys.path, so the top level
"__main__.py" in that location can freely import any other co-located
modules and packages.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From status at bugs.python.org  Fri Jul 17 18:07:04 2009
From: status at bugs.python.org (Python tracker)
Date: Fri, 17 Jul 2009 18:07:04 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20090717160704.57CCA7861B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (07/10/09 - 07/17/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2255 open (+32) / 16072 closed (+15) / 18327 total (+47)

Open issues with patches:   892

Average duration of open issues: 662 days.
Median duration of open issues: 414 days.

Open Issues Breakdown
   open  2222 (+31)
pending    33 ( +1)

Issues Created Or Reopened (48)
_______________________________

locale.D_* and .T_* are int, not string                          07/10/09
CLOSED http://bugs.python.org/issue6456    reopened r.david.murray                
                                                                               

distutils.command.build_ext.get_export_symbols should use the "P 07/11/09
CLOSED http://bugs.python.org/issue6459    reopened tarek                         
       patch                                                                   

test failure in test_xmlrpc on Gentoo in trunk                   07/11/09
       http://bugs.python.org/issue6460    created  r.david.murray                
                                                                               

multiprocessing: freezing apps on Windows                        07/11/09
       http://bugs.python.org/issue6461    created  sgm                           
                                                                               

bsddb3 intermittent test failures                                07/11/09
       http://bugs.python.org/issue6462    created  r.david.murray                
                                                                               

IDLE with Tk-Cocoa: Edit, format menus hang                      07/11/09
       http://bugs.python.org/issue6463    created  wordtech                      
                                                                               

test_normalization failures due to truncated NormalizationTest.t 07/11/09
CLOSED http://bugs.python.org/issue6464    created  r.david.murray                
       easy                                                                    

email.feedparser regular expression bug (NLCRE_crack)            07/11/09
       http://bugs.python.org/issue6465    created  jkg                           
       patch                                                                   

duplicate get_version() code between cygwinccompiler and emxccom 07/12/09
       http://bugs.python.org/issue6466    created  tarek                         
                                                                               

raw_input() doesn't work as expected when it gets multiple ^D    07/12/09
       http://bugs.python.org/issue6467    created  lucaspmelo                    
                                                                               

__missing__ not completely implemented in dictobject.c           07/12/09
CLOSED http://bugs.python.org/issue6468    created  bvukov                        
                                                                               

Function definition expressions feature                          07/12/09
CLOSED http://bugs.python.org/issue6469    created  grammati                      
       patch                                                                   

Tkinter import fails when running Python.exe from a network shar 07/12/09
       http://bugs.python.org/issue6470    created  cgohlke                       
       patch, needs review                                                     

errno and strerror attributes incorrectly set on socket errors w 07/12/09
       http://bugs.python.org/issue6471    created  ezio.melotti                  
                                                                               

Inconsistent use of "iterator" in ElementTree doc & diff between 07/13/09
       http://bugs.python.org/issue6472    created  MLModel                       
                                                                               

hmac sha384/sha512 fails test vectors                            07/13/09
CLOSED http://bugs.python.org/issue6473    created  iwade                         
                                                                               

Inconsistent TypeError message on function calls with wrong numb 07/13/09
       http://bugs.python.org/issue6474    created  hagen                         
                                                                               

Documentation > Tutorial > List Comprehensions                   07/13/09
CLOSED http://bugs.python.org/issue6475    created  Retro                         
                                                                               

MSVCRT's spawnve/spawnvpe are not thread safe                    07/13/09
       http://bugs.python.org/issue6476    created  jfonseca                      
                                                                               

Pickling of NoneType raises PicklingError                        07/13/09
       http://bugs.python.org/issue6477    created  July                          
                                                                               

time.tzset does not reset _strptime's locale time cache          07/13/09
       http://bugs.python.org/issue6478    created  mibanescu                     
                                                                               

platform.py writes to hard coded platform dependant "dev/null"   07/13/09
CLOSED http://bugs.python.org/issue6479    created  john.burnett                  
       patch                                                                   

code.runsource() parsing bug                                     07/14/09
       http://bugs.python.org/issue6480    created  smcallis                      
                                                                               

PATCH: typo in subprocess documentation                          07/14/09
CLOSED http://bugs.python.org/issue6481    created  Yinon                         
       patch                                                                   

PATCH: tiny simplification for subprocess                        07/14/09
CLOSED http://bugs.python.org/issue6482    created  Yinon                         
       patch                                                                   

Modules are not deallocated correctly if m_size = -1             07/14/09
       http://bugs.python.org/issue6483    created  jak                           
                                                                               

No unit test for mailcap module                                  07/14/09
       http://bugs.python.org/issue6484    created  gnofi                         
                                                                               

is_finished not exported by zlib                                 07/14/09
CLOSED http://bugs.python.org/issue6485    created  solinym                       
                                                                               

Library doc introduction strangely points to "Built-in Objects"  07/14/09
CLOSED http://bugs.python.org/issue6486    created  MLModel                       
                                                                               

Some index entries appear in black                               07/14/09
CLOSED http://bugs.python.org/issue6487    created  MLModel                       
                                                                               

ElementTree documentation refers to "path" with no explanation,  07/15/09
       http://bugs.python.org/issue6488    created  MLModel                       
                                                                               

Documentation of ElementTree.Element.getiterator implies element 07/15/09
       http://bugs.python.org/issue6489    created  MLModel                       
                                                                               

os.popen documentation in 2.6 is probably wrong                  07/15/09
       http://bugs.python.org/issue6490    created  krawyoti                      
                                                                               

Improve --with-dbmliborder option                                07/15/09
       http://bugs.python.org/issue6491    created  Arfrever                      
       patch                                                                   

xml.etree does not escape CR, LF and TAB characters within attri 07/15/09
       http://bugs.python.org/issue6492    created  devon                         
                                                                               

Can not set value for structure members larger than 32 bits      07/16/09
       http://bugs.python.org/issue6493    created  higstar                       
       patch                                                                   

xmlrpc client does not support HTTPS                             07/16/09
CLOSED http://bugs.python.org/issue6494    created  lister171254                  
                                                                               

particular variable's name case exception attributeError         07/16/09
CLOSED http://bugs.python.org/issue6495    created  tq0fqeu                       
                                                                               

2to3 generates "from urllib.parse import pathname2url"           07/16/09
       http://bugs.python.org/issue6496    created  srid                          
       patch                                                                   

Support for digital Cinema/film DPX and Kodak Cineon image file  07/16/09
       http://bugs.python.org/issue6497    created  waltermb                      
       patch                                                                   

Py_Main() does not return on SystemExit                          07/16/09
       http://bugs.python.org/issue6498    created  Rogi                          
                                                                               

Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer w 07/17/09
       http://bugs.python.org/issue6499    created  ezio.melotti                  
                                                                               

urllib2 maximum recursion depth exceeded                         07/17/09
       http://bugs.python.org/issue6500    created  nako521                       
                                                                               

Fatal LookupError: unknown encoding: cp0 on Windows embedded sta 07/17/09
       http://bugs.python.org/issue6501    created  grahamd                       
                                                                               

documentation error: missing comma between kwonlyargcount & nloc 07/17/09
       http://bugs.python.org/issue6502    created  kaizhu                        
                                                                               

Python Docs: 6.6.4. Mutable Sequence Types                       07/17/09
CLOSED http://bugs.python.org/issue6503    created  mkesper                       
                                                                               

infinite recursion from calling builtins.open()                  07/17/09
       http://bugs.python.org/issue6504    created  kaizhu                        
                                                                               

Minor typos in tutorial (i/o chapter)                            07/17/09
       http://bugs.python.org/issue6505    created  csmayfield                    
                                                                               



Issues Now Closed (43)
______________________

Doc changes left over after mega-merge from trunk                 622 days
       http://bugs.python.org/issue1370    amaury.forgeotdarc            
                                                                               

Confusing Descrintro example                                      534 days
       http://bugs.python.org/issue1921    benjamin.peterson             
       easy                                                                    

Array pickling exposes internal memory representation of element  484 days
       http://bugs.python.org/issue2389    alexandre.vassalotti          
       patch                                                                   

Check 2to3 for support of print function.                         481 days
       http://bugs.python.org/issue2412    benjamin.peterson             
       patch                                                                   

Import errors in email.message.py                                 461 days
       http://bugs.python.org/issue2622    amaury.forgeotdarc            
       patch                                                                   

"Suggest a change" link                                           370 days
       http://bugs.python.org/issue3341    georg.brandl                  
                                                                               

typo in a module describes utf-8 as uft-8                         223 days
       http://bugs.python.org/issue4540    amaury.forgeotdarc            
       patch, needs review                                                     

kqueue for more than one event is broken.                          74 days
       http://bugs.python.org/issue5910    georg.brandl                  
       patch                                                                   

Missing title for configDialog.py                                  56 days
       http://bugs.python.org/issue6076    amaury.forgeotdarc            
       patch                                                                   

os.popen exit code inconsistent                                    13 days
       http://bugs.python.org/issue6358    amaury.forgeotdarc            
                                                                               

Dictionaries should support __add__                                 6 days
       http://bugs.python.org/issue6410    tjreedy                       
                                                                               

warnings.warn segfaults on bad formatted string                    13 days
       http://bugs.python.org/issue6415    ocean-city                    
       patch                                                                   

unittest.TestProgram change breaks nose                            10 days
       http://bugs.python.org/issue6418    michael.foord                 
       patch                                                                   

errors in docs re module initialization vs self arg to functions    6 days
       http://bugs.python.org/issue6421    georg.brandl                  
       easy                                                                    

2to3: fix_future conflicts with fix_print                           6 days
       http://bugs.python.org/issue6429    benjamin.peterson             
                                                                               

array.array falsely advertises support for 'w' in documentation     5 days
       http://bugs.python.org/issue6430    georg.brandl                  
                                                                               

ImageTk.PhotoImage                                                  7 days
       http://bugs.python.org/issue6432    srid                          
                                                                               

multiprocessing: pool.map hangs on empty list                       9 days
       http://bugs.python.org/issue6433    jnoller                       
       patch                                                                   

cygwinccompiler regular expressions broken                          4 days
       http://bugs.python.org/issue6438    tarek                         
                                                                               

2to3: convert deprecated string.maketrans to str.maketrans          3 days
       http://bugs.python.org/issue6440    benjamin.peterson             
                                                                               

import_spam() in extending python can fail without setting the e    2 days
       http://bugs.python.org/issue6446    georg.brandl                  
                                                                               

PATCH: typo (English) in subprocess module                          1 days
       http://bugs.python.org/issue6447    amaury.forgeotdarc            
       patch                                                                   

imp.find_module() -- be explicity that 'path' must be a list        2 days
       http://bugs.python.org/issue6448    georg.brandl                  
                                                                               

Improve/update python.org/dev/                                      8 days
       http://bugs.python.org/issue6449    ezio.melotti                  
       easy                                                                    

Improve bool TypeError message                                      1 days
       http://bugs.python.org/issue6453    tjreedy                       
       easy                                                                    

locale.D_* and .T_* are int, not string                             1 days
       http://bugs.python.org/issue6456    georg.brandl                  
                                                                               

With logging.config.fileConfig can't create logger without handl    3 days
       http://bugs.python.org/issue6458    vsajip                        
                                                                               

distutils.command.build_ext.get_export_symbols should use the "P    0 days
       http://bugs.python.org/issue6459    tarek                         
       patch                                                                   

test_normalization failures due to truncated NormalizationTest.t    0 days
       http://bugs.python.org/issue6464    benjamin.peterson             
       easy                                                                    

__missing__ not completely implemented in dictobject.c              0 days
       http://bugs.python.org/issue6468    benjamin.peterson             
                                                                               

Function definition expressions feature                             1 days
       http://bugs.python.org/issue6469    r.david.murray                
       patch                                                                   

hmac sha384/sha512 fails test vectors                               4 days
       http://bugs.python.org/issue6473    iwade                         
                                                                               

Documentation > Tutorial > List Comprehensions                      0 days
       http://bugs.python.org/issue6475    georg.brandl                  
                                                                               

platform.py writes to hard coded platform dependant "dev/null"      0 days
       http://bugs.python.org/issue6479    john.burnett                  
       patch                                                                   

PATCH: typo in subprocess documentation                             2 days
       http://bugs.python.org/issue6481    georg.brandl                  
       patch                                                                   

PATCH: tiny simplification for subprocess                           3 days
       http://bugs.python.org/issue6482    georg.brandl                  
       patch                                                                   

is_finished not exported by zlib                                    0 days
       http://bugs.python.org/issue6485    ezio.melotti                  
                                                                               

Library doc introduction strangely points to "Built-in Objects"     1 days
       http://bugs.python.org/issue6486    georg.brandl                  
                                                                               

Some index entries appear in black                                  2 days
       http://bugs.python.org/issue6487    georg.brandl                  
                                                                               

xmlrpc client does not support HTTPS                                0 days
       http://bugs.python.org/issue6494    loewis                        
                                                                               

particular variable's name case exception attributeError            0 days
       http://bugs.python.org/issue6495    amaury.forgeotdarc            
                                                                               

Python Docs: 6.6.4. Mutable Sequence Types                          0 days
       http://bugs.python.org/issue6503    ezio.melotti                  
                                                                               

cp720 encoding map                                                940 days
       http://bugs.python.org/issue1616979 amaury.forgeotdarc            
       patch, needs review                                                     



Top Issues Most Discussed (10)
______________________________

 10 distutils.command.build_ext.get_export_symbols should use the "    0 days
closed  http://bugs.python.org/issue6459   

  7 multiline exception logging via syslog handler                     8 days
open    http://bugs.python.org/issue6444   

  6 os.popen documentation in 2.6 is probably wrong                    2 days
open    http://bugs.python.org/issue6490   

  6 test failure in test_xmlrpc on Gentoo in trunk                     7 days
open    http://bugs.python.org/issue6460   

  6 Fraction fails equality test with a user-defined type             10 days
open    http://bugs.python.org/issue6431   

  6 OS X: python3 from python-3.1.dmg crashes at startup              16 days
open    http://bugs.python.org/issue6393   

  6 CVE-2008-5983 python: untrusted python modules search path        94 days
open    http://bugs.python.org/issue5753   

  5 cp720 encoding map                                               940 days
closed  http://bugs.python.org/issue1616979

  4 PATCH: tiny simplification for subprocess                          3 days
closed  http://bugs.python.org/issue6482   

  4 platform.py writes to hard coded platform dependant "dev/null"     0 days
closed  http://bugs.python.org/issue6479   




From jafo at tummy.com  Sat Jul 18 02:28:36 2009
From: jafo at tummy.com (Sean Reifschneider)
Date: Fri, 17 Jul 2009 18:28:36 -0600
Subject: [Python-Dev] Experiment: Adding "re" to string objects.
Message-ID: <20090718002836.GA26311@tummy.com>

I'm mailing this to python-dev because I'd like feedback on the idea of
adding an "re" attribute to strings.  I'm not sure if it's a good idea or
not yet, but I figure it's worth discussion.  The module mentioned here
includes a class called "restr()" which allows you to play with "s.re".

As some of you may recall, I'm not particularly fond of the recipe:

   m = re.match(r'whatever(.*)', s)
   if m:
      m.group(1)

The other morning I came up on the idea of adding an "re" to strings, so
you could do things like:

   if s.re.match(r'whatever(.*)'):
      print s.re.group(1)

or:

   if (date.re.match(r'(?P<year>\d\d\d\d)-(?P<month>\d\d)' or
         date.re.match(r'(?P<month>\d\d)-(?P<year>\d\d\d\d)'):
      print date.re.groupdict('year')

So I decided to try experimenting with it and see how I like it.  I've also
thrown a bunch of other stuff into it and made a module called
"filtertools":

   http://pypi.python.org/pypi/filtertools/0.01
   ftp://ftp.tummy.com/pub/tummy/Python/python-filtertools/

As the version number is meant to indicate, this is something that I'm
still exploring whether it is the right thing done in the right way.
Though at the moment the only thing I plan to change is that some of the
iterators (having nothing to do with adding "re" to string objects)
probably shouldn't consume the "barrier" such as the "dropwhile()" and
"takewhile()".  You might want to do something like:

   fp = filtertools.reopen('mailbox')
   for header in filtertools.takewhile([ r'^\S' ], fp.readlines()) :
      print 'HEADER:', header.rstrip()
      for continued in filtertools.takewhile([ r'^\s+\S' ], fp.readlines()) :
         print 'CONTINUED:', continued.rstrip()

But, the "takewhile()" I will consume the first non-matching line.

Anyway, I appreciate any feedback folks have.

Thanks,
Sean
-- 
 What we see depends on mainly what we look for.
                 -- John Lubbock
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From aahz at pythoncraft.com  Sat Jul 18 02:48:41 2009
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 17 Jul 2009 17:48:41 -0700
Subject: [Python-Dev] Experiment: Adding "re" to string objects.
In-Reply-To: <20090718002836.GA26311@tummy.com>
References: <20090718002836.GA26311@tummy.com>
Message-ID: <20090718004841.GB15099@panix.com>

On Fri, Jul 17, 2009, Sean Reifschneider wrote:
>
> I'm mailing this to python-dev because I'd like feedback on the idea of
> adding an "re" attribute to strings.  I'm not sure if it's a good idea or
> not yet, but I figure it's worth discussion.  The module mentioned here
> includes a class called "restr()" which allows you to play with "s.re".

Ideas should go to python-ideas, please.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur."  --Red Adair

From python at mrabarnett.plus.com  Sat Jul 18 02:51:54 2009
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 18 Jul 2009 01:51:54 +0100
Subject: [Python-Dev] Experiment: Adding "re" to string objects.
In-Reply-To: <20090718002836.GA26311@tummy.com>
References: <20090718002836.GA26311@tummy.com>
Message-ID: <4A611CAA.5020108@mrabarnett.plus.com>

Sean Reifschneider wrote:
> I'm mailing this to python-dev because I'd like feedback on the idea of
> adding an "re" attribute to strings.  I'm not sure if it's a good idea or
> not yet, but I figure it's worth discussion.  The module mentioned here
> includes a class called "restr()" which allows you to play with "s.re".
> 
> As some of you may recall, I'm not particularly fond of the recipe:
> 
>    m = re.match(r'whatever(.*)', s)
>    if m:
>       m.group(1)
> 
> The other morning I came up on the idea of adding an "re" to strings, so
> you could do things like:
> 
>    if s.re.match(r'whatever(.*)'):
>       print s.re.group(1)
> 
> or:
> 
>    if (date.re.match(r'(?P<year>\d\d\d\d)-(?P<month>\d\d)' or
>          date.re.match(r'(?P<month>\d\d)-(?P<year>\d\d\d\d)'):
>       print date.re.groupdict('year')
> 
[snip]
Why not drop the ".re" part? You would, however, then need a new name
for the re split, eg "re_split".

Or you could make the string the pattern, eg r'whatever(.*)'.match(s).

From p.f.moore at gmail.com  Sat Jul 18 11:55:40 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 18 Jul 2009 10:55:40 +0100
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <bbaeab100907041228l7ac37389s10cbf9ab0d9267a0@mail.gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>
	<bbaeab100907031104u671180akf301d1aaadfacc07@mail.gmail.com>
	<ea2499da0907040103x5df7a15bh8f1dcb86c9adbaa0@mail.gmail.com>
	<bbaeab100907041228l7ac37389s10cbf9ab0d9267a0@mail.gmail.com>
Message-ID: <79990c6b0907180255q5c0482ds5a8d05c3babc7716@mail.gmail.com>

2009/7/4 Brett Cannon <brett at python.org>:
>> > While I really like the idea of using named branches for each release so
>> > that there is a single py3k branch that contains all relevant history
>> > for
>> > every release, I think we should start simple and go with cloned
>> > branches.
>> > That way the workflow does not radically shift from what we do now for
>> > svn
>> > to start. Once the conversion is done and people are comfortable with hg
>> > we
>> > can then discuss moving towards a named branch approach.
>>
>> I don't think the cloned branches is much simpler than the named
>> branches approach, in several ways. For example, populating the branch
>> part of a sys.whatever value is significantly harder. Also, if you
>> follow a useful tagging approach, doing cloned branches means that
>> release tags aren't available on trunk/main/default. That seems like a
>> step backwards.
>
> I personally prefer named branches, but that's just me and I am not about to
> force my preferences on everyone. Guess we just have to see if others have
> opinions against named branches.

Personally, I prefer clones, as it seems to me that Mercurial named
branches are not quite what people generally think of when they think
of "branches", in some subtle respects. I could be wrong, as I don't
personally use named branches. However, there's been a small thread on
distutils-sig recently with a new Mercurial user complaining that he's
got confused and messed up a repository. Without digging particularly
deeply, it appears that the problems were caused by confusion over
named branches.

FWIW, the Mercurial book
(http://hgbook.red-bean.com/read/managing-releases-and-branchy-development.html)
describes named branches with the comment "If you're more in the
?power user? category (*and* your collaborators are too)" (author's
emphasis). I'm not sure we want to require contributors to be "power
users" of Mercurial...

Paul.

PS Sorry for responding to an old thread - I couldn't locate more
recent discussion, although I thought there had been some.

From fuzzyman at voidspace.org.uk  Sat Jul 18 13:21:11 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 18 Jul 2009 12:21:11 +0100
Subject: [Python-Dev] Experiment: Adding "re" to string objects.
In-Reply-To: <4A611CAA.5020108@mrabarnett.plus.com>
References: <20090718002836.GA26311@tummy.com>
	<4A611CAA.5020108@mrabarnett.plus.com>
Message-ID: <4A61B027.4050106@voidspace.org.uk>

MRAB wrote:
> Sean Reifschneider wrote:
>> I'm mailing this to python-dev because I'd like feedback on the idea of
>> adding an "re" attribute to strings.  I'm not sure if it's a good 
>> idea or
>> not yet, but I figure it's worth discussion.  The module mentioned here
>> includes a class called "restr()" which allows you to play with "s.re".
>>
>> As some of you may recall, I'm not particularly fond of the recipe:
>>
>>    m = re.match(r'whatever(.*)', s)
>>    if m:
>>       m.group(1)
>>
>> The other morning I came up on the idea of adding an "re" to strings, so
>> you could do things like:
>>
>>    if s.re.match(r'whatever(.*)'):
>>       print s.re.group(1)
>>
>> or:
>>
>>    if (date.re.match(r'(?P<year>\d\d\d\d)-(?P<month>\d\d)' or
>>          date.re.match(r'(?P<month>\d\d)-(?P<year>\d\d\d\d)'):
>>       print date.re.groupdict('year')
>>
> [snip]
> Why not drop the ".re" part? You would, however, then need a new name
> for the re split, eg "re_split".
>
> Or you could make the string the pattern, eg r'whatever(.*)'.match(s).

+1 for re support built-in to strings.

Michael Foord

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk 
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From jimjjewett at gmail.com  Sat Jul 18 22:07:39 2009
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sat, 18 Jul 2009 16:07:39 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 -
	from PyPM's point of view]
Message-ID: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>

Michael Foord wrote:
>> I agree. People with versioning issues *should* be using virtualenv.

Tarek Ziad? replied (and several people later agreed)
> Let's remove site-packages from Python then.

What about those people *without* versioning issues?

I have no qualms suggesting that package management programs avoid the
use of site-packages.  Such programs do need to cater to edge cases.

But a single drop-it-in directory works great for the vast majority of
*my* needs; requiring me to drink the Kool-Aid from a specific package
management system just to get access to any add-ons -- even those I
wrote myself in pure python -- would be a huge step backwards.

-jJ

From pje at telecommunity.com  Sat Jul 18 22:22:37 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 18 Jul 2009 16:22:37 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from PyPM's point of view]
In-Reply-To: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
Message-ID: <20090718201933.B6AE63A403A@sparrow.telecommunity.com>

At 04:07 PM 7/18/2009 -0400, Jim Jewett wrote:
>Michael Foord wrote:
> >> I agree. People with versioning issues *should* be using virtualenv.
>
>Tarek Ziad? replied (and several people later agreed)
> > Let's remove site-packages from Python then.
>
>What about those people *without* versioning issues?
>
>I have no qualms suggesting that package management programs avoid the
>use of site-packages.  Such programs do need to cater to edge cases.
>
>But a single drop-it-in directory works great for the vast majority of
>*my* needs; requiring me to drink the Kool-Aid from a specific package
>management system just to get access to any add-ons -- even those I
>wrote myself in pure python -- would be a huge step backwards.

IIUC, the new "user" directories supported by the 'site' module would 
fill this purpose.  That is, rather than having a site-wide packages 
directory, there'd just be the user-specific package directories.

(That having been said, I'm not actually in favor of the idea.)


From david.lyon at preisshare.net  Sun Jul 19 02:49:30 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 18 Jul 2009 20:49:30 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-_from_PyPM=27s_point_of_view=5D?=
In-Reply-To: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
Message-ID: <d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>

On Sat, 18 Jul 2009 16:07:39 -0400, Jim Jewett <jimjjewett at gmail.com>
wrote:
> What about those people *without* versioning issues?
> 
> I have no qualms suggesting that package management programs avoid the
> use of site-packages.  Such programs do need to cater to edge cases.

I'm just wondering how that is possible? Given that a package management
system (pythonpkgmgr - for example) is a tool to assist in the management
of packages in site-packages.

If you take away the drop in directory (site-packages), effectively
you're taking away the place for system drop in packages. I can't see
the point in that.

> But a single drop-it-in directory works great for the vast majority of
> *my* needs; requiring me to drink the Kool-Aid from a specific package
> management system just to get access to any add-ons -- even those I
> wrote myself in pure python -- would be a huge step backwards.

Where you keep your packages is entirely up to you.

Even with the Python Package Manager, you can still have unversioned
packages in project directories. That doesn't change.

And you can still have unversioned packages in site-packages.

So I can't really see that having a Package Manager program
takes your use case scenerio backwards because it wouldn't really
change it.

(repeating)
> But a single drop-it-in directory works great for the vast majority of
> *my* needs;

That's exactly what site-packages is.....

So it isn't clear why you want to remove the thing that you are
advocating works so great....

David


From ncoghlan at gmail.com  Sun Jul 19 08:07:28 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 19 Jul 2009 16:07:28 +1000
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from PyPM's point of view]
In-Reply-To: <d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
Message-ID: <4A62B820.4030903@gmail.com>

David Lyon wrote:
> So it isn't clear why you want to remove the thing that you are
> advocating works so great....

Jim was quoting someone *else* that had suggested removing it (I'm not
sure how serious the original suggestion actually was though).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From rdmurray at bitdance.com  Sun Jul 19 15:08:20 2009
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 19 Jul 2009 09:08:20 -0400 (EDT)
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from PyPM's point of view]
In-Reply-To: <4A62B820.4030903@gmail.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
Message-ID: <Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>

On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote:
> David Lyon wrote:
>> So it isn't clear why you want to remove the thing that you are
>> advocating works so great....
>
> Jim was quoting someone *else* that had suggested removing it (I'm not
> sure how serious the original suggestion actually was though).

It was Tarek, and I'm pretty sure it was in the nature of what
rhetoricians (among others) call "reductio ad absurdum".  But
Tarek would have to let us know for sure, since it seems from
subsequent discussion that the notion is not _completely_ absurd,
just mostly :)

--David

From skippy.hammond at gmail.com  Sun Jul 19 15:27:19 2009
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Sun, 19 Jul 2009 23:27:19 +1000
Subject: [Python-Dev] Mercurial migration: progress report (PEP 385)
In-Reply-To: <4A4ED492.1050307@gmail.com>
References: <ea2499da0907021342g7d5e7f8dx80f908da00514836@mail.gmail.com>		<4A4D3C44.60504@gmail.com>	<ea2499da0907030428l4eb10cdev1576334a78bd944c@mail.gmail.com>	<4A4E0825.5020407@skippinet.com.au>
	<4A4E1009.4050901@gmail.com>	<4A4E9494.2040406@skippinet.com.au>
	<4A4EAE7A.5090406@gmail.com>	<4A4EB152.6040406@skippinet.com.au>
	<4A4EBEBE.7040604@gmail.com> <4A4ED492.1050307@gmail.com>
Message-ID: <4A631F37.6080601@gmail.com>

Sorry Dirkjan - I just noticed I didn't CC you on this mail originally. 
  I'm wondering if you have any more thoughts on these EOL issues and if 
there is anything I can do to help?

Cheers,

Mark

On 4/07/2009 2:03 PM, Mark Hammond wrote:
> On 4/07/2009 12:30 PM, Nick Coghlan wrote:
> ...
>> I think there needs to be a solid answer in place for these use cases
>> before the actual migration to Mercurial takes place. A hand-waving "use
>> win32text" isn't enough - it needs to be "use win32text with these exact
>> settings" (with server side hook support to enforce the rules).
>>
>> And since Mercurial doesn't even allow us to say "this is a binary file"
>> the way CVS used to I'm currently not seeing any way for that to happen
>> except for win32text to be updated to correctly handle wild cards in
>> combination with negative filters.
>
> I agree with your conclusion. My ruminating on this over the last few
> months leaves me thinking this would involve:
>
> * my older 'accepted but then lost' hg patch to allow an explicit 'none'
> rule for a single file to override wildcards.
>
> * win32text be enhanced to use a normal versioned file in the root of
> the repo, much like hgingore, where a project can maintain project wide
> rules.
>
> * win32text be enhanced such that all python developers, regardless of
> platform, are willing to use this extension, even if the majority of
> files happen to use their native line ending (sauce for the goose is
> sauce for the gander, and all that...)
>
> * commit hooks be implemented to enforce this - but this should not be
> necessary if the above was implemented and socially enforced.
>
> Cheers,
>
> Mark
>>
>> Cheers,
>> Nick.
>>
>> [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html
>>
>


From ziade.tarek at gmail.com  Sun Jul 19 17:15:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 19 Jul 2009 17:15:44 +0200
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from PyPM's point of view]
In-Reply-To: <Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
Message-ID: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>

On Sun, Jul 19, 2009 at 3:08 PM, R. David Murray<rdmurray at bitdance.com> wrote:
> On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote:
>>
>> David Lyon wrote:
>>>
>>> So it isn't clear why you want to remove the thing that you are
>>> advocating works so great....
>>
>> Jim was quoting someone *else* that had suggested removing it (I'm not
>> sure how serious the original suggestion actually was though).
>
> It was Tarek, and I'm pretty sure it was in the nature of what
> rhetoricians (among others) call "reductio ad absurdum". ?But
> Tarek would have to let us know for sure, since it seems from
> subsequent discussion that the notion is not _completely_ absurd,
> just mostly :)
>

Yes, well, maybe we should go back to Python-ideas to continue this
discussion, :)

What made me say that, is the unecessary complexity of the situation
because we have right now :

"Python the application" v.s. "Applications that uses Python the interpreter"

So we have:

A - a drop-it-in directory were we can put packages and modules

B - installers that adds automatically packages and modules in the
drop-it-in directory

C - applications that uses these installers to install themselves and
their dependencies in the drop-it-in directory.

D - a tool like virtualenv that duplicates a Python installation so
this drop-in-in directory is not shared globally

C - installation documentation that tells you to use virtualenv to
create a custom environment
      so you are sure not to break dependencies. (But you can break
them nevertheless because using virtualenv
      is not enforced :) )

E - OS packagers that installs all distributions in the drop-it-in
directory (or a similar place)


So if you remove the global site-packages, "Python the application"
dissapear in favor of "Python the interpreter",
and people have to point a local, non-shared drop-it-in directory to
run any python application that uses extra modules:

1- For Jim's case, the per-user site packages (PEP 370) can be used
2- For every application it can be a local directory loaded in
sys.path at startup
3- For OS packagers a single directory where they maintain a
collection of packages and modules for their system.

But unfortunately, the problem remains because of (3) : OS packagers
will fight against applications (2) that deal with their own
dependencies to avoid having duplicate packages in their system.

That's why a shared space, not loaded at startup by site.py, that
knows how to handle multiple versions for the same distribution
would be required. Every script (eg application) would decide which
version to use, or use the default version by default.
And a smart PEP 302 loader/importer would get the right versions.

Regards
Tarek

From david.lyon at preisshare.net  Mon Jul 20 01:13:30 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 19 Jul 2009 19:13:30 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?=
In-Reply-To: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
Message-ID: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net>

On Sun, 19 Jul 2009 17:15:44 +0200, Tarek Ziad? <ziade.tarek at gmail.com>
wrote:
> What made me say that, is the unecessary complexity of the situation
> because we have right now :
>
> ..
> 
> So if you remove the global site-packages, "Python the application"
> dissapear in favor of "Python the interpreter",
> and people have to point a local, non-shared drop-it-in directory to
> run any python application that uses extra modules:
> 
> 1- For Jim's case, the per-user site packages (PEP 370) can be used
> 2- For every application it can be a local directory loaded in
> sys.path at startup
> 3- For OS packagers a single directory where they maintain a
> collection of packages and modules for their system.
> 
> But unfortunately, the problem remains because of (3) : OS packagers
> will fight against applications (2) that deal with their own
> dependencies to avoid having duplicate packages in their system.

Well now I completely understand the jist of what you are saying..

And I completely agree with the direction of your comments.

This problem is being solved within the pythonpkgmgr project.

As we are allowing a distinction between operating system
packages and developer packages.

I only have ubuntu, and not OS-X at the moment, but let me
explain with that.

The operating system can install packages into /usr/lib/pythonX.X/
site-packages.

The develeloper can install packages into /usr/local/pythonX.x/
site-packages.

Therefore providing two seperate areas for packages to reside
and eliminating the chance for serious conflict.

The main problem imho has been that easy_install by default
has installed to /usr/lib/pythonX.X/site-packages when in
fact that really is an operating system controlled directory.

So naughty naughty to setuptools....

That's why in pythonpkgmgr, we're moving to an entirely
different place where there is no chance of conflict with
the os package manager and python packages can be installed
without su privelages.

If the user wishes to install o/s packages that's fine 
and if they wish to install developer packages that's fine too.

The saving grace here is the gui that remembers the appropriate
command line switches that are needed to direct this behaviour
because it doesn't happen by default in easy_install.

Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages
and I am checking other posix operating systems and trying to
work out what the equivilents might be.

When I have it working.. I will let you know..

I'm just trying to work out how to get the permissions going
correctly.

David


From mike.klaas at gmail.com  Mon Jul 20 20:26:50 2009
From: mike.klaas at gmail.com (Mike Klaas)
Date: Mon, 20 Jul 2009 11:26:50 -0700
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>
	<44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de>
	<44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>
	<446CB69E.4090609@v.loewis.de>
	<9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
Message-ID: <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com>

On Thu, Jul 16, 2009 at 1:08 PM, Thomas Wouters <thomas at python.org> wrote:

>
> Picking up a rather old discussion... We encountered this bug at Google and
> I'm now "incentivized" to fix it.
>
> For a short recap: Python has an import lock that prevents more than one
> thread from doing an import at any given time. However, unlike most of the
> locks we have lying around, we don't clear that lock in the child after an
> os.fork(). That means that doing an os.fork() during an import means the
> child process can't do any other imports. It also means that doing an
> os.fork() *while another thread is doing an import* means the child process
> can't do any other imports.
>
> Since this three-year-old discussion we've added a couple of
> post-fork-cleanups to CPython (the TLS, the threading module's idea of
> active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do
> simply discard the memory for other locks held during fork (the GIL, see
> Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in
> Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock,
> except when the platform is AIX. I don't see any particular reason why we
> aren't doing the same thing to the import lock that we do to the other
> locks, on all platforms. It's a quick fix for a real problem (see
> http://bugs.python.org/issue1590864 and
> http://bugs.python.org/issue1404925 for two bugreports that seem to be
> this very issue.)
>

+1.  We were also affected by this bug, getting sporatic deadlocks in a
multi-threaded program that fork()s subprocesses to do processing.
 It took a while to figure out what was going on.

-Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090720/211660ed/attachment.htm>

From thomas at python.org  Tue Jul 21 02:16:30 2009
From: thomas at python.org (Thomas Wouters)
Date: Mon, 20 Jul 2009 17:16:30 -0700
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>
	<446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com>
	<446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>
	<446CB69E.4090609@v.loewis.de>
	<9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
	<3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com>
Message-ID: <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com>

On Mon, Jul 20, 2009 at 11:26, Mike Klaas <mike.klaas at gmail.com> wrote:

>
>
> On Thu, Jul 16, 2009 at 1:08 PM, Thomas Wouters <thomas at python.org> wrote:
>
>>
>> Picking up a rather old discussion... We encountered this bug at Google
>> and I'm now "incentivized" to fix it.
>>
>> For a short recap: Python has an import lock that prevents more than one
>> thread from doing an import at any given time. However, unlike most of the
>> locks we have lying around, we don't clear that lock in the child after an
>> os.fork(). That means that doing an os.fork() during an import means the
>> child process can't do any other imports. It also means that doing an
>> os.fork() *while another thread is doing an import* means the child process
>> can't do any other imports.
>>
>> Since this three-year-old discussion we've added a couple of
>> post-fork-cleanups to CPython (the TLS, the threading module's idea of
>> active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do
>> simply discard the memory for other locks held during fork (the GIL, see
>> Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in
>> Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock,
>> except when the platform is AIX. I don't see any particular reason why we
>> aren't doing the same thing to the import lock that we do to the other
>> locks, on all platforms. It's a quick fix for a real problem (see
>> http://bugs.python.org/issue1590864 and
>> http://bugs.python.org/issue1404925 for two bugreports that seem to be
>> this very issue.)
>>
>
> +1.  We were also affected by this bug, getting sporatic deadlocks in a
> multi-threaded program that fork()s subprocesses to do processing.
>  It took a while to figure out what was going on.
>

Actually, after careful consideration I have come to realize that simply
resetting the lock is exactly the wrong thing to do.

The import lock exists to prevent threads from getting a half-initialized
module *while another module is creating the module*. That is to say, if
threads 'A' and 'B' both import 'mod', it is not only important that threads
A and B don't *both* try to execute mod.py, but also that thread B doesn't
get a half-initialized module object that thread A is busy populating.

if the import lock is held by the thread doing fork(), this is not a
problem: the module is still being imported in the child process, and the
fork doesn't affect it. If the import lock is held by another thread than
the one doing fork(), any partially imported modules would remain in
sys.modules in the child process, without ever getting finished.

So we need to actually acquire the import lock before forking. We can do
that in os.fork() and os.forkpty(), but we can't fix third-party extension
modules; we'll have to introduce a new set of API functions, for getting and
releasing the import lock. I would suggest we don't expose it as that, but
instead call it a fork lock or such, so we can add extra lock
acquire/release pairs as necessary.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090720/1dd82f4a/attachment.htm>

From lanyjie at yahoo.com  Tue Jul 21 06:19:10 2009
From: lanyjie at yahoo.com (Yingjie Lan)
Date: Mon, 20 Jul 2009 21:19:10 -0700 (PDT)
Subject: [Python-Dev] expy: an expressway to extend Python
Message-ID: <290588.53242.qm@web54205.mail.re2.yahoo.com>


Hi All,

This is to announce the initial release of expy 0.1.0.

More details at http://expy.sourceforge.net/

Thanks!

Yingjie


      

From phd at phd.pp.ru  Tue Jul 21 08:44:04 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 21 Jul 2009 10:44:04 +0400
Subject: [Python-Dev] expy: an expressway to extend Python
In-Reply-To: <290588.53242.qm@web54205.mail.re2.yahoo.com>
References: <290588.53242.qm@web54205.mail.re2.yahoo.com>
Message-ID: <20090721064404.GA3038@phd.pp.ru>

On Mon, Jul 20, 2009 at 09:19:10PM -0700, Yingjie Lan wrote:
> This is to announce the initial release of expy 0.1.0.
> More details at http://expy.sourceforge.net/

   What is it (the announcement is too brief and I'm not going to click on
a link without a real need)? How does it help core developers to develop
Python (if you posted this to the python-dev mailing list instead of c.l.p
or c.l.p.a)?

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From john at szakmeister.net  Tue Jul 21 11:44:03 2009
From: john at szakmeister.net (John Szakmeister)
Date: Tue, 21 Jul 2009 05:44:03 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from pythonpkgmgr's point of view]
In-Reply-To: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
Message-ID: <a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>

On Sun, Jul 19, 2009 at 7:13 PM, David Lyon<david.lyon at preisshare.net> wrote:
[snip]
> The main problem imho has been that easy_install by default
> has installed to /usr/lib/pythonX.X/site-packages when in
> fact that really is an operating system controlled directory.
>
> So naughty naughty to setuptools....

I've witnessed this a few times... and I completely agree with that
statement. :-)

> That's why in pythonpkgmgr, we're moving to an entirely
> different place where there is no chance of conflict with
> the os package manager and python packages can be installed
> without su privelages.
[snip
> Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages
> and I am checking other posix operating systems and trying to
> work out what the equivilents might be.

Perhaps I'm not reading what you're saying correctly.  Doesn't
/usr/local/pythonX.x/site-packages require su privileges to install?

FWIW, the equivalent of /usr/local/pythonX.x/site-packages on Mac OS X
seems to be /Library/Python/X.x/site-packages

-John

From tseaver at palladion.com  Tue Jul 21 19:41:41 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 21 Jul 2009 13:41:41 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
Message-ID: <h44ukm$fuk$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Szakmeister wrote:
> On Sun, Jul 19, 2009 at 7:13 PM, David Lyon<david.lyon at preisshare.net> wrote:
> [snip]
>> The main problem imho has been that easy_install by default
>> has installed to /usr/lib/pythonX.X/site-packages when in
>> fact that really is an operating system controlled directory.
>>
>> So naughty naughty to setuptools....
> 
> I've witnessed this a few times... and I completely agree with that
> statement. :-)

That behvior isn't setuptools' fault:  it is the behavior of *any*
distutils-derived app.  "OS controlled directory" is not a concept which
the distutils honors at all:  $prefix/lib/pytoonX.Y/site-packages is the
*intended* installation location for any modules not shipped with Python.

Consider the case where the user has built a separate python (e.g., to
avoid conflicting with the OS version) in /opt/PythonX.y:  are you
actually saying that distutils / setuptools should somehow know the
difference betweeen /usr/lib/pythonX.y/site-packages and
/opt/PythonX.Y/lib/pythonX.Y/site-packages?  Or are you asserting that
it is somehow invalid for distutils / setuptools to install things into
/opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an
"OS controlled directory."


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKZf3V+gerLs4ltQ4RAud7AJ9GG9NY4UaKpQ5bBIQ+N6tztPVJSgCdEEIa
DUugfdzvBYgPtaas9c+fTTg=
=gvLY
-----END PGP SIGNATURE-----


From lanyjie at yahoo.com  Tue Jul 21 20:22:19 2009
From: lanyjie at yahoo.com (Yingjie Lan)
Date: Tue, 21 Jul 2009 11:22:19 -0700 (PDT)
Subject: [Python-Dev] expy: an expressway to extend Python
Message-ID: <765806.59082.qm@web54202.mail.re2.yahoo.com>


Hi, 

Here is a brief example on how to use expy to implement the math module:
(for more details, see http://expy.sf.net/)


"""Python math module by expy-cxpy."""
from expy import * 
expymodule(__name__)

#includes, defines, etc.
@prologue 
def myprolog():
   return """
#include <math.h>
"""

@function(double) #return type: double
def sqrt(x=double): #argument x: double
   """sqrt(x) --> the square root of x."""
   return "sqrt(x)" #as an expression

#a more terse way
@function(double)
def sin(x=double):
   """sin(x) --> the sin of x."""
   pass #the deduced call: sin(x)

#more functions ...

expymodule(__name__) #end of module


      

From steve at holdenweb.com  Tue Jul 21 20:34:59 2009
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 21 Jul 2009 14:34:59 -0400
Subject: [Python-Dev] Support for Python/Windows
Message-ID: <4A660A53.4050806@holdenweb.com>

Devs:

I've been in correspondence with Microsoft about the provision of
software, and it transpires that if you want to support Windows better
Microsoft will be quite liberal about licensing: they will *give* you a
Microsoft Developer Network license.

If you are interested in offering better Windows support then please
read the email below (note: Windows buildbot support would be a
qualifying activity) and let me have the required details. I will pass
them to Tom in bulk to simplify the processing.

Note that I'm not following python-dev right now due to pressure of
work, so PLEASE EMAIL ME DIRECTLY (or Cc me on your list replies) to
make sure I get your information.

Thanks!

regards
 Steve

-------- Original Message --------
Subject: RE: Support for Python: Windows Buildbots
Date: Tue, 7 Jul 2009 08:52:10 -0700
From: Tom Hanrahan

For the purposes of providing MSDN licenses to an open source
development community, I consider anyone who writes, builds, tests or
documents software to be a "developer who contributes" to the project.
(In fact, having started out as a test engineer, I would take exception
to anyone who claimed only people who write code are "developers" :-)
We do ask that requests are for people who are active contributors and
not just minor/occasional participants. [...]

Here's what we need for each request:

First Name
Last Name
Email Address (the subscription will be sent here, and this will also be
used to log into the MSDN site)
Project/Company (Python Software Foundation)
Complete Mailing Address
     (Street Address)
     (City), (State/Province)
     (Postal or Zip Code)
     (Country)
Phone Number

-- Tom

-----Original Message-----
From: Steve Holden [mailto:steve at holdenweb.com]
Sent: Tuesday, July 07, 2009 6:01 AM
To: Tom Hanrahan
Cc: Anandeep Pannu; Pat Campbell; Python Board; Jim Hugunin
Subject: Re: Support for Python: Windows Buildbots

Tom:

Further to Sam's email, in fact the original inquiry was instituted by
the need of our part-time administrator to acquire an Office license. I
am guessing she wouldn't qualify as an Open Source Developer, but that
least naturally to the more interesting question of who would.

A Bing search for "Microsoft Open Source Developer Program" didn't yield
any usable hits, so it might be helpful if you could point me to some
web resources that will help me make sense of what's available, who's
eligible and how they apply for it. I will be happy to publicize the
details to the development team.

It's true, I believe, that most of the core Python developers use Linux,
but both Tim Peters and I are primarily on the Windows platform. What's
more, with the emergence of virtualization environments having Linux on
your desktop is no hindrance to running Windows in a virtual machine (I
run Linux on virtuals when appropriate).

So let's take it from here and see where we go.

regards
 Steve

Sam Ramji wrote:
> Steve:
> 
> If the PSF's Windows users are developers who contribute to Python, we can offer them MSDN subscriptions as part of the Open Source Developer program.
> 
> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software.  For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). 
> 
> Tom Hanrahan is the Director of the Open Source Technology Center, and is the sponsor of the Open Source Developer program.  I've copied him here - please contact him directly with the details of the people who would like to participate. He's at hanrahat at microsoft.com. 
> 
> We definitely want to make Windows a friendly place for Python developers!  
> 
> Cheers,
> 
> Sam
> 
[...]
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
Holden Web LLC                 http://www.holdenweb.com/
Watch PyCon on video now!          http://pycon.blip.tv/


From phd at phd.pp.ru  Tue Jul 21 20:52:03 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 21 Jul 2009 22:52:03 +0400
Subject: [Python-Dev] expy: an expressway to extend Python
In-Reply-To: <765806.59082.qm@web54202.mail.re2.yahoo.com>
References: <765806.59082.qm@web54202.mail.re2.yahoo.com>
Message-ID: <20090721185203.GA16546@phd.pp.ru>

On Tue, Jul 21, 2009 at 11:22:19AM -0700, Yingjie Lan wrote:
> @function(double) #return type: double
> def sqrt(x=double): #argument x: double

   Python 3.0 has arguments and return value annotations:

http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax
http://www.python.org/dev/peps/pep-3107/

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From eric at ericentin.com  Tue Jul 21 22:26:52 2009
From: eric at ericentin.com (Eric Entin)
Date: Tue, 21 Jul 2009 16:26:52 -0400
Subject: [Python-Dev] expy: an expressway to extend Python
In-Reply-To: <20090721185203.GA16546@phd.pp.ru>
References: <765806.59082.qm@web54202.mail.re2.yahoo.com>
	<20090721185203.GA16546@phd.pp.ru>
Message-ID: <f4f77c440907211326y5012bafbo505aea86954049ae@mail.gmail.com>

I think the point of his software is to make it easier to interface Python
with C code, although his examples were probably not the best.
Eric Entin

On Tue, Jul 21, 2009 at 2:52 PM, Oleg Broytmann <phd at phd.pp.ru> wrote:

> On Tue, Jul 21, 2009 at 11:22:19AM -0700, Yingjie Lan wrote:
> > @function(double) #return type: double
> > def sqrt(x=double): #argument x: double
>
>    Python 3.0 has arguments and return value annotations:
>
> http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax
> http://www.python.org/dev/peps/pep-3107/
>
> Oleg.
> --
>     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/eric%40ericentin.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090721/8db68e5b/attachment.htm>

From phd at phd.pp.ru  Tue Jul 21 22:45:10 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Wed, 22 Jul 2009 00:45:10 +0400
Subject: [Python-Dev] expy: an expressway to extend Python
In-Reply-To: <f4f77c440907211326y5012bafbo505aea86954049ae@mail.gmail.com>
References: <765806.59082.qm@web54202.mail.re2.yahoo.com>
	<20090721185203.GA16546@phd.pp.ru>
	<f4f77c440907211326y5012bafbo505aea86954049ae@mail.gmail.com>
Message-ID: <20090721204510.GA22163@phd.pp.ru>

On Tue, Jul 21, 2009 at 04:26:52PM -0400, Eric Entin wrote:
> I think the point of his software is to make it easier to interface Python
> with C code

   I think I understand that. And I think this

> > > @function(double) #return type: double
> > > def sqrt(x=double): #argument x: double

   is how C functions are declared in Python, so I think annotations is the
way to go for such declarations.

> >    Python 3.0 has arguments and return value annotations:
> >
> > http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax
> > http://www.python.org/dev/peps/pep-3107/

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From lists at cheimes.de  Tue Jul 21 23:36:50 2009
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 21 Jul 2009 23:36:50 +0200
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A660A53.4050806@holdenweb.com>
References: <4A660A53.4050806@holdenweb.com>
Message-ID: <4A6634F2.1040003@cheimes.de>

Steve Holden wrote:
> Devs:
> 
> I've been in correspondence with Microsoft about the provision of
> software, and it transpires that if you want to support Windows better
> Microsoft will be quite liberal about licensing: they will *give* you a
> Microsoft Developer Network license.
> 
> If you are interested in offering better Windows support then please
> read the email below (note: Windows buildbot support would be a
> qualifying activity) and let me have the required details. I will pass
> them to Tom in bulk to simplify the processing.

[...]

>> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software.  For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). 

Thanks you for getting in touch with Microsoft. The deal is worth a
fortune for any Windows developer!

Does the MSDN subscription also include the permission to create and
release binaries? Sam Ramji wrote just "developing and testing". Me and
probably all other subscribers like to use the MSDN subscription to
build Windows binaries of Python and Python related extensions. Can you
please verify that we are allowed to use the subscription for that
purpose, too?

Christian

From curt at hagenlocher.org  Tue Jul 21 23:53:20 2009
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Tue, 21 Jul 2009 14:53:20 -0700
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A6634F2.1040003@cheimes.de>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
Message-ID: <d2155e360907211453l7e8cfad4q6ad51bbe7a572870@mail.gmail.com>

On Tue, Jul 21, 2009 at 2:36 PM, Christian Heimes <lists at cheimes.de> wrote:
>
> Does the MSDN subscription also include the permission to create and
> release binaries? Sam Ramji wrote just "developing and testing". Me and
> probably all other subscribers like to use the MSDN subscription to
> build Windows binaries of Python and Python related extensions. Can you
> please verify that we are allowed to use the subscription for that
> purpose, too?

Considering that the compilers are all freely downloadable as part of
the Windows SDK, it would be weird if the subscription were *more*
restrictive than what you can get without it.

Disclaimer: I work for Microsoft, but eh, I'm just guessing.

--
Curt Hagenlocher
curt at hagenlocher.org

From lists at cheimes.de  Wed Jul 22 00:04:05 2009
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 22 Jul 2009 00:04:05 +0200
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <d2155e360907211453l7e8cfad4q6ad51bbe7a572870@mail.gmail.com>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
	<d2155e360907211453l7e8cfad4q6ad51bbe7a572870@mail.gmail.com>
Message-ID: <4A663B55.7000500@cheimes.de>

Curt Hagenlocher wrote:
> Considering that the compilers are all freely downloadable as part of
> the Windows SDK, it would be weird if the subscription were *more*
> restrictive than what you can get without it.
> 
> Disclaimer: I work for Microsoft, but eh, I'm just guessing.

Some features like PGO aren't available in VS Express Edition or Windows
SDK. I'm not sure about all aspects of X86_64 builds, too. I'm prefer
better safe than sorry.

Disclaimer: I would never disclose that I'm a tiny bit paranoid ... :)

Christian

From steve at holdenweb.com  Wed Jul 22 00:32:20 2009
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 21 Jul 2009 18:32:20 -0400
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A6634F2.1040003@cheimes.de>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
Message-ID: <4A6641F4.9000200@holdenweb.com>

Christian Heimes wrote:
> Steve Holden wrote:
>> Devs:
>>
>> I've been in correspondence with Microsoft about the provision of
>> software, and it transpires that if you want to support Windows better
>> Microsoft will be quite liberal about licensing: they will *give* you a
>> Microsoft Developer Network license.
>>
>> If you are interested in offering better Windows support then please
>> read the email below (note: Windows buildbot support would be a
>> qualifying activity) and let me have the required details. I will pass
>> them to Tom in bulk to simplify the processing.
> 
> [...]
> 
>>> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software.  For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). 
> 
> Thanks you for getting in touch with Microsoft. The deal is worth a
> fortune for any Windows developer!
> 
> Does the MSDN subscription also include the permission to create and
> release binaries? Sam Ramji wrote just "developing and testing". Me and
> probably all other subscribers like to use the MSDN subscription to
> build Windows binaries of Python and Python related extensions. Can you
> please verify that we are allowed to use the subscription for that
> purpose, too?
> 
I'll ask. I don't see why not (it would hardly be in Microsoft's
interest to help us create unreleasable open source projects, would it?)

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
Holden Web LLC                 http://www.holdenweb.com/
Watch PyCon on video now!          http://pycon.blip.tv/


From curt at hagenlocher.org  Wed Jul 22 01:08:34 2009
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Tue, 21 Jul 2009 16:08:34 -0700
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A663B55.7000500@cheimes.de>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
	<d2155e360907211453l7e8cfad4q6ad51bbe7a572870@mail.gmail.com>
	<4A663B55.7000500@cheimes.de>
Message-ID: <d2155e360907211608w1f80f86dm94229a3b9dbfbbdf@mail.gmail.com>

On Tue, Jul 21, 2009 at 3:04 PM, Christian Heimes<lists at cheimes.de> wrote:
>
> Some features like PGO aren't available in VS Express Edition or Windows
> SDK. I'm not sure about all aspects of X86_64 builds, too. I'm prefer
> better safe than sorry.

Ah, you're right -- the PGO bits probably need VS Pro. The 64-bit
compilers should be in the Windows SDK, but it wouldn't surprise me if
they were not included in Express.

--
Curt Hagenlocher
curt at hagenlocher.org

From nyamatongwe at gmail.com  Wed Jul 22 01:19:29 2009
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Wed, 22 Jul 2009 09:19:29 +1000
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <d2155e360907211608w1f80f86dm94229a3b9dbfbbdf@mail.gmail.com>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
	<d2155e360907211453l7e8cfad4q6ad51bbe7a572870@mail.gmail.com>
	<4A663B55.7000500@cheimes.de>
	<d2155e360907211608w1f80f86dm94229a3b9dbfbbdf@mail.gmail.com>
Message-ID: <50862ebd0907211619p6c8b71edxcfd1307d4c83a835@mail.gmail.com>

Curt Hagenlocher:

> Ah, you're right -- the PGO bits probably need VS Pro. The 64-bit
> compilers should be in the Windows SDK, but it wouldn't surprise me if
> they were not included in Express.

   64-bit isn't in Express and merging the 64 bit compiler from the
SDK into Express may be possible but certainly isn't easy. I just use
the command line compiler to check 64 bit issues.

   Neil

From david.lyon at preisshare.net  Wed Jul 22 01:38:20 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 21 Jul 2009 19:38:20 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?=
In-Reply-To: <h44ukm$fuk$1@ger.gmane.org>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
Message-ID: <742787a67a8690d390c83a11a10aeb2a@preisshare.net>

On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver at palladion.com>
wrote:
>>> The main problem imho has been that easy_install by default
>>> has installed to /usr/lib/pythonX.X/site-packages when in
>>> fact that really is an operating system controlled directory.
>>>
>>> So naughty naughty to setuptools....
..
> That behvior isn't setuptools' fault:  it is the behavior of *any*
> distutils-derived app.  "OS controlled directory" is not a concept which
> the distutils honors at all:  $prefix/lib/pytoonX.Y/site-packages is the
> *intended* installation location for any modules not shipped with Python.

Ok - so naughty distutils...

> Consider the case where the user has built a separate python (e.g., to
> avoid conflicting with the OS version) in /opt/PythonX.y:  are you
> actually saying that distutils / setuptools should somehow know the
> difference betweeen /usr/lib/pythonX.y/site-packages and
> /opt/PythonX.Y/lib/pythonX.Y/site-packages?  

It already does.

If you run the python in /opt/pythonX.X you'll get a different value
for sys.path than when you run the other python from /usr/lib/pythonX.X.

Try it and see for yourself..

That's because sys.path is calculated relatively within site.py
depending on where the interpretor is run from.

> Or are you asserting that
> it is somehow invalid for distutils / setuptools to install things into
> /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an
> "OS controlled directory."

Not at all.

All I'm saying is that after so many years of python under linux and
operating system we should have observed the common use cases and
learned a thing or two.

If you hadn't noticed, there is mayhem in package installation because
the operating system tries to install its packages in the same place
as setuptools does in development mode.

That has got to be asking for trouble..

When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/
site-packages and I'm wondering why the hubba setuptools/distutils
doesn't put packages there by default. That would solve a lot of
problems.

Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S.

Wrapping up.. most of the functionality that I'm asking for is
already there. So I don't require anything extra. I don't want
things taken out.

All I'm suggesting is that we use what we already have a bit smarter.

David



From lanyjie at yahoo.com  Wed Jul 22 03:15:45 2009
From: lanyjie at yahoo.com (Yingjie Lan)
Date: Tue, 21 Jul 2009 18:15:45 -0700 (PDT)
Subject: [Python-Dev] expy: an expressway to extend Python
Message-ID: <728371.42493.qm@web54201.mail.re2.yahoo.com>


--- On Wed, 7/22/09, Oleg Broytmann <phd at phd.pp.ru> wrote:

> From: Oleg Broytmann <phd at phd.pp.ru>
> Subject: Re: [Python-Dev] expy: an expressway to extend Python
> To: python-dev at python.org
> Date: Wednesday, July 22, 2009, 12:45 AM
> On Tue, Jul 21, 2009 at 04:26:52PM
> -0400, Eric Entin wrote:
> > I think the point of his software is to make it easier
> to interface Python
> > with C code
> 
> ???I think I understand that. And I think
> this
> 
> > > > @function(double) #return type: double
> > > > def sqrt(x=double): #argument x: double
> 
> ???is how C functions are declared in
> Python, so I think annotations is the
> way to go for such declarations.
> 
> > >? ? Python 3.0 has arguments and return
> value annotations:
> > >
> > > http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax
> > > http://www.python.org/dev/peps/pep-3107/
> 

Thanks, I think that is a brilliant suggestion,
when expy is implemented for Python 3.0, this 
will greatly improve readability, and make it 
more like a natural part of Python.

Yingjie


      

From foom at fuhm.net  Wed Jul 22 03:46:11 2009
From: foom at fuhm.net (James Y Knight)
Date: Tue, 21 Jul 2009 21:46:11 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from pythonpkgmgr's point of view]
In-Reply-To: <742787a67a8690d390c83a11a10aeb2a@preisshare.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
Message-ID: <F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>

On Jul 21, 2009, at 7:38 PM, David Lyon wrote:
> When I go into python on ubuntu I see there is /usr/local/pythonX.X/ 
> lib/
> site-packages and I'm wondering why the hubba setuptools/distutils
> doesn't put packages there by default. That would solve a lot of
> problems.
>
> Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S.

Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu  
have switched to install their distribution-supplied python libraries  
into:
/usr/lib/pythonX.Y/lib/dist-packages
and distutils by default will install into
/usr/local/lib/pythonX.Y/dist-packages

starting with python 2.6.

See:
http://lists.debian.org/debian-devel/2009/02/msg00431.html

Since that email says "Discussed this with Barry Warsaw and Martin v.  
Loewis", I'd assume this change would be more widely known in the  
distutils/python-dev community, but apparently not??

James

From david.lyon at preisshare.net  Wed Jul 22 03:57:46 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 21 Jul 2009 21:57:46 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?=
In-Reply-To: <F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
Message-ID: <de7fb138098a5a79c9602b0aa2d8bf42@preisshare.net>

On Tue, 21 Jul 2009 21:46:11 -0400, James Y Knight <foom at fuhm.net> wrote:
> Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu  
> have switched to install their distribution-supplied python libraries  
> into:
> /usr/lib/pythonX.Y/lib/dist-packages
> and distutils by default will install into
> /usr/local/lib/pythonX.Y/dist-packages
> 
> starting with python 2.6.
> ..
> Since that email says "Discussed this with Barry Warsaw and Martin v.  
> Loewis", I'd assume this change would be more widely known in the  
> distutils/python-dev community, but apparently not??

Forgive my ignorance...

I'm working with 2.5 on ubuntu and didn't think to try 2.6...

It's good to know that the problem is fixed in 2.6 but I still need to
make things work for pythonpkgmgr on ubuntu py2.5 as 2.5 is the default
for ubuntu 8.10.

Take care

David

From mal at egenix.com  Wed Jul 22 10:49:20 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 22 Jul 2009 10:49:20 +0200
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
Message-ID: <4A66D290.9050103@egenix.com>

James Y Knight wrote:
> On Jul 21, 2009, at 7:38 PM, David Lyon wrote:
>> When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/
>> site-packages and I'm wondering why the hubba setuptools/distutils
>> doesn't put packages there by default. That would solve a lot of
>> problems.
>>
>> Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S.
> 
> Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu have
> switched to install their distribution-supplied python libraries into:
> /usr/lib/pythonX.Y/lib/dist-packages
> and distutils by default will install into
> /usr/local/lib/pythonX.Y/dist-packages
> 
> starting with python 2.6.
> 
> See:
> http://lists.debian.org/debian-devel/2009/02/msg00431.html
> 
> Since that email says "Discussed this with Barry Warsaw and Martin v.
> Loewis", I'd assume this change would be more widely known in the
> distutils/python-dev community, but apparently not??

Debian has a long history of doing this different, so it's
not much of a surprise. They also apply such changes to
Python packages.

However, all of this is non-standard and will cause problems
with tools that rely on the standard site-packages/ location. Such
changes should be discouraged.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 22 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From olemis at gmail.com  Wed Jul 22 15:48:04 2009
From: olemis at gmail.com (Olemis Lang)
Date: Wed, 22 Jul 2009 08:48:04 -0500
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A6641F4.9000200@holdenweb.com>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
	<4A6641F4.9000200@holdenweb.com>
Message-ID: <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com>

Hi !

On Tue, Jul 21, 2009 at 5:32 PM, Steve Holden<steve at holdenweb.com> wrote:
> Christian Heimes wrote:
>> Steve Holden wrote:
>>> Devs:
>>>
>>> If you are interested in offering better Windows support then please
>>> read the email below
>>
>> [...]
>>
>>>> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software. ?For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx).
>>
>> Thanks you for getting in touch with Microsoft. The deal is worth a
>> fortune for any Windows developer!
>>
>> Does the MSDN subscription also include the permission to create and
>> release binaries?
[...]
>> Can you
>> please verify that we are allowed to use the subscription for that
>> purpose, too?
>>
> I'll ask. I don't see why not (it would hardly be in Microsoft's
> interest to help us create unreleasable open source projects, would it?)
>

When talking about MS + FOSS everything is possible :-/

My question is the following :

- What are the implications for Py users ?

I mean, even if somebody (not me but enterprises & organizations I
work or may work for in the future ;o) decides to use Windows pay for
that and everything else, I'd not like to qualify as a "pirate" (or
alike) for using a Py distribution or app including MS Intelectual
Property (MSIP) (and MS loves MSIP -even if nobody can see it- and all
kind of legal issues, especially with FOSS) nor even have Py in the
middle of a patent dispute or something ...

And they have some "great" innovations [1]_ to ensure (sometimes, I
know) that (some) apps (who decides ?) wont run on a Win host. I could
mention a lot of snippets in that text (yes it's very "interesting"
and "substantial", and "useful" ) here goes one of them :

{{{
According to another aspect of the invention, the digest catalog
includes, for each program file corresponding to an application or
driver that should be executable by the computer, a digitally signed
hash value that is generated from a hash function based on the
corresponding program file. When attempting to load a particular file,
the loader generates a hash value and compares it to the decrypted
hash values in the digest catalog. If the comparison results in no
matches, then the corresponding program file (and thus the application
or driver) is not loaded.
}}}

OTOH :

- What are the implications for other devs (not core ;o) who use to
download sources and try new things, or perhaps use Py code the way
they want to solve an specific issue, or modify it somehow to
experiment or learn something, or whatever ?
- Will that affect contributions from ?future or potential? devs ?
- Will they also need an MS license to see or compile (or whatever)
the changes contributed by Py devs ?
- What about if for some reason, a idea or impl or alg or snippet (or
whatever) is propagated to GNU/Linux distributions and it's MSIP?
(considering former disputes like ?Linux kernel violates 42 of MS
patents?) ?


.. [1] Restricted software and hardware usage on a computer
        (http://patft.uspto.gov/netacgi/nph-Parser?patentnumber=7,536,726)

PD: My question is not technical at all but at least for me is
important (even if I'm not a lawyer, nor a core Py dev  ;o) since I
manage (and develop ;o) several Py-based apps running on Win hosts in
different locations .

Finally I clearly see that this msg is strongly influenced by my
biases, paranoia, and maybe I'm overreacting ... but I prefer to ask
before things actually happen (and MS has a long history specially
with FOSS + patents + legal affaires).

I apologize in advance if I'm being rude or na?ve or *

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

From eric.pruitt at gmail.com  Wed Jul 22 16:01:19 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Wed, 22 Jul 2009 09:01:19 -0500
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and WriteFile
Message-ID: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>

Hello,

I have written replacements for a few of Mark Hammond's PyWin32 functions
using ctypes to call upon the Windows kernel API. Code can be found here;
http://pastebin.com/m46de01f . When calling ReadFile, it appears that the
kernel API converts '/n' newlines '/r/n' newlines. I have not been able to
find any information about this but as far as I can tell, there is nothing
in my code that is causing the problem. Initially I suspected it related to
newline translation but the function in subprocess.Popen for translating
newlines only converts to '/n' newlines. When I replaced my ReadFile and
WriteFile functions with the PyWin32 functions I was imitating, the problem
still arose. I was hoping someone could confirm this problem for me from
experience or by testing out my code. If you would like the see the
functions used in full context, I have a Mercurial Google Code project that
can be checked out at
http://code.google.com/p/subprocdev/source/list(branch "python3k"). My
work entails modifications to subprocess.py so when
running the code, please update your PYTHONPATH variable so that my
subprocess.py will be imported.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090722/b0cf7808/attachment.htm>

From tseaver at palladion.com  Wed Jul 22 16:26:59 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 22 Jul 2009 10:26:59 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <742787a67a8690d390c83a11a10aeb2a@preisshare.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
Message-ID: <h477jl$vch$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Lyon wrote:
> On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver at palladion.com>
> wrote:

>> Consider the case where the user has built a separate python (e.g., to
>> avoid conflicting with the OS version) in /opt/PythonX.y:  are you
>> actually saying that distutils / setuptools should somehow know the
>> difference betweeen /usr/lib/pythonX.y/site-packages and
>> /opt/PythonX.Y/lib/pythonX.Y/site-packages?  
> 
> It already does.
> 
> If you run the python in /opt/pythonX.X you'll get a different value
> for sys.path than when you run the other python from /usr/lib/pythonX.X.
> 
> Try it and see for yourself..

You missed my point:  I know that sys.prefix etc. is set based on the
location of the executable.  You were complaining that setuptools /
distutils wants to put files in an "OS-controlled directory":  I was
asking how Python was supposed to know the difference between
/usr/lib/python2.x/site-packages (presumably "OS-controlled") and
/opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled").

The solution is to avoid doing distutils / setuptools operations as
root, *ever*, and instead use either an alternate install location (such
as /opt/Python2.x) or use something like virtualenv, which provides a
"safe" sandbox for installing stuff.

> That's because sys.path is calculated relatively within site.py
> depending on where the interpretor is run from.
> 
>> Or are you asserting that
>> it is somehow invalid for distutils / setuptools to install things into
>> /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an
>> "OS controlled directory."
> 
> Not at all.
> 
> All I'm saying is that after so many years of python under linux and
> operating system we should have observed the common use cases and
> learned a thing or two.
> 
> If you hadn't noticed, there is mayhem in package installation because
> the operating system tries to install its packages in the same place
> as setuptools does in development mode.
> 
> That has got to be asking for trouble..

Users who install anything into /usr/lib/python2.x in "development" mode
has already invited all the trouble they can handle into the house.

> When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/
> site-packages and I'm wondering why the hubba setuptools/distutils
> doesn't put packages there by default. That would solve a lot of
> problems.
> 
> Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S.
> 
> Wrapping up.. most of the functionality that I'm asking for is
> already there. So I don't require anything extra. I don't want
> things taken out.
> 
> All I'm suggesting is that we use what we already have a bit smarter.

That is precisely how I think of virtualenv: it sits on top of distutils
/ setuptools, and makes it sane to do Python development without
compromising the core system.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKZyGz+gerLs4ltQ4RAupxAJ95RZQJm6t1FtcFY+DMyHHmwRjmSACglo+W
TehyHzxUwHQ5LfdibJsgUcA=
=nf13
-----END PGP SIGNATURE-----


From david.lyon at preisshare.net  Wed Jul 22 16:54:16 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 22 Jul 2009 10:54:16 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?=
In-Reply-To: <h477jl$vch$1@ger.gmane.org>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<h477jl$vch$1@ger.gmane.org>
Message-ID: <b327d780cce9f9688680c664ab856b55@preisshare.net>

On Wed, 22 Jul 2009 10:26:59 -0400, Tres Seaver <tseaver at palladion.com>
wrote:
> You were complaining that setuptools /
> distutils wants to put files in an "OS-controlled directory":  

I complain about many things - so what.

> I was
> asking how Python was supposed to know the difference between
> /usr/lib/python2.x/site-packages (presumably "OS-controlled") and
> /opt/Python2.x/lib/python2.x/site-packages (definitely not
> "OS-controlled").

Well, at an interpreter level it doesn't, and I don't expect it
to.

> The solution is to avoid doing distutils / setuptools operations as
> root, *ever*, and instead use either an alternate install location (such
> as /opt/Python2.x) 

Have you tried this ?

When I try to run it without su privelages setuptools crashes... (ubuntu
8.10)

------------------------------------------------------------------------------------------------
Running installer ... /usr/bin/easy_install "/media/Elements/Pythn
Packages/html5lib-0.10.zip"
ERRORS: Traceback (most recent call last):
  File "/usr/bin/easy_install", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78,
in apport_excepthook
    report_file = open(pr_filename, 'wt')
IOError: [Errno 13] Permission denied:
'/var/crash/_usr_bin_easy_install.1000.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/bin/easy_install", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
------------------------------------------------------------------------------------------------

I can only get it to work with sudo.

> That is precisely how I think of virtualenv: it sits on top of distutils
> / setuptools, and makes it sane to do Python development without
> compromising the core system.

For me I prefer using a tool with a GUI interface.

Take care

David


From amauryfa at gmail.com  Wed Jul 22 17:33:35 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 22 Jul 2009 17:33:35 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and 	WriteFile
In-Reply-To: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>
Message-ID: <e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>

Hi,

2009/7/22 Eric Pruitt <eric.pruitt at gmail.com>:
> Hello,
>
> I have written replacements for a few of Mark Hammond's PyWin32 functions
> using ctypes to call upon the Windows kernel API. Code can be found here;
> http://pastebin.com/m46de01f . When calling ReadFile, it appears that the
> kernel API converts '/n' newlines '/r/n' newlines. I have not been able to
> find any information about this but as far as I can tell, there is nothing
> in my code that is causing the problem. Initially I suspected it related to
> newline translation but the function in subprocess.Popen for translating
> newlines only converts to '/n' newlines. When I replaced my ReadFile and
> WriteFile functions with the PyWin32 functions I was imitating, the problem
> still arose. I was hoping someone could confirm this problem for me from
> experience or by testing out my code. If you would like the see the
> functions used in full context, I have a Mercurial Google Code project that
> can be checked out at http://code.google.com/p/subprocdev/source/list
> (branch "python3k"). My work entails modifications to subprocess.py so when
> running the code, please update your PYTHONPATH variable so that my
> subprocess.py will be imported.

These questions should be redirected to comp.lang.python.

But as a quick response, the subprocess stdout is likely to be opened
in text mode.
So reading \r\n is not a surprise to me.

-- 
Amaury Forgeot d'Arc

From tismer at stackless.com  Wed Jul 22 20:36:00 2009
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 22 Jul 2009 11:36:00 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
Message-ID: <4A675C10.6060200@stackless.com>

Hi all,

I was hacking to get mingw32 builds of psyco to work
and found a pretty weird thing:

I used mingw32 (stable distro) to build the psyco extension
on top of standard python2.6, built with Visual Studio,
and got weird crashes.

The reason is in objimpl.h:

typedef union _gc_head {
	struct {
		union _gc_head *gc_next;
		union _gc_head *gc_prev;
		Py_ssize_t gc_refs;
	} gc;
	long double dummy;  /* force worst-case alignment */
} PyGC_Head;

Mingw32 behaves funny here. The size of long double is 12 ! 8-)

I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead
of the function, and that caused the errors, because psython
and the extension disagreed on the location of the gc pointers...
Using PyObject_GC_Untrack instead fixed the problem,
but there is a bad feeling left.

Question:
Is that considered a mingw bug?
Should we change the above union to a safer construct?

Or maybe I just missed something obvious and made a fool out of me?

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From solipsis at pitrou.net  Wed Jul 22 21:40:08 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 22 Jul 2009 19:40:08 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>
Message-ID: <loom.20090722T193713-455@post.gmane.org>

Hi,

> I used mingw32 (stable distro) to build the psyco extension
> on top of standard python2.6, built with Visual Studio,
> and got weird crashes.
> 
> The reason is in objimpl.h:
> 
> typedef union _gc_head {
> 	struct {
> 		union _gc_head *gc_next;
> 		union _gc_head *gc_prev;
> 		Py_ssize_t gc_refs;
> 	} gc;
> 	long double dummy;  /* force worst-case alignment */
> } PyGC_Head;
> 
> Mingw32 behaves funny here. The size of long double is 12 ! 

The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees
with Visual Studio on some ABI subtleties (is it expected? is mingw supposed to
be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)).

Regards

Antoine.



From martin at v.loewis.de  Wed Jul 22 21:43:45 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 22 Jul 2009 21:43:45 +0200
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com>
References: <4A660A53.4050806@holdenweb.com>
	<4A6634F2.1040003@cheimes.de>	<4A6641F4.9000200@holdenweb.com>
	<24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com>
Message-ID: <4A676BF1.4000303@v.loewis.de>

> My question is the following :
> 
> - What are the implications for Py users ?

After reading your message, I had a difficult time getting your point,
or even finding out what your question is.

So I stick with what you said is your question: What are the
implications for Py users ?

To this, the answer is mostly: none at all. There may be vague indirect
effects (such as more Python software being available on Windows), but
I doubt it will be significantly noticeable.

> I mean, even if somebody (not me but enterprises & organizations I
> work or may work for in the future ;o) decides to use Windows pay for
> that and everything else, I'd not like to qualify as a "pirate" (or
> alike) for using a Py distribution or app including MS Intelectual
> Property (MSIP) (and MS loves MSIP -even if nobody can see it- and all
> kind of legal issues, especially with FOSS) nor even have Py in the
> middle of a patent dispute or something ...

I can't follow here. Why would using a Py distribution or app that
includes MSIP qualify you as a "pirate"? And why would Microsoft's
making free (as in beer) tools available to Python contributors
make you qualify more as a "pirate" than you are currently qualified
as?

> - What are the implications for other devs (not core ;o) who use to
> download sources and try new things, or perhaps use Py code the way
> they want to solve an specific issue, or modify it somehow to
> experiment or learn something, or whatever ?

They can now get tools for free that they previously had to buy.

> - Will that affect contributions from ?future or potential? devs ?

This is an indirect effect; I doubt there is any noticable change
(in particular as VS Express is free (as in beer) already).

> - Will they also need an MS license to see or compile (or whatever)
> the changes contributed by Py devs ?

Not more than currently already, no. You may not be aware that Python
is *already* compiled by MSVC on Windows.

> - What about if for some reason, a idea or impl or alg or snippet (or
> whatever) is propagated to GNU/Linux distributions and it's MSIP?
> (considering former disputes like ?Linux kernel violates 42 of MS
> patents?) ?

Python contributors should always have the copyright to all changes
they contribute. They should not contribute code owned by Microsoft,
nor should they contribute code owned by anybody else (except
themselves). The latter is the bigger problem: sometimes people
contribute code that is owned by their employer, without an agreement
of the employer.

> I apologize in advance if I'm being rude or na?ve or *

I didn't consider your message rude. It is perhaps na?ve (apparently
ignoring the status quo), but you don't have to apologize for that.

Regards,
Martin


From glenn at zewt.org  Wed Jul 22 21:45:29 2009
From: glenn at zewt.org (Glenn Maynard)
Date: Wed, 22 Jul 2009 15:45:29 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from pythonpkgmgr's point of view]
In-Reply-To: <h477jl$vch$1@ger.gmane.org>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<h477jl$vch$1@ger.gmane.org>
Message-ID: <bd36f99e0907221245v7138c7d0w86f8ef25f346de68@mail.gmail.com>

On Wed, Jul 22, 2009 at 10:26 AM, Tres Seaver<tseaver at palladion.com> wrote:
> You missed my point: ?I know that sys.prefix etc. is set based on the
> location of the executable. ?You were complaining that setuptools /
> distutils wants to put files in an "OS-controlled directory": ?I was
> asking how Python was supposed to know the difference between
> /usr/lib/python2.x/site-packages (presumably "OS-controlled") and
> /opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled").

autoconf--for all its warts--solved this ages ago.  On installation,
you specify what prefix you want to install stuff into; if you don't,
you get a sensible default (/usr/local).  If you're creating an OS
package, or if installing directly into /usr is otherwise really what
you really want to do, you specify --prefix=/usr.  The Python core
doesn't need to know anything about this; it just needs to have a
sys.path that includes both /usr/lib/... and /usr/local/lib/....

This works for everything else--C headers, libraries, manpages, shared
libraries.  Why is Python special?

> The solution is to avoid doing distutils / setuptools operations as
> root, *ever*, and instead use either an alternate install location (such
> as /opt/Python2.x) or use something like virtualenv, which provides a
> "safe" sandbox for installing stuff.

This is nasty.  I should certainly be able to globally install
packages on my own, without creating a whole new Python installation
or other hacks, just like I can install anything else into /usr/local.

-- 
Glenn Maynard

From martin at v.loewis.de  Wed Jul 22 21:55:24 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 22 Jul 2009 21:55:24 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A675C10.6060200@stackless.com>
References: <4A675C10.6060200@stackless.com>
Message-ID: <4A676EAC.7080206@v.loewis.de>

> Is that considered a mingw bug?

Perhaps. I can't test right now: *if* VS has a long double type
that is smaller, it is apparently an ABI violation for mingw to
not follow VS. It appears that other people also consider it a
bug:

http://www.velocityreviews.com/forums/showpost.php?p=1514923&postcount=2

OTOH, they probably cannot change it without causing

> Should we change the above union to a safer construct?

No. tim_one changed it to be long double in r25454 to support some
system that Dave Abrahams uses, so it needs to stay that way :-)

However, we can certainly acknowledge that this is a bug in MingW,
and special case it. Either introduce a symbolic type gchead_align_t
which gets defined to just double on MingW, or put the #ifdef right
into the structure.

It might also be useful to assert that sizeof(gchead_align_t) is
8 or 16, and reject 12 as a value. The point is that we need the
maximum alignment, and that certainly shouldn't be 12.

Regards,
Martin


From foom at fuhm.net  Wed Jul 22 22:40:29 2009
From: foom at fuhm.net (James Y Knight)
Date: Wed, 22 Jul 2009 16:40:29 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from pythonpkgmgr's point of view]
In-Reply-To: <4A66D290.9050103@egenix.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
Message-ID: <CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>


On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote:

> Debian has a long history of doing this different, so it's
> not much of a surprise. They also apply such changes to
> Python packages.
>
> However, all of this is non-standard and will cause problems
> with tools that rely on the standard site-packages/ location. Such
> changes should be discouraged.

And yet, the change seems to have some strong reasoning, solves the  
problem discussed in this thread, and was apparently discussed and  
approved of by some core python developers before being implemented.  
It seems a bit foolish to me to thus just dismiss it as "evil debian  
being different"...

If anything it seems like it's a failure of the Python project to make  
easily deployable software, compounded with a failure of communication  
within the python community.

James

From mal at egenix.com  Wed Jul 22 23:22:56 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 22 Jul 2009 23:22:56 +0200
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 -	from pythonpkgmgr's point of view]
In-Reply-To: <CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
Message-ID: <4A678330.7010103@egenix.com>

James Y Knight wrote:
> 
> On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote:
> 
>> Debian has a long history of doing this different, so it's
>> not much of a surprise. They also apply such changes to
>> Python packages.
>>
>> However, all of this is non-standard and will cause problems
>> with tools that rely on the standard site-packages/ location. Such
>> changes should be discouraged.
> 
> And yet, the change seems to have some strong reasoning, solves the
> problem discussed in this thread, and was apparently discussed and
> approved of by some core python developers before being implemented. It
> seems a bit foolish to me to thus just dismiss it as "evil debian being
> different"...

Maybe I've misunderstood some important detail, but how will
their "change" help with anything other than making their
distribution a non-standard Python installation ?

distutils allows for a great deal of flexibility. For some
reason, this does not appear to be known to a larger
audience.

I can only recommend reading Greg's great write-up about the
end-user perspective of installing Python modules:

    http://docs.python.org/install/

A little known fact is that distutils can easily be customized
using config files:

    http://docs.python.org/install/#distutils-configuration-files

Together with the PYTHONHOME and PYTHONPATH environment
variables setup in your .bashrc, this gives you full flexibility
regarding the Python package setup and permits setting up
private Python installations, parallel Python installations,
Python installations sharing packages, etc. etc.

It really depends a lot on what you want to achieve. There's
no one fits all configuration.

Using a non-standard Python installation as system Python is
certainly not a good approach to solving anything. It only
makes it harder for users having to fight problems arising
out of this.

You have the same problem in other areas as well. A prominent
example being Microsoft's tendency (in the past) to tweak standards
to better meet their requirements.

> If anything it seems like it's a failure of the Python project to make
> easily deployable software, compounded with a failure of communication
> within the python community.

I lost you there.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 22 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From glenn at zewt.org  Thu Jul 23 01:08:26 2009
From: glenn at zewt.org (Glenn Maynard)
Date: Wed, 22 Jul 2009 19:08:26 -0400
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
	- from pythonpkgmgr's point of view]
In-Reply-To: <4A678330.7010103@egenix.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
Message-ID: <bd36f99e0907221608i460f4b4aq51b2c5cec90c2370@mail.gmail.com>

On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg<mal at egenix.com> wrote:
> Maybe I've misunderstood some important detail, but how will
> their "change" help with anything other than making their
> distribution a non-standard Python installation ?

I think I'm a little confused, too, because Python supports the
/usr|/usr/local separation just fine (setup.py install
--prefix=/usr/local).

It seems like it's also using "dist-packages" instead of
"site-packages".  That part, I don't understand at all--distribution
packages should go in /usr/lib/pythonx.y/site-packages, and "site"
packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself*
means "non-distribution site-installed stuff".  If that's what you're
referring to, then at least on first impression I agree.

-- 
Glenn Maynard

From david.lyon at preisshare.net  Thu Jul 23 01:10:19 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 22 Jul 2009 19:10:19 -0400
Subject: [Python-Dev]
 =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?=
 =?utf-8?q?istutils=5D_PEP_376_-=09from_pythonpkgmgr=27s_point_of_view=5D?=
In-Reply-To: <4A678330.7010103@egenix.com>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
Message-ID: <423776adb90814245468ea5d7d809f26@preisshare.net>

On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" <mal at egenix.com> wrote:
> Maybe I've misunderstood some important detail, but how will
> their "change" help with anything other than making their
> distribution a non-standard Python installation ?

The Debian/ubuntu distribution isn't non-standard. If anything
I'd prefer to suggest that it is in many ways "a standard"

Here's a sys.pth from a mac...

['', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin',

'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac',

'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages',

'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload',

'/Users/david/.local/lib/python2.6/site-packages', 
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages']

You can see that there are many choices along sys.path for installing
packages.

> distutils allows for a great deal of flexibility. For some
> reason, this does not appear to be known to a larger
> audience.

People forget command lines - that's why.

> I can only recommend reading Greg's great write-up about the
> end-user perspective of installing Python modules:
> 
>     http://docs.python.org/install/

It's good documentation of course. Cheers to Greg but the
old method is so tedious. That really is the hard way.

pythonpkgmgr offers a much easier solution by wrapping
easy_install and/or pip. You just type in parts of the
package name into a search box, click [search], a search
of pypi is done, click [install] and your package is
downloaded and installed.

It's a much more modern way of doing package installation
and requires absolutely no typing on a command line.

> A little known fact is that distutils can easily be customized
> using config files:
> 
>     http://docs.python.org/install/#distutils-configuration-files

Perphaps.

But it seems only for advanced users.. and I couldn't figure
out on the face of it what advantage it would have.

Take care

David




From bugtrack at roumenpetrov.info  Thu Jul 23 01:45:51 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Thu, 23 Jul 2009 02:45:51 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A675C10.6060200@stackless.com>
References: <4A675C10.6060200@stackless.com>
Message-ID: <4A67A4AF.1060802@roumenpetrov.info>

Christian Tismer wrote:
> Hi all,
> 
> I was hacking to get mingw32 builds of psyco to work
> and found a pretty weird thing:
> 
> I used mingw32 (stable distro) to build the psyco extension
> on top of standard python2.6, built with Visual Studio,
> and got weird crashes.
> 
> The reason is in objimpl.h:
> 
> typedef union _gc_head {
>     struct {
>         union _gc_head *gc_next;
>         union _gc_head *gc_prev;
>         Py_ssize_t gc_refs;
>     } gc;
>     long double dummy;  /* force worst-case alignment */
> } PyGC_Head;
> 
> Mingw32 behaves funny here. The size of long double is 12 ! 8-)

No it is correct. GNU C compiler for windows support 80-bit extended 
precision.


> I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead
> of the function, and that caused the errors, because psython
> and the extension disagreed on the location of the gc pointers...
> Using PyObject_GC_Untrack instead fixed the problem,
> but there is a bad feeling left.
> 
> Question:
> Is that considered a mingw bug?

No as sizeof(long double) = sizeof(double) is MSVC limitation.


> Should we change the above union to a safer construct?
> 
> Or maybe I just missed something obvious and made a fool out of me?
> 
> cheers - chris
> 

Regards,
Roumen

From bugtrack at roumenpetrov.info  Thu Jul 23 01:56:01 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Thu, 23 Jul 2009 02:56:01 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A676EAC.7080206@v.loewis.de>
References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de>
Message-ID: <4A67A711.7020006@roumenpetrov.info>

Martin v. L?wis wrote:
[SNIP]
> No. tim_one changed it to be long double in r25454 to support some
> system that Dave Abrahams uses, so it needs to stay that way :-)
> 
> However, we can certainly acknowledge that this is a bug in MingW,
> and special case it. Either introduce a symbolic type gchead_align_t
> which gets defined to just double on MingW, or put the #ifdef right
> into the structure.

No this is not GCC bug. GCC support "hardware extended precision" as 
implement long double and mingw w32api implement long double functions.
According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is 
MSVC limitation.


> It might also be useful to assert that sizeof(gchead_align_t) is
> 8 or 16, and reject 12 as a value. The point is that we need the
> maximum alignment, and that certainly shouldn't be 12.

So look like python bug.


> Regards,
> Martin

Regards,
Roumen


From david.lyon at preisshare.net  Thu Jul 23 01:53:29 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 22 Jul 2009 19:53:29 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
Message-ID: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>

Hi Greg,

I'm on the python-dev mailing list and somebody gave me a link
to a page that you have done:

  http://docs.python.org/install/

Can I ask that you also provide a link for windows users
to my project:

  http://sourceforge.net/projects/pythonpkgmgr/

Our project provides an alternative to command line installation.

And can save a lot of time for users when they wish to
install packages.

Thanks

David


From ben+python at benfinney.id.au  Thu Jul 23 02:00:25 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 23 Jul 2009 10:00:25 +1000
Subject: [Python-Dev] Remove site-packages?!?
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>
	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>
	<4A62B820.4030903@gmail.com>
	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>
	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
Message-ID: <87k5206zfq.fsf@benfinney.id.au>

"M.-A. Lemburg" <mal at egenix.com> writes:

> James Y Knight wrote:
> > And yet, the change seems to have some strong reasoning, solves the
> > problem discussed in this thread, and was apparently discussed and
> > approved of by some core python developers before being implemented.
> > It seems a bit foolish to me to thus just dismiss it as "evil debian
> > being different"...
> 
> Maybe I've misunderstood some important detail, but how will
> their "change" help with anything other than making their
> distribution a non-standard Python installation ?

Before this discussion descends further into circular speculation on the
Debian Python developers's intentions, perhaps someone should ask them
directly: <URL:http://lists.debian.org/debian-python/>.

-- 
 \           ?Let others praise ancient times; I am glad I was born in |
  `\                                      these.? ?Ovid (43 BCE?18 CE) |
_o__)                                                                  |
Ben Finney


From tismer at stackless.com  Thu Jul 23 02:13:51 2009
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 22 Jul 2009 17:13:51 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A67A711.7020006@roumenpetrov.info>
References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de>
	<4A67A711.7020006@roumenpetrov.info>
Message-ID: <4A67AB3F.2060405@stackless.com>

On 7/22/09 4:56 PM, Roumen Petrov wrote:
> Martin v. L?wis wrote:
> [SNIP]
>> No. tim_one changed it to be long double in r25454 to support some
>> system that Dave Abrahams uses, so it needs to stay that way :-)
>>
>> However, we can certainly acknowledge that this is a bug in MingW,
>> and special case it. Either introduce a symbolic type gchead_align_t
>> which gets defined to just double on MingW, or put the #ifdef right
>> into the structure.
>
> No this is not GCC bug. GCC support "hardware extended precision" as
> implement long double and mingw w32api implement long double functions.
> According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is
> MSVC limitation.
>
>
>> It might also be useful to assert that sizeof(gchead_align_t) is
>> 8 or 16, and reject 12 as a value. The point is that we need the
>> maximum alignment, and that certainly shouldn't be 12.
>
> So look like python bug.

The assumption is that the union with long double gives alignment
to the largest possible integral type with a power of 2 size,
which is then wrong, because of the unexpected size.

What do you propose for doing proper alignment, then?

I fear this needs to become yet another special case in pyconfig.h

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From cournape at gmail.com  Thu Jul 23 02:17:41 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 23 Jul 2009 09:17:41 +0900
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090722T193713-455@post.gmane.org>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
Message-ID: <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>

On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou<solipsis at pitrou.net> wrote:

>
> The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees
> with Visual Studio

Yes, mingw and VS do not have the same long double type. This has been
the source of some problems in numpy as well, since mingw uses the MS
runtime, and everything involving long double and the runtime is
broken (printf, math library calls). I wish there was a way to disable
this in mingw, but there isn't AFAIK.

> on some ABI subtleties (is it expected? is mingw supposed to
> be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)).

I think mostly ABI compatible is the best description :)

David

From tismer at stackless.com  Thu Jul 23 02:28:51 2009
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 22 Jul 2009 17:28:51 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
Message-ID: <4A67AEC3.6070000@stackless.com>

On 7/22/09 5:17 PM, David Cournapeau wrote:
> On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou<solipsis at pitrou.net>  wrote:
>
>> The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees
>> with Visual Studio
>
> Yes, mingw and VS do not have the same long double type. This has been
> the source of some problems in numpy as well, since mingw uses the MS
> runtime, and everything involving long double and the runtime is
> broken (printf, math library calls). I wish there was a way to disable
> this in mingw, but there isn't AFAIK.
>
>> on some ABI subtleties (is it expected? is mingw supposed to
>> be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)).
>
> I think mostly ABI compatible is the best description :)

Well, the source of my problems is simply that I tried to build
an extension for a Visual Studio built Python, using mingw.
Did that, because it seems to be common practice so far.

Maybe the simple solution is to prevent building extensions
with mingw, if the python executable was not also built with it?
Then, all would be fine I guess.

Despite the fact that Python probably has to be changed:
If it is true then all the 32-bit Linux Pythons have a 12
byte GC head, IOW they are *all* badly aligned.
If that matters, of course.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From songofacandy at gmail.com  Thu Jul 23 03:15:16 2009
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 23 Jul 2009 10:15:16 +0900
Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI
In-Reply-To: <49C9454A.7090600@v.loewis.de>
References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com>
	<gq70q2$hul$1@ger.gmane.org> <49C70C6A.7070106@g.nevcal.com>
	<gq8mu2$mfl$1@ger.gmane.org> <49C7FF1F.6080707@g.nevcal.com>
	<ee9557410903241016h2f1e7b1cp4afdf96afd7f293f@mail.gmail.com>
	<49C9454A.7090600@v.loewis.de>
Message-ID: <ee9557410907221815j65381fd2uc33eae4e2d969da9@mail.gmail.com>

Hi.

>> We have to use "IME" to input non-ascii charactor in Windows.
>> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
>>
>> So setting codepage to 65001 make output universal but make input ascii-only.
>> Sit!!!
>
> Is there a code page that still allows IME input, but supports all
> of Unicode? I know there is GB18030 - is it any good?

I found WriteConsoleW() API recently.
This API can write utf16 string to console directly, without change
OutputCodepage.

example:
http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42

I think this API is good for py3k.
When stdout is console and not redirected to [pipe|file],
sys.stdout.write(u"foo")
can avoid encoding and use WriteConsoleW(L"foo")

-- 
Naoki INADA  <songofacandy at gmail.com>

From bugtrack at roumenpetrov.info  Thu Jul 23 03:46:32 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Thu, 23 Jul 2009 04:46:32 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A67AB3F.2060405@stackless.com>
References: <4A675C10.6060200@stackless.com>
	<4A676EAC.7080206@v.loewis.de>	<4A67A711.7020006@roumenpetrov.info>
	<4A67AB3F.2060405@stackless.com>
Message-ID: <4A67C0F8.6020902@roumenpetrov.info>

Christian Tismer wrote:
> On 7/22/09 4:56 PM, Roumen Petrov wrote:
>> Martin v. L?wis wrote:
>> [SNIP]
>>> No. tim_one changed it to be long double in r25454 to support some
>>> system that Dave Abrahams uses, so it needs to stay that way :-)
>>>
>>> However, we can certainly acknowledge that this is a bug in MingW,
>>> and special case it. Either introduce a symbolic type gchead_align_t
>>> which gets defined to just double on MingW, or put the #ifdef right
>>> into the structure.
>>
>> No this is not GCC bug. GCC support "hardware extended precision" as
>> implement long double and mingw w32api implement long double functions.
>> According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is
>> MSVC limitation.
>>
>>
>>> It might also be useful to assert that sizeof(gchead_align_t) is
>>> 8 or 16, and reject 12 as a value. The point is that we need the
>>> maximum alignment, and that certainly shouldn't be 12.
>>
>> So look like python bug.
> 
> The assumption is that the union with long double gives alignment
> to the largest possible integral type with a power of 2 size,
> which is then wrong, because of the unexpected size.

As is posted for GCC on linux(32-bit) size of structure is 12 = 
sizeof(struct in union) = sizeof(pointer)+sizeof(pointer)+sizeof(ssize_t).

And sizeof(ssize_t) <= sizeof(long) <= sizeof(pointer) .


> What do you propose for doing proper alignment, then?

May be "void* dummy[4]" is better for force alignment ?

What about alignment on platforms with pointers > 32 bit ?


> I fear this needs to become yet another special case in pyconfig.h
> 
> cheers - chris
> 

Regards,
Roumen

From martin at v.loewis.de  Thu Jul 23 05:21:09 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Jul 2009 05:21:09 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A67A711.7020006@roumenpetrov.info>
References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de>
	<4A67A711.7020006@roumenpetrov.info>
Message-ID: <4A67D725.6060503@v.loewis.de>

Roumen Petrov wrote:
> Martin v. L?wis wrote:
> [SNIP]
>> No. tim_one changed it to be long double in r25454 to support some
>> system that Dave Abrahams uses, so it needs to stay that way :-)
>>
>> However, we can certainly acknowledge that this is a bug in MingW,
>> and special case it. Either introduce a symbolic type gchead_align_t
>> which gets defined to just double on MingW, or put the #ifdef right
>> into the structure.
> 
> No this is not GCC bug. GCC support "hardware extended precision" as
> implement long double and mingw w32api implement long double functions.

And that exactly is the bug. GCC cannot simultaneously be
Windows-compatible, and also support extended precision floating point
under the name "long double".

> According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is
> MSVC limitation.

Therefore, mingw should also implement that limitation, to be
binary-compatible with MSVC. Since binary compatibility is an explicit
goal of mingw, failing to provide binary compatibility is a bug.

Regards,
Martin

From dalcinl at gmail.com  Thu Jul 23 05:23:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 00:23:56 -0300
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and 	WriteFile
In-Reply-To: <e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>
	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>
Message-ID: <e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>

On Wed, Jul 22, 2009 at 12:33 PM, Amaury Forgeot
d'Arc<amauryfa at gmail.com> wrote:
>
> These questions should be redirected to comp.lang.python.
>

Eric seems to be working on a GSoC for PFS related to improving
subprocess. Even in such case this list is not the right place to make
these posts??



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From martin at v.loewis.de  Thu Jul 23 05:26:20 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Jul 2009 05:26:20 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A67C0F8.6020902@roumenpetrov.info>
References: <4A675C10.6060200@stackless.com>	<4A676EAC.7080206@v.loewis.de>	<4A67A711.7020006@roumenpetrov.info>	<4A67AB3F.2060405@stackless.com>
	<4A67C0F8.6020902@roumenpetrov.info>
Message-ID: <4A67D85C.1010809@v.loewis.de>

>> What do you propose for doing proper alignment, then?
> 
> May be "void* dummy[4]" is better for force alignment ?

That would have the same alignment as a single pointer.

> What about alignment on platforms with pointers > 32 bit ?

The C compiler will choose the alignment of the union to be the
alignment of the union branch that requires that largest alignment.
So if the largest alignment is the one that a pointer has,
then alignof(PyGC_HEAD) == alignof(gc_next). The double is in the
union only in case it has larger alignment than a pointer.

Regards,
Martin

From martin at v.loewis.de  Thu Jul 23 05:36:03 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Jul 2009 05:36:03 +0200
Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI
In-Reply-To: <ee9557410907221815j65381fd2uc33eae4e2d969da9@mail.gmail.com>
References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com>	<gq70q2$hul$1@ger.gmane.org>
	<49C70C6A.7070106@g.nevcal.com>	<gq8mu2$mfl$1@ger.gmane.org>
	<49C7FF1F.6080707@g.nevcal.com>	<ee9557410903241016h2f1e7b1cp4afdf96afd7f293f@mail.gmail.com>	<49C9454A.7090600@v.loewis.de>
	<ee9557410907221815j65381fd2uc33eae4e2d969da9@mail.gmail.com>
Message-ID: <4A67DAA3.1050909@v.loewis.de>

> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

I think this would be fairly difficult to implement given
the way the output libraries work. If you think it can be
done, please provide a patch to bugs.python.org.

Regards,
Martin


From martin at v.loewis.de  Thu Jul 23 05:33:24 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Jul 2009 05:33:24 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
Message-ID: <4A67DA04.70305@v.loewis.de>

> Can I ask that you also provide a link for windows users
> to my project:
> 
>   http://sourceforge.net/projects/pythonpkgmgr/

I fail to see how this project is relevant in the context
of explaining distutils. So you would have to come up with
a proposal of specific wording that makes the relevance more
apparent.

Please provide a patch to bugs.python.org.

Regards,
Martin

From ben+python at benfinney.id.au  Thu Jul 23 09:03:00 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 23 Jul 2009 17:03:00 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
Message-ID: <87bpnb7ufv.fsf@benfinney.id.au>

David Lyon <david.lyon at preisshare.net> writes:

> I'm on the python-dev mailing list and somebody gave me a link
> to a page that you have done:
> 
>   http://docs.python.org/install/

That's a document describing how to use ?distutils?, which is what every
recipient of Python will already have installed.

> Can I ask that you also provide a link for windows users
> to my project:
> 
>   http://sourceforge.net/projects/pythonpkgmgr/

That doesn't seem at all appropriate; promoting third-party packages
isn't at all what the above document should be doing.

> Our project provides an alternative to command line installation.

If people want an alternative to the standard tools provided in Python,
the documentation for Python itself is not the right place to be
looking. The Python Package Index <URL:http://pypi.python.org/> is the
right place to be promoting (free-software) third-party tools.

-- 
 \               ?There's no excuse to be bored. Sad, yes. Angry, yes. |
  `\    Depressed, yes. Crazy, yes. But there's no excuse for boredom, |
_o__)                                          ever.? ?Viggo Mortensen |
Ben Finney


From david.lyon at preisshare.net  Thu Jul 23 09:22:37 2009
From: david.lyon at preisshare.net (david.lyon at preisshare.net)
Date: Thu, 23 Jul 2009 17:22:37 +1000 (EST)
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <87bpnb7ufv.fsf@benfinney.id.au>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
Message-ID: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>


> That's a document describing how to use ?distutils?, which is what
> every
> recipient of Python will already have installed.
>
>> Can I ask that you also provide a link for windows users
>> to my project:
>>
>>   http://sourceforge.net/projects/pythonpkgmgr/
>
> That doesn't seem at all appropriate; promoting third-party packages
> isn't at all what the above document should be doing.

Distutils was once seperate and was then included in the standard python.

So i guess that I am working with the same goal in mind.

> If people want an alternative to the standard tools provided in Python,
The Python Package Index <URL:http://pypi.python.org/> is the
> right place to be promoting (free-software) third-party tools.

Well I can sure try that and thank you for the tip.

Btw, at the moment there exists no inbuilt mechanism within python for
retrieving packages from pypi.

Imho this is an omission for a so called 'batteries included' language.

Distutils is a builtin module for 'pushing' a developer package 'to' pypi.

But there is no corresponging mechanise for a user to 'pull' packages back.

Surely this is a gap in the standard distro?

So it is not inappropriate for me to ask about this on this list.

Take care

David



From aahz at pythoncraft.com  Thu Jul 23 05:03:00 2009
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 22 Jul 2009 20:03:00 -0700
Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI
In-Reply-To: <ee9557410907221815j65381fd2uc33eae4e2d969da9@mail.gmail.com>
References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com>
	<gq70q2$hul$1@ger.gmane.org> <49C70C6A.7070106@g.nevcal.com>
	<gq8mu2$mfl$1@ger.gmane.org> <49C7FF1F.6080707@g.nevcal.com>
	<ee9557410903241016h2f1e7b1cp4afdf96afd7f293f@mail.gmail.com>
	<49C9454A.7090600@v.loewis.de>
	<ee9557410907221815j65381fd2uc33eae4e2d969da9@mail.gmail.com>
Message-ID: <20090723030300.GA22273@panix.com>

On Thu, Jul 23, 2009, INADA Naoki wrote:
>
> I found WriteConsoleW() API recently.
> This API can write utf16 string to console directly, without change
> OutputCodepage.
> 
> example:
> http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42
> 
> I think this API is good for py3k.
> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

Please submit a feature request to bugs.python.org -- with a patch would
be even nicer, of course.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"The volume of a pizza of thickness 'a' and radius 'z' is
given by pi*z*z*a"

From ben+python at benfinney.id.au  Thu Jul 23 10:30:58 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 23 Jul 2009 18:30:58 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
Message-ID: <873a8n7qd9.fsf@benfinney.id.au>

david.lyon at preisshare.net writes:

> >> Can I ask that you also provide a link for windows users
> >> to my project:
> >>
> >>   http://sourceforge.net/projects/pythonpkgmgr/
> >
> > That doesn't seem at all appropriate; promoting third-party packages
> > isn't at all what the above document should be doing.
> 
> Distutils was once seperate and was then included in the standard
> python.
> 
> So i guess that I am working with the same goal in mind.

In which case you should work to get it accepted into standard Python
*before* asking for it to be promoted in the standard Python
documentation.

> So it is not inappropriate for me to ask about this on this list.

Discussing it here is appropriate. The modification you request is not,
IMO.

-- 
 \       ?It's easy to play any musical instrument: all you have to do |
  `\       is touch the right key at the right time and the instrument |
_o__)                        will play itself.? ?Johann Sebastian Bach |
Ben Finney


From mal at egenix.com  Thu Jul 23 10:34:30 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 23 Jul 2009 10:34:30 +0200
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <423776adb90814245468ea5d7d809f26@preisshare.net>
References: <fb6fbf560907181307lac01a3v269d79d55a7a233f@mail.gmail.com>	<d6d32580972d12da8cf34b4e6fcdea88@preisshare.net>	<4A62B820.4030903@gmail.com>	<Pine.LNX.4.64.0907190900080.6306@kimball.webabinitio.net>	<94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>	<h44ukm$fuk$1@ger.gmane.org>	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>	<4A66D290.9050103@egenix.com>	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>	<4A678330.7010103@egenix.com>
	<423776adb90814245468ea5d7d809f26@preisshare.net>
Message-ID: <4A682096.8010707@egenix.com>

David Lyon wrote:
> On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" <mal at egenix.com> wrote:
>> Maybe I've misunderstood some important detail, but how will
>> their "change" help with anything other than making their
>> distribution a non-standard Python installation ?
> 
> The Debian/ubuntu distribution isn't non-standard. If anything
> I'd prefer to suggest that it is in many ways "a standard"

It's not just Debian mangling their Python installation. Many
other Linux distros do the same.

Keeping up with all those changes has become so hard, that we have
chosen to always build our projects with private (regular)
Python installations on Linux and Macs.

Disk space is cheap and so is bandwidth. Compared to the hours
spent on resolving issues with system provided Python
installations, it's a no-brainer.

> Here's a sys.pth from a mac...
> 
> ['', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin',
> 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac',
> 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages',
> 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload',
> 
> '/Users/david/.local/lib/python2.6/site-packages', 
> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages']
> 
> You can see that there are many choices along sys.path for installing
> packages.

No, not really. The above is perfectly standard.

The directories on sys.path are setup by Python itself and make
sure that it can find the stdlib modules. Those entries are not
meant to be used as installation targets.

Python 2.6 has two standard places for installing packages:

1. In the stdlib site-packages/ subdir

2. In the user home dir's .local/lib/python2.6/site-packages dir

The second is a new feature in 2.6.

>> distutils allows for a great deal of flexibility. For some
>> reason, this does not appear to be known to a larger
>> audience.
> 
> People forget command lines - that's why.
> 
>> I can only recommend reading Greg's great write-up about the
>> end-user perspective of installing Python modules:
>>
>>     http://docs.python.org/install/
> 
> It's good documentation of course. Cheers to Greg but the
> old method is so tedious. That really is the hard way.

I think that's the main problem in all this... people
just keep on complaining regardless of what you give
them.

If people don't read documentation, are not willing
to at least spend a few minutes on considering what they
really want to do and then shout out loud when Python
doesn't do what they thought it should do (without telling
Python, of course), there's little hope.

> pythonpkgmgr offers a much easier solution by wrapping
> easy_install and/or pip. You just type in parts of the
> package name into a search box, click [search], a search
> of pypi is done, click [install] and your package is
> downloaded and installed.
> 
> It's a much more modern way of doing package installation
> and requires absolutely no typing on a command line.
> 
>> A little known fact is that distutils can easily be customized
>> using config files:
>>
>>     http://docs.python.org/install/#distutils-configuration-files
> 
> Perphaps.
> 
> But it seems only for advanced users.. and I couldn't figure
> out on the face of it what advantage it would have.

You define once and for all where you want your
Python packages to install themselves when typing
"python setup.py install". After that's done, you can
forget about all this and move on to get some real
work done ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 23 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Thu Jul 23 10:58:52 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 23 Jul 2009 18:58:52 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
Message-ID: <4A68264C.9060700@gmail.com>

david.lyon at preisshare.net wrote:
> Distutils is a builtin module for 'pushing' a developer package 'to' pypi.
> 
> But there is no corresponging mechanise for a user to 'pull' packages back.
> 
> Surely this is a gap in the standard distro?
> 
> So it is not inappropriate for me to ask about this on this list.

Raising it without at least glancing at the list archives which hold
copious amounts of virtual text on that topic is somewhat inappropriate
though :)

Cheers,
Nick.

P.S.
http://www.google.com/search?q=site:mail.python.org+inurl:python-dev+easy_install

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From floris.bruynooghe at gmail.com  Thu Jul 23 10:50:44 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 23 Jul 2009 09:50:44 +0100
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <bd36f99e0907221608i460f4b4aq51b2c5cec90c2370@mail.gmail.com>
References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
	<bd36f99e0907221608i460f4b4aq51b2c5cec90c2370@mail.gmail.com>
Message-ID: <20090723085044.GA3378@laurie.devork>

On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote:
> On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg<mal at egenix.com> wrote:
> > Maybe I've misunderstood some important detail, but how will
> > their "change" help with anything other than making their
> > distribution a non-standard Python installation ?
> 
> I think I'm a little confused, too, because Python supports the
> /usr|/usr/local separation just fine (setup.py install
> --prefix=/usr/local).
> 
> It seems like it's also using "dist-packages" instead of
> "site-packages".  That part, I don't understand at all--distribution
> packages should go in /usr/lib/pythonx.y/site-packages, and "site"
> packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself*
> means "non-distribution site-installed stuff".  If that's what you're
> referring to, then at least on first impression I agree.

The way Debian does this isn't that stupid.  It solves the problem of
the sysadmin intalling Python distibutions for the system Python
without writing in the system locations, which is a very reasonable
thing to do.

Before Debian used dist-packages,
/usr/local/lib/pythonX.Y/site-packages was added to the sys.path of
the system Python and this was the location where a sysadmin should
add packages (although distutils did not default to that location, so
it wasn so it was still easy to mess up the system package
management).  But this caused trouble for people who installed their
own Python in /usr/local since now you have the same
/usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters.

To solve this the system python moved to dist-packages instead, this
allows the local admin location to use
/usr/local/lib/pythonX.Y/dist-packages which does not conflict with
locally installed interpreters.  Together with this they changed
distutils to install to /usr/local/lib/pythonX.Y/dist-packages by
default, so that a sysadmin would have to go out of their way to mess
up the system package management.

This is a pretty neat solution to the problem.  But the weird thing
(IMHO) is that distutils-sig completely refuses to see this
requirement.  Generally it seems accepted that installing modules in
the system location (/usr/) is a bad thing, but last time it got
discussed there was a point blank refusal to accept that the local
admin needs a location in to install packages in.  I think it would be
great if Python instead provided a general guideline or best practice
for this situation which would be explained in the documentation.


Regards
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From solipsis at pitrou.net  Thu Jul 23 11:04:30 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Jul 2009 09:04:30 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
Message-ID: <loom.20090723T090139-199@post.gmane.org>

Christian Tismer <tismer <at> stackless.com> writes:
> 
> Despite the fact that Python probably has to be changed:
> If it is true then all the 32-bit Linux Pythons have a 12
> byte GC head, IOW they are *all* badly aligned.

Why are they badly aligned?
The fact that long double is 12 bytes long doesn't mean it will force a 12-byte
alignment - just whatever alignment is enough for a long double on the target
machine. This could be 4, 8 or 16 bytes.



From david.lyon at preisshare.net  Thu Jul 23 11:22:32 2009
From: david.lyon at preisshare.net (david.lyon at preisshare.net)
Date: Thu, 23 Jul 2009 19:22:32 +1000 (EST)
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <20090723085044.GA3378@laurie.devork>
References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
	<bd36f99e0907221608i460f4b4aq51b2c5cec90c2370@mail.gmail.com>
	<20090723085044.GA3378@laurie.devork>
Message-ID: <60480.203.171.195.84.1248340952.squirrel@syd-srv02.ezyreg.com>

Hi Floris,

That's exactly how I see it and i totally agree.

My contribution is to make a Package Manager Gui that tries to be
supportive of what you describe so well.

If i have any complaint about the state of affairs it would only be that
it takes a newcomer such a long time (months) to fully understand what
should be a 5 minute thing as you so well describe below.

> On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote:
>> On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg<mal at egenix.com> wrote:
>> > Maybe I've misunderstood some important detail, but how will
>> > their "change" help with anything other than making their
>> > distribution a non-standard Python installation ?
>>
>> I think I'm a little confused, too, because Python supports the
>> /usr|/usr/local separation just fine (setup.py install
>> --prefix=/usr/local).
>>
>> It seems like it's also using "dist-packages" instead of
>> "site-packages".  That part, I don't understand at all--distribution
>> packages should go in /usr/lib/pythonx.y/site-packages, and "site"
>> packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself*
>> means "non-distribution site-installed stuff".  If that's what you're
>> referring to, then at least on first impression I agree.
>
> The way Debian does this isn't that stupid.  It solves the problem of
> the sysadmin intalling Python distibutions for the system Python
> without writing in the system locations, which is a very reasonable
> thing to do.
>
> Before Debian used dist-packages,
> /usr/local/lib/pythonX.Y/site-packages was added to the sys.path of
> the system Python and this was the location where a sysadmin should
> add packages (although distutils did not default to that location, so
> it wasn so it was still easy to mess up the system package
> management).  But this caused trouble for people who installed their
> own Python in /usr/local since now you have the same
> /usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters.
>
> To solve this the system python moved to dist-packages instead, this
> allows the local admin location to use
> /usr/local/lib/pythonX.Y/dist-packages which does not conflict with
> locally installed interpreters.  Together with this they changed
> distutils to install to /usr/local/lib/pythonX.Y/dist-packages by
> default, so that a sysadmin would have to go out of their way to mess
> up the system package management.
>
> This is a pretty neat solution to the problem.  But the weird thing
> (IMHO) is that distutils-sig completely refuses to see this
> requirement.  Generally it seems accepted that installing modules in
> the system location (/usr/) is a bad thing, but last time it got
> discussed there was a point blank refusal to accept that the local
> admin needs a location in to install packages in.  I think it would be
> great if Python instead provided a general guideline or best practice
> for this situation which would be explained in the documentation.
>
>
> Regards
> Floris
>
>
> --
> Debian GNU/Linux -- The Power of Freedom
> www.debian.org | www.gnu.org | www.kernel.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/david.lyon%40preisshare.net
>



From david.lyon at preisshare.net  Thu Jul 23 11:43:14 2009
From: david.lyon at preisshare.net (david.lyon at preisshare.net)
Date: Thu, 23 Jul 2009 19:43:14 +1000 (EST)
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A68264C.9060700@gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<4A68264C.9060700@gmail.com>
Message-ID: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>


> Raising it without at least glancing at the list archives which hold
> copious amounts of virtual text on that topic is somewhat inappropriate
> though :)

Well I have consulted every available expert on the distutils list to the
point where I feel 'up' with the issues at hand.

They're great people.. And as helpful as they possibly can be..

But surely Python isn't only about archives and age old problems.

Sure they might be there.. So lets get in and fix them..

Which is totally what i'm attempting to do at the moment even if packaging
isnt perceived as being the most interesting part of python develepment
going on today.

Inapropriate or not, i want to donate my time to it.. Because i think we
need 'fresh' thinking - not archive regurgitation.


David


From p.f.moore at gmail.com  Thu Jul 23 11:49:03 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 23 Jul 2009 02:49:03 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A67AEC3.6070000@stackless.com>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
Message-ID: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>

2009/7/22 Christian Tismer <tismer at stackless.com>:
> Maybe the simple solution is to prevent building extensions
> with mingw, if the python executable was not also built with it?
> Then, all would be fine I guess.

I have never had problems in practice with extensions built with mingw
rather than MSVC - so while I'm not saying that the issue doesn't
exist, it certainly doesn't affect all extensions, so disabling mingw
support seems a bit of an extreme measure.

Having a preprocessor check in the Python headers to check if this bug
is triggered in the actual code, and raising a compile error in that
case, may be an option (but I expect that ensuring such a check is
precise enough would be hard).

Now that VC Express is freely available, the importance of mingw
support is less, but AFAIK many developers cross-compile Windows
binaries on Linux using mingw, or develop on gcc and use mingw to
build on Windows (or Wine) but wouldn't install VC. So removing mingw
support in cases where it does work would cause definite issues for
Windows users relying on prebuilt binaries.

Paul.

From floris.bruynooghe at gmail.com  Thu Jul 23 12:15:27 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 23 Jul 2009 11:15:27 +0100
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <4A682096.8010707@egenix.com>
References: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
	<423776adb90814245468ea5d7d809f26@preisshare.net>
	<4A682096.8010707@egenix.com>
Message-ID: <20090723101527.GA4322@laurie.devork>

On Thu, Jul 23, 2009 at 10:34:30AM +0200, M.-A. Lemburg wrote:
> Python 2.6 has two standard places for installing packages:
> 
> 1. In the stdlib site-packages/ subdir
> 
> 2. In the user home dir's .local/lib/python2.6/site-packages dir

And is missing a 3rd one.  The sysadmin who wants to install packages
globally has to use the first option here, but that conflicts with the
system package management on UNIXes.

Regards
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From lists at cheimes.de  Thu Jul 23 13:45:31 2009
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 23 Jul 2009 13:45:31 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and  WriteFile
In-Reply-To: <e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>
	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
Message-ID: <4A684D5B.5030104@cheimes.de>

Lisandro Dalcin wrote:
> Eric seems to be working on a GSoC for PFS related to improving
> subprocess. Even in such case this list is not the right place to make
> these posts??

Eric didn't say that he is working on a GSoC project for the PSF. Anyway
the Python general mailing list might still be a better place. IMHO he
can reach many more competent Python developers there who can help him
with this particular problem.

By the way I don't think that ctypes is the right way to go here. ctypes
is very handy if you need a quick solution. However I wouldn't use it as
a permanent solution for the subprocess module. It's tricky to get
ctypes based solutions right on multiple platforms (32 vs 64bit). It's
harder to debug a ctypes solutions rather than a C extension, too. I
assume that calling overhead is greater than a pure C extension. Can
ctypes release the GIL for a function call?

Christian

From amauryfa at gmail.com  Thu Jul 23 14:02:27 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 23 Jul 2009 14:02:27 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and WriteFile
In-Reply-To: <4A684D5B.5030104@cheimes.de>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>
	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>
	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
	<4A684D5B.5030104@cheimes.de>
Message-ID: <e27efe130907230502t20ccb588t6ed3e9eedea299e0@mail.gmail.com>

Hi,

2009/7/23 Christian Heimes <lists at cheimes.de>:
> Lisandro Dalcin wrote:
>> Eric seems to be working on a GSoC for PFS related to improving
>> subprocess. Even in such case this list is not the right place to make
>> these posts??
>
> Eric didn't say that he is working on a GSoC project for the PSF. Anyway
> the Python general mailing list might still be a better place. IMHO he
> can reach many more competent Python developers there who can help him
> with this particular problem.

Furthermore it was more a Windows question than a Python one.
Anyway we continued the discussion privately and I try to be helpful.

> By the way I don't think that ctypes is the right way to go here. ctypes
> is very handy if you need a quick solution. However I wouldn't use it as
> a permanent solution for the subprocess module. It's tricky to get
> ctypes based solutions right on multiple platforms (32 vs 64bit). It's
> harder to debug a ctypes solutions rather than a C extension, too. I
> assume that calling overhead is greater than a pure C extension.

And some distributions may choose to not distribute the ctypes module.
No stdlib module should require it.
OTOH ctypes is the perfect tool for rapid development with the win32
api, specially when the compiler is far away.

> Can ctypes release the GIL for a function call?

Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE

-- 
Amaury Forgeot d'Arc

From theller at ctypes.org  Thu Jul 23 14:04:44 2009
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 23 Jul 2009 14:04:44 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and  WriteFile
In-Reply-To: <4A684D5B.5030104@cheimes.de>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
	<4A684D5B.5030104@cheimes.de>
Message-ID: <h49jkr$fgs$1@ger.gmane.org>

Christian Heimes schrieb:

> Can ctypes release the GIL for a function call?

It will do that automatically, except for functions
using the pythonapi callign convention.

-- 
Thanks,
Thomas


From fuzzyman at voidspace.org.uk  Thu Jul 23 14:10:46 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 23 Jul 2009 13:10:46 +0100
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and WriteFile
In-Reply-To: <e27efe130907230502t20ccb588t6ed3e9eedea299e0@mail.gmail.com>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>	<4A684D5B.5030104@cheimes.de>
	<e27efe130907230502t20ccb588t6ed3e9eedea299e0@mail.gmail.com>
Message-ID: <4A685346.6000206@voidspace.org.uk>

Amaury Forgeot d'Arc wrote:
> Hi,
>
> 2009/7/23 Christian Heimes <lists at cheimes.de>:
>   
>> Lisandro Dalcin wrote:
>>     
>>> Eric seems to be working on a GSoC for PFS related to improving
>>> subprocess. Even in such case this list is not the right place to make
>>> these posts??
>>>       
>> Eric didn't say that he is working on a GSoC project for the PSF. Anyway
>> the Python general mailing list might still be a better place. IMHO he
>> can reach many more competent Python developers there who can help him
>> with this particular problem.
>>     
>
> Furthermore it was more a Windows question than a Python one.
> Anyway we continued the discussion privately and I try to be helpful.
>
>   
>> By the way I don't think that ctypes is the right way to go here. ctypes
>> is very handy if you need a quick solution. However I wouldn't use it as
>> a permanent solution for the subprocess module. It's tricky to get
>> ctypes based solutions right on multiple platforms (32 vs 64bit). It's
>> harder to debug a ctypes solutions rather than a C extension, too. I
>> assume that calling overhead is greater than a pure C extension.
>>     
>
> And some distributions may choose to not distribute the ctypes module.
> No stdlib module should require it.
>   

A big advantage of using ctypes is that it works cross-implementation - 
on IronPython and PyPy already and on Jython soon. I'd like to see more 
standard library modules use it. Distributions that choose not to 
include it are crippling their Python distribution.

Michael Foord

> OTOH ctypes is the perfect tool for rapid development with the win32
> api, specially when the compiler is far away.
>
>   
>> Can ctypes release the GIL for a function call?
>>     
>
> Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE
>
>   


-- 
http://www.ironpythoninaction.com/


From ncoghlan at gmail.com  Thu Jul 23 14:13:28 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 23 Jul 2009 22:13:28 +1000
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and  WriteFile
In-Reply-To: <4A684D5B.5030104@cheimes.de>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
	<4A684D5B.5030104@cheimes.de>
Message-ID: <4A6853E8.6060606@gmail.com>

Christian Heimes wrote:
> By the way I don't think that ctypes is the right way to go here. ctypes
> is very handy if you need a quick solution. However I wouldn't use it as
> a permanent solution for the subprocess module. It's tricky to get
> ctypes based solutions right on multiple platforms (32 vs 64bit). It's
> harder to debug a ctypes solutions rather than a C extension, too. I
> assume that calling overhead is greater than a pure C extension.

I see ctypes as largely useful when you want to call a native DLL but
don't have any existing infrastructure for accessing native code from
your project. A few lines of ctypes code is then a much better solution
than adding a C or C++ compilation dependency just to access a couple of
functions.

Of course, that definitely isn't the case for CPython - we not only have
plenty of existing C infrastructure, but in the specific case of
subprocess on Windows we already have a dedicated extension module
(PC/_subprocess.c).

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Jul 23 14:18:00 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 23 Jul 2009 22:18:00 +1000
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>
	<79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>
Message-ID: <4A6854F8.70308@gmail.com>

Paul Moore wrote:
> 2009/7/22 Christian Tismer <tismer at stackless.com>:
>> Maybe the simple solution is to prevent building extensions
>> with mingw, if the python executable was not also built with it?
>> Then, all would be fine I guess.
> 
> I have never had problems in practice with extensions built with mingw
> rather than MSVC - so while I'm not saying that the issue doesn't
> exist, it certainly doesn't affect all extensions, so disabling mingw
> support seems a bit of an extreme measure.

For runtime compatibility purposes, mingw is no worse than using
versions of MSVC other than the one that was used to build Python (i.e.
primarily making sure you use the correct memory management APIs and not
trying to call the FILE* APIs).

If there are other things that cause mingw to get structure sizes wrong
then I don't see a major problem with a config check or some #ifdef'ery
to work around it. A patch added to a tracker item would be a necessary
starting point for that idea though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From lists at cheimes.de  Thu Jul 23 14:21:38 2009
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 23 Jul 2009 14:21:38 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and WriteFile
In-Reply-To: <4A685346.6000206@voidspace.org.uk>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>	<4A684D5B.5030104@cheimes.de>
	<e27efe130907230502t20ccb588t6ed3e9eedea299e0@mail.gmail.com>
	<4A685346.6000206@voidspace.org.uk>
Message-ID: <4A6855D2.3030009@cheimes.de>

Michael Foord wrote:
> A big advantage of using ctypes is that it works cross-implementation -
> on IronPython and PyPy already and on Jython soon. I'd like to see more
> standard library modules use it. Distributions that choose not to
> include it are crippling their Python distribution.

Interesting, I didn't know that IronPython supports ctypes, too. I still
find ctypes a bit problematic because it doesn't us header files for its
types, structs and function definitions.

Christian

PS: I'n still reading your IronPython book. I hope I'll have some spare
time to write a review anytime soon.

From cournape at gmail.com  Thu Jul 23 14:22:27 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 23 Jul 2009 21:22:27 +0900
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>
Message-ID: <5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com>

On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore<p.f.moore at gmail.com> wrote:
> 2009/7/22 Christian Tismer <tismer at stackless.com>:
>> Maybe the simple solution is to prevent building extensions
>> with mingw, if the python executable was not also built with it?
>> Then, all would be fine I guess.
>
> I have never had problems in practice with extensions built with mingw
> rather than MSVC - so while I'm not saying that the issue doesn't
> exist, it certainly doesn't affect all extensions, so disabling mingw
> support seems a bit of an extreme measure.

I am strongly against this as well. We build numpy with mingw on
windows, and disabling it would make my life even more miserable on
windows. One constant source of pain with MS compilers is when
supporting different versions of python - 2.4, 2.5 and 2.6 require a
different VS version (and free versions are available only for the
last version of VS usually).

I am far from a windows specialist, but I understand that quite a few
problems with mingw-built extensions with python are caused by some
Python decisions as well (the C API with runtime-dependent structures
like FILE, etc...). So mingw is not the only to blame :)

David

From lists at cheimes.de  Thu Jul 23 14:23:56 2009
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 23 Jul 2009 14:23:56 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and  WriteFile
In-Reply-To: <4A6853E8.6060606@gmail.com>
References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com>	<e27efe130907220833o3bf39c6ah57b4de3459819be9@mail.gmail.com>	<e7ba66e40907222023k90135carec89cfeb9cefd562@mail.gmail.com>
	<4A684D5B.5030104@cheimes.de> <4A6853E8.6060606@gmail.com>
Message-ID: <4A68565C.3020500@cheimes.de>

Nick Coghlan wrote:
> I see ctypes as largely useful when you want to call a native DLL but
> don't have any existing infrastructure for accessing native code from
> your project. A few lines of ctypes code is then a much better solution
> than adding a C or C++ compilation dependency just to access a couple of
> functions.
> 
> Of course, that definitely isn't the case for CPython - we not only have
> plenty of existing C infrastructure, but in the specific case of
> subprocess on Windows we already have a dedicated extension module
> (PC/_subprocess.c).

You've hit the nail on the head! That's it.

Christian

From ncoghlan at gmail.com  Thu Jul 23 14:30:34 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 23 Jul 2009 22:30:34 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<4A68264C.9060700@gmail.com>
	<36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
Message-ID: <4A6857EA.4060002@gmail.com>

david.lyon at preisshare.net wrote:
>> Raising it without at least glancing at the list archives which hold
>> copious amounts of virtual text on that topic is somewhat inappropriate
>> though :)
> 
> Well I have consulted every available expert on the distutils list to the
> point where I feel 'up' with the issues at hand.

If you're actually up to speed with the issues, then I apologise. It was
just something of a novelty to see the topic brought up without
easy_install and setuptools even getting a mention.

However, the reason for the asymmetry has less to do with code
(easy_install exists after all) and more to do with the complexities of
system administration.

Providing a native ability to download and install packages from PyPI is
a major maintenance commitment due to a couple of major issues:

1. Providing an installation mechanism that is compatibility with a wide
variety of package management systems across at least Windows, Mac OS X
and the assorted flavours of *nix (Linux RPM, Linux APT, Solaris, *BSD,
etc, etc).

distutils cops a lot of heat already for not playing nicely with distro
packages. easy_install is loathed even more by many system
administrators (and that loathing often appears to flow over onto other
parts of setuptools).

2. There are some serious security implications in providing a native
mechanism for downloading, installing and running code in a
non-sandboxed environment.

The latter problem is probably the more technical of the two, but both
pose fairly complex social issues as well in terms of getting agreement
across disparate groups.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From exarkun at divmod.com  Thu Jul 23 14:55:14 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 23 Jul 2009 08:55:14 -0400
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and WriteFile
In-Reply-To: <4A6855D2.3030009@cheimes.de>
Message-ID: <20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com>

On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes <lists at cheimes.de> wrote:
>Michael Foord wrote:
>> A big advantage of using ctypes is that it works cross-implementation -
>> on IronPython and PyPy already and on Jython soon. I'd like to see more
>> standard library modules use it. Distributions that choose not to
>> include it are crippling their Python distribution.
>
>Interesting, I didn't know that IronPython supports ctypes, too. I still
>find ctypes a bit problematic because it doesn't us header files for its
>types, structs and function definitions.

This is indeed a big problem with ctypes.  Fortunately, a project exists
to correct it:

    http://pypi.python.org/pypi/ctypes_configure/0.1

Anyone writing code with ctypes should be looking at ctypes_configure as
well.

Jean-Paul

From exarkun at divmod.com  Thu Jul 23 14:57:33 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 23 Jul 2009 08:57:33 -0400
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
 and  WriteFile
In-Reply-To: <4A68565C.3020500@cheimes.de>
Message-ID: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>

On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes <lists at cheimes.de> wrote:
>Nick Coghlan wrote:
>> I see ctypes as largely useful when you want to call a native DLL but
>> don't have any existing infrastructure for accessing native code from
>> your project. A few lines of ctypes code is then a much better solution
>> than adding a C or C++ compilation dependency just to access a couple of
>> functions.
>>
>> Of course, that definitely isn't the case for CPython - we not only have
>> plenty of existing C infrastructure, but in the specific case of
>> subprocess on Windows we already have a dedicated extension module
>> (PC/_subprocess.c).
>
>You've hit the nail on the head! That's it.
>

True, CPython has C infrastructure.  What about the other Python runtimes,
though?  At the language summit, there was a lot of discussion (and, I
thought, agreement) about moving the standard library to be a collaborative
project between several of the major runtime projects.  A ctypes-based
solution seems more aligned with this goal than more C code.

Jean-Paul

From jnoller at gmail.com  Thu Jul 23 15:11:38 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 23 Jul 2009 09:11:38 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<4A68264C.9060700@gmail.com>
	<36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
Message-ID: <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com>

On Thu, Jul 23, 2009 at 5:43 AM, <david.lyon at preisshare.net> wrote:
>
>> Raising it without at least glancing at the list archives which hold
>> copious amounts of virtual text on that topic is somewhat inappropriate
>> though :)
>
> Well I have consulted every available expert on the distutils list to the
> point where I feel 'up' with the issues at hand.
>
> They're great people.. And as helpful as they possibly can be..
>
> But surely Python isn't only about archives and age old problems.
>
> Sure they might be there.. So lets get in and fix them..
>
> Which is totally what i'm attempting to do at the moment even if packaging
> isnt perceived as being the most interesting part of python develepment
> going on today.
>
> Inapropriate or not, i want to donate my time to it.. Because i think we
> need 'fresh' thinking - not archive regurgitation.
>
>
> David

Then why not include pip, easy_install, and this bash script I use to
install packages into core? The more the merrier, right?

Answer: None of these are standards, and as nick points out, there's
issues with sysadmins, security, and other things. Not to mention
they're fundamentally not part of the language.

At the language summit, this was hashed out quite a bit. I think most
everyone agreed that tools like easy_install, pip, virtualenv,
zc.buildout, etc simply do not belong in core python. The
"installation" landscape varies from platform to platform, it can run
full in the face of people who just want to use apt or yum, etc.

What *does* belong in core (distutils) however, and is something Tarek
has been working on, are APIs/Hooks/standards for tools like the ones
I've mentioned, and yours, to be able to do their job better, cleaner
and easier. Standards on metadata, uninstall hooks, etc are what the
stdlib should provide, not applications and tools built on top of
those things.

IMHO, the only "binary" python-core itself should "ship" is the python
binary itself (and pydoc, maybe :-)) - everything else should be built
with the idea of making integration with internals easier and
standard. That way OS package maintainers can hook into these APIs
(because they don't want to use something "non standard" to them),
people such as yourself can build GUI apps for downloading and
managing things, etc.

I know you want to help, and I don't think anyone is discouraging that
- but I think instead of solely focusing on your application is a
mistake. The bulk of the effort should be spent helping Tarek and
others "fix" distutils and spraying down bikeshedders so progress can
be made.

jesse

From jnoller at gmail.com  Thu Jul 23 15:14:43 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 23 Jul 2009 09:14:43 -0400
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and 	WriteFile
In-Reply-To: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
References: <4A68565C.3020500@cheimes.de>
	<20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
Message-ID: <4222a8490907230614v53d11614q6f059dadd0e35d70@mail.gmail.com>

On Thu, Jul 23, 2009 at 8:57 AM, Jean-Paul Calderone<exarkun at divmod.com> wrote:
> On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes <lists at cheimes.de>
> wrote:
>>
>> Nick Coghlan wrote:
>>>
>>> I see ctypes as largely useful when you want to call a native DLL but
>>> don't have any existing infrastructure for accessing native code from
>>> your project. A few lines of ctypes code is then a much better solution
>>> than adding a C or C++ compilation dependency just to access a couple of
>>> functions.
>>>
>>> Of course, that definitely isn't the case for CPython - we not only have
>>> plenty of existing C infrastructure, but in the specific case of
>>> subprocess on Windows we already have a dedicated extension module
>>> (PC/_subprocess.c).
>>
>> You've hit the nail on the head! That's it.
>>
>
> True, CPython has C infrastructure. ?What about the other Python runtimes,
> though? ?At the language summit, there was a lot of discussion (and, I
> thought, agreement) about moving the standard library to be a collaborative
> project between several of the major runtime projects. ?A ctypes-based
> solution seems more aligned with this goal than more C code.
>
> Jean-Paul

Yeah, I remember that too - all of us discussed "breaking out" the
stdlib post mercurial migration to be shared amongst all of the
implementations, marking CPython things as "CPython only" and so on.
That way we all have a common base to work from.

jesse

From olemis at gmail.com  Thu Jul 23 15:15:45 2009
From: olemis at gmail.com (Olemis Lang)
Date: Thu, 23 Jul 2009 08:15:45 -0500
Subject: [Python-Dev] Support for Python/Windows
In-Reply-To: <4A676BF1.4000303@v.loewis.de>
References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de>
	<4A6641F4.9000200@holdenweb.com>
	<24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com>
	<4A676BF1.4000303@v.loewis.de>
Message-ID: <24ea26600907230615s65500422kd9d722d2cffeb81c@mail.gmail.com>

On Wed, Jul 22, 2009 at 2:43 PM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>> My question is the following :
>>
>> - What are the implications for Py users ?
>
> So I stick with what you said is your question: What are the
> implications for Py users ?
>
> To this, the answer is mostly: none at all. There may be vague indirect
> effects (such as more Python software being available on Windows),
>

Well this being said, it seems to be cool.

>> - What are the implications for other devs (not core ;o) who use to
>> download sources and try new things, or perhaps use Py code the way
>> they want to solve an specific issue, or modify it somehow to
>> experiment or learn something, or whatever ?
>
> They can now get tools for free that they previously had to buy.
>

Well it seems that this applies to core devs and I was talking about
people not being Py core devs. But anyway, if everybody can still
compile Py sources without worrying about further licensing
side-effects (i.e. more than we have today ;) then the storm is gone.

>> - Will that affect contributions from ?future or potential? devs ?
>
> This is an indirect effect; I doubt there is any noticable change
> (in particular as VS Express is free (as in beer) already).
>

Well my concern (and what I didnt understand) was that if some
people, in this case core devs, (need | like to have) the
license to do (use)  something they cannot do (use) without the
license then (possibly) everybody else (i.e. those
not having the license) trying to reproduce what others did (e.g.
compilation) had to purchase the license.

If this is not the case, and non-core devs can do what they do the
way they do it so far, then the storm is over ...

>> - Will they also need an MS license to see or compile (or whatever)
>> the changes contributed by Py devs ?
>
> Not more than currently already, no.

... and it seems that's the case ;o)

> You may not be aware that Python
> is *already* compiled by MSVC on Windows.
>

Yes I am, but since I'm a frustrated lawyer I didnt understand a few
things (and I couldnt sleep yesterday because of that ... XD ...)

>> I apologize in advance if I'm being rude or na?ve or *
>
> I didn't consider your message rude. It is perhaps na?ve (apparently
> ignoring the status quo), but you don't have to apologize for that.
>

Well I wanted to avoid flamewars or unnecessary disputes or whatever
(you know, this licensing and FOSS vs proprietary debates may be
complicated and sometimes people get excited /me included, of course :
I'm human ;o)

Thnx !

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

From amauryfa at gmail.com  Thu Jul 23 16:05:40 2009
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 23 Jul 2009 16:05:40 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com>
	<5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com>
Message-ID: <e27efe130907230705x610b74cbx8a771b880b118286@mail.gmail.com>

2009/7/23 David Cournapeau <cournape at gmail.com>:
> On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore<p.f.moore at gmail.com> wrote:
>> 2009/7/22 Christian Tismer <tismer at stackless.com>:
>>> Maybe the simple solution is to prevent building extensions
>>> with mingw, if the python executable was not also built with it?
>>> Then, all would be fine I guess.
>>
>> I have never had problems in practice with extensions built with mingw
>> rather than MSVC - so while I'm not saying that the issue doesn't
>> exist, it certainly doesn't affect all extensions, so disabling mingw
>> support seems a bit of an extreme measure.
>
> I am strongly against this as well. We build numpy with mingw on
> windows, and disabling it would make my life even more miserable on
> windows. One constant source of pain with MS compilers is when
> supporting different versions of python - 2.4, 2.5 and 2.6 require a
> different VS version (and free versions are available only for the
> last version of VS usually).
>
> I am far from a windows specialist, but I understand that quite a few
> problems with mingw-built extensions with python are caused by some
> Python decisions as well (the C API with runtime-dependent structures
> like FILE, etc...). So mingw is not the only to blame :)

In this case, the OP tries to use an API that is explicitly documented
as dangerous for extension modules.
The recommended function is not sensitive to compiler differences.

-- 
Amaury Forgeot d'Arc

From stephen at xemacs.org  Thu Jul 23 16:53:04 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 23 Jul 2009 23:53:04 +0900
Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376
 - from pythonpkgmgr's point of view]
In-Reply-To: <20090723085044.GA3378@laurie.devork>
References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com>
	<40e590ca51381ba25fc8b6f2853d1325@preisshare.net>
	<a1c1f4750907210244o2985ed4esa89fa0b4dfbcede3@mail.gmail.com>
	<h44ukm$fuk$1@ger.gmane.org>
	<742787a67a8690d390c83a11a10aeb2a@preisshare.net>
	<F44AC7E0-B987-449E-8A2E-21943E646885@fuhm.net>
	<4A66D290.9050103@egenix.com>
	<CE72D1F6-6E7F-4177-8831-B8848812F74E@fuhm.net>
	<4A678330.7010103@egenix.com>
	<bd36f99e0907221608i460f4b4aq51b2c5cec90c2370@mail.gmail.com>
	<20090723085044.GA3378@laurie.devork>
Message-ID: <87tz13v4bz.fsf@uwakimon.sk.tsukuba.ac.jp>

Floris Bruynooghe writes:

 > [dist-packages] is a pretty neat solution to the problem.

To what problem?  I admit I am no expert on Python packaging, but my
experience with XEmacs suggests that this is the distro trying to help
with a *set* of problems that the user/sysadmins really should be
handling themselves.

There are varied requirements that lead to a need for yet another
place to install modules, but there's no guarantee that a given
user/sysadmin has *just one*.  With the Emacsen, often in trying to
satisfy several requirements they end up with conflicts (different
libraries in the site-lisp that are compiled for, or even require,
different versions of (X)Emacs or even trying to deal with Emacs and
XEmacs simultaneously).  And it's a pain (as an XEmacs maintainer) to
have to field questions about why things don't work in a set up I
didn't design, and don't understand the rationale for.

 > Generally it seems accepted that installing modules in the system
 > location (/usr/) is a bad thing, but last time it got discussed
 > there was a point blank refusal to accept that the local admin
 > needs a location in to install packages in.  I think it would be
 > great if Python instead provided a general guideline or best
 > practice for this situation which would be explained in the
 > documentation.

Yeah, so do we all, but I suspect that for Python, like the Emacsen,
there is *no* general guideline.  You're just asking for various kinds
of trouble, all of which require expert attention, by installing a
command that is invoked as "python" (as opposed to "pythonX.Y") in
/usr/local/bin unless you intend it to be a complete substitute for
the system python.  And that itself is a serious tour de force (as
anybody who remembers Red Hat's love affair with Python 1.5.2 will
agree, I suspect).

As far as I can see, *in general* /usr/local/lib/python/site-packages
is the right place to install local packages for use with the system
python, and no other.  That doesn't mean that there aren't *other*
valid uses for that location, just that I see no general rule for
supporting *all* of them at the same time.

From theller at ctypes.org  Thu Jul 23 17:35:12 2009
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 23 Jul 2009 17:35:12 +0200
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and WriteFile
In-Reply-To: <20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com>
References: <4A6855D2.3030009@cheimes.de>
	<20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com>
Message-ID: <h49vvf$q7m$1@ger.gmane.org>

Jean-Paul Calderone schrieb:
> On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes <lists at cheimes.de> wrote:
>>Michael Foord wrote:
>>> A big advantage of using ctypes is that it works cross-implementation -
>>> on IronPython and PyPy already and on Jython soon. I'd like to see more
>>> standard library modules use it. Distributions that choose not to
>>> include it are crippling their Python distribution.
>>
>>Interesting, I didn't know that IronPython supports ctypes, too. I still
>>find ctypes a bit problematic because it doesn't us header files for its
>>types, structs and function definitions.
> 
> This is indeed a big problem with ctypes.  Fortunately, a project exists
> to correct it:
> 
>     http://pypi.python.org/pypi/ctypes_configure/0.1
> 
> Anyone writing code with ctypes should be looking at ctypes_configure as
> well.

There is also another project that uses gccxml to parse the header files
and generate ctypes-compatible code.

http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/wiki/CodeGenerator

Especially well on Windows works the dynamic, on-demand code generation.

http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/wiki/CodeGenerator/DynamicModule

I do all win32 api programming with this stuff.

I should note that the ctypeslib project is not well maintained because
I don't have the time for that any longer.

-- 
Thanks,
Thomas


From kalman.gergely at duodecad.hu  Thu Jul 23 17:34:47 2009
From: kalman.gergely at duodecad.hu (=?UTF-8?B?S8OhbG3DoW4gR2VyZ2VseQ==?=)
Date: Thu, 23 Jul 2009 17:34:47 +0200
Subject: [Python-Dev] python sendmsg()/recvmsg() implementation
In-Reply-To: <4A365B93.7020501@gmail.com>
References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com>	<4A3634EB.2020800@duodecad.hu>
	<4A365B93.7020501@gmail.com>
Message-ID: <4A688317.9060600@duodecad.hu>

Hello

This is the rewritten-from-scratch implementation of the 
sendmsg()/recvmsg() methods.
Any comments / suggestions / flames are very welcome. Currently it 
supports what I need
and I'm only releasing it, because I don't have much time to develop it 
further in the
forseeable future (1-2 months). It is rewritten from scratch, using the 
python c-api
documents. I've tried my best, but I wouldn't bet that it works as it's 
supposed to. I'd be
glad if someone could give me a review on what I've done wrong.

The core parts are implemented correctly (I think), the features that 
are missing:
- using scatter/gather
- using it with non-stream oriented sockets (doesn't support addresses 
/msg_name/)

These should be very easy to implement though. I will fix the errors 
that are present right
now, and if no one takes up the task I will implement the missing 
features also. You might
have to wait for it a little though.

Thanks in advance

Cheers,
Kalman Gergely
-------------- next part --------------
A non-text attachment was scrubbed...
Name: srmsg.patch
Type: text/x-diff
Size: 8341 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/1a0e7a2e/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sendrecvmsgtest.py
Type: text/x-python
Size: 769 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/1a0e7a2e/attachment.py>

From dalcinl at gmail.com  Thu Jul 23 18:13:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 13:13:59 -0300
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and 	WriteFile
In-Reply-To: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
References: <4A68565C.3020500@cheimes.de>
	<20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
Message-ID: <e7ba66e40907230913g44125a18j2944bdf27680f0ec@mail.gmail.com>

On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun at divmod.com> wrote:

>
> True, CPython has C infrastructure. ?What about the other Python runtimes,
> though?
>

Perhaps these other Python runtimes could implement the functionality
of PC/_subprocess.c and use ctypes for that ?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From aahz at pythoncraft.com  Thu Jul 23 19:37:39 2009
From: aahz at pythoncraft.com (Aahz)
Date: Thu, 23 Jul 2009 10:37:39 -0700
Subject: [Python-Dev] python sendmsg()/recvmsg() implementation
In-Reply-To: <4A688317.9060600@duodecad.hu>
References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com>
	<4A3634EB.2020800@duodecad.hu> <4A365B93.7020501@gmail.com>
	<4A688317.9060600@duodecad.hu>
Message-ID: <20090723173739.GB565@panix.com>

On Thu, Jul 23, 2009, K?lm?n Gergely wrote:
>
> This is the rewritten-from-scratch implementation of the
> sendmsg()/recvmsg() methods.  Any comments / suggestions / flames are
> very welcome. Currently it supports what I need and I'm only releasing
> it, because I don't have much time to develop it further in the
> forseeable future (1-2 months). It is rewritten from scratch, using
> the python c-api documents. I've tried my best, but I wouldn't bet
> that it works as it's supposed to. I'd be glad if someone could give
> me a review on what I've done wrong.

Please post this to bugs.python.org
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"The volume of a pizza of thickness 'a' and radius 'z' is
given by pi*z*z*a"

From g.brandl at gmx.net  Thu Jul 23 20:59:42 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 23 Jul 2009 20:59:42 +0200
Subject: [Python-Dev] Document None values in sys.modules?
Message-ID: <h4ac0g$519$1@ger.gmane.org>

Hi all,

is the presence of None values in sys.modules considered an implementation
detail?  If not, it should be documented what the None values mean to the
interpreter.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From fdrake at acm.org  Thu Jul 23 21:09:10 2009
From: fdrake at acm.org (Fred Drake)
Date: Thu, 23 Jul 2009 15:09:10 -0400
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <h4ac0g$519$1@ger.gmane.org>
References: <h4ac0g$519$1@ger.gmane.org>
Message-ID: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org>

On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote:
> is the presence of None values in sys.modules considered an  
> implementation
> detail?  If not, it should be documented what the None values mean  
> to the
> interpreter.


As I recall, they're an optimization.  But since sys.modules is itself  
documented, and many applications actually use it, I think it's worth  
explaining that the None values can reasonably be expected, and what  
they indicate.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From guido at python.org  Thu Jul 23 22:05:51 2009
From: guido at python.org (Guido van Rossum)
Date: Thu, 23 Jul 2009 13:05:51 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org>
Message-ID: <ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com>

On Thu, Jul 23, 2009 at 12:09 PM, Fred Drake<fdrake at acm.org> wrote:
> On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote:
>>
>> is the presence of None values in sys.modules considered an implementation
>> detail? ?If not, it should be documented what the None values mean to the
>> interpreter.
>
> As I recall, they're an optimization. ?But since sys.modules is itself
> documented, and many applications actually use it, I think it's worth
> explaining that the None values can reasonably be expected, and what they
> indicate.

They should certainly be documented -- without them imports from
inside package would be super expensive (at least for Python versions
where implicit relative import exists). I'm somewhat surprised this
isn't documented, I don't think I've tried to keep this usage hidden.
I've also sometimes abused this to force some module to believe that a
certain other module doesn't exist.

OTOH in Py3k I'm not sure that we even *need* them any more, since
there is no more implicit relative import... They would only speed up
the raising of ImportError, not the finding of a similar-named module
elsewhere in the hierarchy.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tjreedy at udel.edu  Thu Jul 23 22:42:37 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 23 Jul 2009 16:42:37 -0400
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and  WriteFile
In-Reply-To: <e7ba66e40907230913g44125a18j2944bdf27680f0ec@mail.gmail.com>
References: <4A68565C.3020500@cheimes.de>	<20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
	<e7ba66e40907230913g44125a18j2944bdf27680f0ec@mail.gmail.com>
Message-ID: <h4ahvs$q46$1@ger.gmane.org>

Lisandro Dalcin wrote:
> On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun at divmod.com> wrote:
> 
>> True, CPython has C infrastructure.  What about the other Python runtimes,
>> though?
>>
> 
> Perhaps these other Python runtimes could implement the functionality
> of PC/_subprocess.c and use ctypes for that ?

Is it sensibly possible to augment a C file with #ifdefs so that it can 
be compiled either as a directly importable module or as a 
ctypes-accessible shared library?

tjr


From tismer at stackless.com  Thu Jul 23 23:14:35 2009
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 23 Jul 2009 14:14:35 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090723T090139-199@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
Message-ID: <4A68D2BB.8060607@stackless.com>

On 7/23/09 2:04 AM, Antoine Pitrou wrote:
> Christian Tismer<tismer<at>  stackless.com>  writes:
>> Despite the fact that Python probably has to be changed:
>> If it is true then all the 32-bit Linux Pythons have a 12
>> byte GC head, IOW they are *all* badly aligned.
>
> Why are they badly aligned?
> The fact that long double is 12 bytes long doesn't mean it will force a 12-byte
> alignment - just whatever alignment is enough for a long double on the target
> machine. This could be 4, 8 or 16 bytes.

Things are a bit different:

Alignment is not the primary concern of the gc header structure.
Note that all the objects are created by malloc (system or python's
arena allocator), and therefore all objects are correctly aligned
by construction.

The point is: The GC header is a structure invisible to the "real"
gc allocated objects. It is opaquely prepended to every gc aware
object. Therefore, it *needs* to have the correct size, in order
to propagate its (already correct) alignment to the real object.

It appears that python, compiled with gcc for all x64 32bit Linuxen
(and mingw32) produces a 12 byte GC header. Not relevant for the
header itself, but all GC objects are misaligned.

This may not be recognized so far, because there is no builtin
GC-enabled type that embeds a double.
But if you create such a type, then the double will be correctly
aligned in your object's structure, but then, when the object
gets allocated, it is misaligned, because the header size is not a
multiple of 8.

To Martin: So I disagree. The gc header is not responsible for
alignment in the first place, but to propagate it, correctly.
And this fails miserably (in principle) since years.

Proposal: We should use a simple construct that makes the
gc header size simply a multiple of 8 or 16, whatever needed.
Even a byte array would be ok.

But please no long double :-)

cheers - chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From solipsis at pitrou.net  Thu Jul 23 23:27:15 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Jul 2009 21:27:15 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com>
Message-ID: <loom.20090723T212150-107@post.gmane.org>

Christian Tismer <tismer <at> stackless.com> writes:
> 
> The point is: The GC header is a structure invisible to the "real"
> gc allocated objects. It is opaquely prepended to every gc aware
> object. Therefore, it *needs* to have the correct size, in order
> to propagate its (already correct) alignment to the real object.

Indeed.

> This may not be recognized so far, because there is no builtin
> GC-enabled type that embeds a double.
> But if you create such a type, then the double will be correctly
> aligned in your object's structure, but then, when the object
> gets allocated, it is misaligned, because the header size is not a
> multiple of 8.

I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU.
Alignment is primarily important to avoid access violations on CPUs and
datatypes which don't support arbitrary alignment, although it can also be
useful for performance reasons.

In any case, you seem to be right on this particular point: the PyGC_Head union
should probably contain a "double" alternative in addition to the "long double"
(and perhaps even a "long long" one).
Of course, it will also make memory consumption a tad bigger for GC-enabled
objects (but GC-enabled objects are generally not that small anyway).

(I disagree, however, that we should remove the "long double". After all, we
also want alignment of PyObjects to allow inclusion of a "long double" in them)

Feel free to propose a patch on bugs.python.org, by the way.

Regards

Antoine.



From solipsis at pitrou.net  Thu Jul 23 23:38:47 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Jul 2009 21:38:47 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com>
	<loom.20090723T212150-107@post.gmane.org>
Message-ID: <loom.20090723T213301-657@post.gmane.org>

Antoine Pitrou <solipsis <at> pitrou.net> writes:
> 
> In any case, you seem to be right on this particular point: the PyGC_Head union
> should probably contain a "double" alternative in addition to the "long double"
> (and perhaps even a "long long" one).

Sorry, I realize that this doesn't really address the point.
In addition to that union, we should also have a particular mechanism to compute
what the proper offset should be between the PyGC_Head and the PyObject.
Probably something like:

typedef struct {
    PyGC_Head head;
    union {
      /* ... similar union as in PyGC_head */
    } body;
} _PyGC_dummy;

#define _PyGC_Head_OFFSET offsetof(_PyGC_dummy, body)
#define _Py_AS_GC(o) ((PyGC_Head *) ((void *)(o) - _PyGC_Head_OFFSET))

Regards

Antoine.




From tismer at stackless.com  Fri Jul 24 00:14:56 2009
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 23 Jul 2009 15:14:56 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090723T212150-107@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>
	<loom.20090723T212150-107@post.gmane.org>
Message-ID: <4A68E0E0.7010508@stackless.com>

On 7/23/09 2:27 PM, Antoine Pitrou wrote:
> Christian Tismer<tismer<at>  stackless.com>  writes:
...

> I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU.

I'm also not sure. Anyway, the result was neither intended nor
expected, I guess.

> Alignment is primarily important to avoid access violations on CPUs and
> datatypes which don't support arbitrary alignment, although it can also be
> useful for performance reasons.

Performance, performance, of course (that's my job, after all :-) )
...

> Of course, it will also make memory consumption a tad bigger for GC-enabled
> objects (but GC-enabled objects are generally not that small anyway).

For that reason, I don't like the addition of the opaque header
too much. If there were an option to make the header explicit,
we would not have to round up the object size to a multiple
of (8, 16), but could arrange embedded doubles as they fit the best.

> (I disagree, however, that we should remove the "long double". After all, we
> also want alignment of PyObjects to allow inclusion of a "long double" in them)

Well, I doubt that a 12 byte long double causes any other
alignment but 4.
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From solipsis at pitrou.net  Fri Jul 24 00:55:13 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Jul 2009 22:55:13 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>
	<loom.20090723T212150-107@post.gmane.org>
	<4A68E0E0.7010508@stackless.com>
Message-ID: <loom.20090723T225426-200@post.gmane.org>

Christian Tismer <tismer <at> stackless.com> writes:
> 
> Well, I doubt that a 12 byte long double causes any other
> alignment but 4.

In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).

Regards

Antoine.



From david.lyon at preisshare.net  Fri Jul 24 01:01:31 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Thu, 23 Jul 2009 19:01:31 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <873a8n7qd9.fsf@benfinney.id.au>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<873a8n7qd9.fsf@benfinney.id.au>
Message-ID: <ed9825cf74c0750bfa9c13ad6a1782f3@preisshare.net>

On Thu, 23 Jul 2009 18:30:58 +1000, Ben Finney <ben+python at benfinney.id.au>
wrote:
> In which case you should work to get it accepted into standard Python
> *before* asking for it to be promoted in the standard Python
> documentation.

I'm very interested in how I would go about doing that. 

Die-hard users probably know all the python issues relating to
package installation and appear to have thick skins. An "easy-way"
might not mean to much to them because they've settled down into
their own comfort zone.

But new users, who's first operating system comes complete with a
polished GUI, have an expectation for having a polished GUI tool
to download their python packages in.

Under Windows, we get IDLE. And ActiveState as I understand are
working on their own package manager for their own python distribution.

So maybe it is appropriate to consider having a GUI Package Manager
within at least the Windows distribution of Python.

Yes, please tell me the process...

I'm very interested.

David



From thomas at python.org  Fri Jul 24 01:28:05 2009
From: thomas at python.org (Thomas Wouters)
Date: Thu, 23 Jul 2009 16:28:05 -0700
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com>
	<44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com>
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com>
	<446CB69E.4090609@v.loewis.de>
	<9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com>
	<3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com>
	<9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com>
Message-ID: <9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com>

So attached (and at http://codereview.appspot.com/96125/show ) is a
preliminary fix, correcting the problem with os.fork(), os.forkpty() and
os.fork1(). This doesn't expose a general API for C code to use, for two
reasons: it's not easy, and I need this fix more than I need the API change
:-) (I actually need this fix myself for Python 2.4, but it applies fairly
cleanly.)

To fix the mutex-across-fork problem correctly we should really acquire
three locks (the import lock, the GIL and the TLS lock, in that order.) The
import lock is re-entrant, and the TLS lock is tightly confined to the
thread-local-storage lookup functions, but the GIL is neither re-entrant nor
inspectable. That means we can't use pthread_atfork (we can't tell whether
we have the GIL already or not, right before the fork), nor can we provide a
convenient API for extension modules to use, really. The best we can do is
provide three functions, pthread_atfork-style: a 'prepare' function, an
'after-fork-in-child' function, and an 'after-fork-in-parent' function. The
'prepare' function would start by releasing the GIL, then acquire the import
lock, the GIL and the TLS lock in that order. It would also record
*somewhere* the thread_ident of the current thread. The 'in-parent' function
would simply release the TLS lock and the import lock, and the 'in-child'
would release those locks, call the threading._at_fork() function, and fix
up the TLS data, using the recorded thread ident to do lookups. The
'in-child' function would replace the current PyOS_AfterFork() function
(which blindly reinitializes the GIL and the TLS lock, and calls
threading._at_fork().)

Alternatively we could do what we do now, which is to ignore the fact that
thread idents may change by fork() and thus that thread-local data may
disappear, in which case the 'in-child' function could do a little less
work. I'm suitably scared and disgusted of the TLS implementation, the
threading implementations we support (including the pthread one) and the way
we blindly type-pun a pthread_t to a long, that I'm not sure I want to try
and build something correct and reliable on top of it.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/d90c9d95/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_lock_fork_deadlock.diff
Type: text/x-diff
Size: 5418 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/d90c9d95/attachment.diff>

From jafo at tummy.com  Fri Jul 24 02:57:34 2009
From: jafo at tummy.com (Sean Reifschneider)
Date: Thu, 23 Jul 2009 18:57:34 -0600
Subject: [Python-Dev] REVIEW: PyArg_ParseTuple with "s" format and NUL:
	Bogus TypeError	detail string.
Message-ID: <20090724005734.GA20019@tummy.com>

Please review this, I'm worried that there are cases where convertitem() is
returning a string that really should be overridden by the argument "help
string".  However, I'm worried that this change will get rid of useful
messages (via the format "; help string"), when there otherwise wouldn't
be.

PyArg_ParseTuple when handling a string format "s", raises a TypeError
when the passed string contains a NUL.  However, the TypeError doesn't
contain useful information.

For example:

   syslog.syslog('hello\0there')
   TypeError: [priority,] message string

This seems to be a thinko in Python/getargs.c at line 331:

   msg = convertitem(PyTuple_GET_ITEM(args, i), &format, p_va,
        flags, levels, msgbuf,
        sizeof(msgbuf), &freelist);
   if (msg) {
      seterror(i+1, msg, levels, fname, message);   <<< Line 331
      return cleanreturn(0, freelist);
   }

This also applies to Python 3 trunk in line 390.

I think that's supposed to be "msg" instead of "message" in the last
argument.  If I change it, I get:

   >>> import syslog; syslog.syslog('hello\0there')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   TypeError: must be string without null bytes, not str

I think it's safe to change "message" to "msg" because:

   "message" is the "help" string "[priority,] message string",
   but "msg" contains much more useful information.

   "msg" is in the "fall back if the last argument doesn't contain useful
   information" argument position, but "msg" is never NULL.

   "message" only is NULL if the format string doesn't contain ";".

   In every case I can find in the code, convertitem() is returning a
   much more useful string than the help string.

Or perhaps it should do something like:

   if (msg) {
      seterror(i+1, msg, levels, fname, '%s (%s)' % ( msg, message ));

Pardon my mixed C+Python, but you get the idea.

Thoughts?

Thanks,
Sean
-- 
 [...] Premature optimization is the root of all evil.
                 -- Donald Knuth
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From lists at cheimes.de  Fri Jul 24 03:23:57 2009
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 24 Jul 2009 03:23:57 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
Message-ID: <h4b2fd$6uc$1@ger.gmane.org>

david.lyon at preisshare.net wrote:
>> That's a document describing how to use ?distutils?, which is what
>> every
>> recipient of Python will already have installed.
>>
>>> Can I ask that you also provide a link for windows users
>>> to my project:
>>>
>>>   http://sourceforge.net/projects/pythonpkgmgr/
>> That doesn't seem at all appropriate; promoting third-party packages
>> isn't at all what the above document should be doing.
> 
> Distutils was once seperate and was then included in the standard python.
> 
> So i guess that I am working with the same goal in mind.

I'm sorry to inform you that a wxWindows based solution has zero change
to get into the Python standard library ever. We are not going to add
another GUI toolkit to the core distribution. We might even remove TK
from the core and ship it as a separate package like bsddb3.

Christian


From david.lyon at preisshare.net  Fri Jul 24 03:55:18 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Thu, 23 Jul 2009 21:55:18 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <h4b2fd$6uc$1@ger.gmane.org>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
Message-ID: <5ecc64238b6e429c72441815bf818f48@preisshare.net>

On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes <lists at cheimes.de>
wrote:
> I'm sorry to inform you that a wxWindows based solution has zero change
> to get into the Python standard library ever. We are not going to add
> another GUI toolkit to the core distribution. 

In executable form, the Package Manager does not require wxWidgets
to be installed.

There is no dependency for this to be installed.

I'm not requesting that wxPython be added to the standard python
distribution.

> We might even remove TK
> from the core and ship it as a separate package like bsddb3.

That doesn't affect the Python Package Manager in any way.

David


From brett at python.org  Fri Jul 24 04:31:24 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Jul 2009 19:31:24 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com>
Message-ID: <bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com>

On Thu, Jul 23, 2009 at 13:05, Guido van Rossum <guido at python.org> wrote:

> On Thu, Jul 23, 2009 at 12:09 PM, Fred Drake<fdrake at acm.org> wrote:
> > On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote:
> >>
> >> is the presence of None values in sys.modules considered an
> implementation
> >> detail?  If not, it should be documented what the None values mean to
> the
> >> interpreter.
> >
> > As I recall, they're an optimization.  But since sys.modules is itself
> > documented, and many applications actually use it, I think it's worth
> > explaining that the None values can reasonably be expected, and what they
> > indicate.
>
> They should certainly be documented -- without them imports from
> inside package would be super expensive (at least for Python versions
> where implicit relative import exists). I'm somewhat surprised this
> isn't documented, I don't think I've tried to keep this usage hidden.
> I've also sometimes abused this to force some module to believe that a
> certain other module doesn't exist.
>

It has not been hidden, but the only place I ever came across it was in the
original essay about packages.


>
> OTOH in Py3k I'm not sure that we even *need* them any more, since
> there is no more implicit relative import... They would only speed up
> the raising of ImportError, not the finding of a similar-named module
> elsewhere in the hierarchy.


None in Python 3.1 is really useless in terms of its semantics in relative
imports; importlib doesn't support it and still passes as __import__ (at
least last time I ran the test suite that way). I thought we had agreed a
while back that supporting None was not warranted in Python 3.0? Otherwise I
will do whatever work is necessary for this to happen.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/961093cc/attachment.htm>

From benjamin at python.org  Fri Jul 24 04:38:07 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 23 Jul 2009 21:38:07 -0500
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org>
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com>
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com>
Message-ID: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com>

2009/7/23 Brett Cannon <brett at python.org>:
> None in Python 3.1 is really useless in terms of its semantics in relative
> imports; importlib doesn't support it and still passes as __import__ (at
> least last time I ran the test suite that way). I thought we had agreed a
> while back that supporting None was not warranted in Python 3.0? Otherwise I
> will do whatever work is necessary for this to happen.

I think it's still nice for the rare cases where you need to trick a
module into thinking another one doesn't exist.

-- 
Regards,
Benjamin

From brett at python.org  Fri Jul 24 04:42:21 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Jul 2009 19:42:21 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com>
Message-ID: <bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com>

On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson <benjamin at python.org>wrote:

> 2009/7/23 Brett Cannon <brett at python.org>:
> > None in Python 3.1 is really useless in terms of its semantics in
> relative
> > imports; importlib doesn't support it and still passes as __import__ (at
> > least last time I ran the test suite that way). I thought we had agreed a
> > while back that supporting None was not warranted in Python 3.0?
> Otherwise I
> > will do whatever work is necessary for this to happen.
>
> I think it's still nice for the rare cases where you need to trick a
> module into thinking another one doesn't exist.


But None does not strictly mean "I don't exist". None is supposed to trigger
an another import attempt for the module with a top-level name. It's that
extra import trigger that has no real use in 3.0 and just complicates import
semantics (IMO) needlessly. If you want a module to not exist then you
either stick something else in (e.g. '42') or we remove the special
semantics for None (which I thought we had).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/2941b0a2/attachment.htm>

From benjamin at python.org  Fri Jul 24 04:48:05 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 23 Jul 2009 21:48:05 -0500
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org>
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com>
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com>
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com>
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com>
Message-ID: <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com>

2009/7/23 Brett Cannon <brett at python.org>:
>
>
> On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> 2009/7/23 Brett Cannon <brett at python.org>:
>> > None in Python 3.1 is really useless in terms of its semantics in
>> > relative
>> > imports; importlib doesn't support it and still passes as __import__ (at
>> > least last time I ran the test suite that way). I thought we had agreed
>> > a
>> > while back that supporting None was not warranted in Python 3.0?
>> > Otherwise I
>> > will do whatever work is necessary for this to happen.
>>
>> I think it's still nice for the rare cases where you need to trick a
>> module into thinking another one doesn't exist.
>
> But None does not strictly mean "I don't exist". None is supposed to trigger
> an another import attempt for the module with a top-level name. It's that
> extra import trigger that has no real use in 3.0 and just complicates import
> semantics (IMO) needlessly. If you want a module to not exist then you
> either stick something else in (e.g. '42') or we remove the special
> semantics for None (which I thought we had).


I didn't realize None had other semantics attached to it. (Imagine
that dealing with import!) +1 for making it simply indicate an
ImportError.



-- 
Regards,
Benjamin

From brett at python.org  Fri Jul 24 04:50:32 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Jul 2009 19:50:32 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com>
Message-ID: <bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com>

On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org>wrote:

> 2009/7/23 Brett Cannon <brett at python.org>:
> >
> >
> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson <benjamin at python.org>
> > wrote:
> >>
> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> > None in Python 3.1 is really useless in terms of its semantics in
> >> > relative
> >> > imports; importlib doesn't support it and still passes as __import__
> (at
> >> > least last time I ran the test suite that way). I thought we had
> agreed
> >> > a
> >> > while back that supporting None was not warranted in Python 3.0?
> >> > Otherwise I
> >> > will do whatever work is necessary for this to happen.
> >>
> >> I think it's still nice for the rare cases where you need to trick a
> >> module into thinking another one doesn't exist.
> >
> > But None does not strictly mean "I don't exist". None is supposed to
> trigger
> > an another import attempt for the module with a top-level name. It's that
> > extra import trigger that has no real use in 3.0 and just complicates
> import
> > semantics (IMO) needlessly. If you want a module to not exist then you
> > either stick something else in (e.g. '42') or we remove the special
> > semantics for None (which I thought we had).
>
>
> I didn't realize None had other semantics attached to it. (Imagine
> that dealing with import!) +1 for making it simply indicate an
> ImportError.


I'm +0 with having import raise ImportError if None is set in sys.modules as
long as we don't suddenly expect loaders to trigger the same thing if they
return None (actually, as of right now what loaders return count for
nothing, but just want to be clear).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/c2d5b1b9/attachment.htm>

From dalcinl at gmail.com  Fri Jul 24 05:18:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Jul 2009 00:18:17 -0300
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and 	WriteFile
In-Reply-To: <h4ahvs$q46$1@ger.gmane.org>
References: <4A68565C.3020500@cheimes.de>
	<20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>
	<e7ba66e40907230913g44125a18j2944bdf27680f0ec@mail.gmail.com>
	<h4ahvs$q46$1@ger.gmane.org>
Message-ID: <e7ba66e40907232018r643fa4b8paa5e4ddd71af47ea@mail.gmail.com>

On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy<tjreedy at udel.edu> wrote:
> Lisandro Dalcin wrote:
>>
>> On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun at divmod.com>
>> wrote:
>>
>>> True, CPython has C infrastructure. ?What about the other Python
>>> runtimes,
>>> though?
>>>
>>
>> Perhaps these other Python runtimes could implement the functionality
>> of PC/_subprocess.c and use ctypes for that ?
>
> Is it sensibly possible to augment a C file with #ifdefs so that it can be
> compiled either as a directly importable module or as a ctypes-accessible
> shared library?
>

Of course it is posible... Moreover, you could likely reuse 100 % of
code intended for ctypes in implementing the extension module calls
intended for Python. Mac OS X could have some issues though (IIUC, .so
ext modules are not the same as .dylib dynamic libs, though perhaps
ctypes can still dlopen() .so files?)

However, how that would help these other Python runtimes without C
infraestructure ??



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From guido at python.org  Fri Jul 24 05:18:35 2009
From: guido at python.org (Guido van Rossum)
Date: Thu, 23 Jul 2009 20:18:35 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com>
Message-ID: <ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com>

So, I guess, we'll live with it for a while longer. Given that it
managed to evade our attention for so long, I think that's fine.

I agree that there's no reason for a None result from loaders to be
interpreted the same way, assuming that's not how it works ATM.

And we can live with import and importlib differing on this in 3.1
(though you could call it a bug in importlib and fix it for 3.1.1 --
not sure if you were planning on that).

--Guido

On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon<brett at python.org> wrote:
>
>
> On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> 2009/7/23 Brett Cannon <brett at python.org>:
>> >
>> >
>> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson <benjamin at python.org>
>> > wrote:
>> >>
>> >> 2009/7/23 Brett Cannon <brett at python.org>:
>> >> > None in Python 3.1 is really useless in terms of its semantics in
>> >> > relative
>> >> > imports; importlib doesn't support it and still passes as __import__
>> >> > (at
>> >> > least last time I ran the test suite that way). I thought we had
>> >> > agreed
>> >> > a
>> >> > while back that supporting None was not warranted in Python 3.0?
>> >> > Otherwise I
>> >> > will do whatever work is necessary for this to happen.
>> >>
>> >> I think it's still nice for the rare cases where you need to trick a
>> >> module into thinking another one doesn't exist.
>> >
>> > But None does not strictly mean "I don't exist". None is supposed to
>> > trigger
>> > an another import attempt for the module with a top-level name. It's
>> > that
>> > extra import trigger that has no real use in 3.0 and just complicates
>> > import
>> > semantics (IMO) needlessly. If you want a module to not exist then you
>> > either stick something else in (e.g. '42') or we remove the special
>> > semantics for None (which I thought we had).
>>
>>
>> I didn't realize None had other semantics attached to it. (Imagine
>> that dealing with import!) +1 for making it simply indicate an
>> ImportError.
>
> I'm +0 with having import raise ImportError if None is set in sys.modules as
> long as we don't suddenly expect loaders to trigger the same thing if they
> return None (actually, as of right now what loaders return count for
> nothing, but just want to be clear).
> -Brett



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul 24 05:35:46 2009
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Jul 2009 20:35:46 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com>
Message-ID: <bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com>

On Thu, Jul 23, 2009 at 20:18, Guido van Rossum <guido at python.org> wrote:

> So, I guess, we'll live with it for a while longer. Given that it
> managed to evade our attention for so long, I think that's fine.
>

Can someone double-check me that the semantics can even be triggered in 3.1?
I just tried and couldn't come up with anything. Heck, I quick search for a
Py_None comparison in 3.1's import.c turned up nothing useful (other than
mark_miss() is the function used to set None in sys.modules). We might have
actually already removed it or made it so that the semantics can't be
triggered.


>
> I agree that there's no reason for a None result from loaders to be
> interpreted the same way, assuming that's not how it works ATM.
>
> And we can live with import and importlib differing on this in 3.1
> (though you could call it a bug in importlib and fix it for 3.1.1 --
> not sure if you were planning on that).
>

I can if people can trigger the semantics somehow so I have a test to go by.

-Brett


>
> --Guido
>
> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon<brett at python.org> wrote:
> >
> >
> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org>
> > wrote:
> >>
> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >
> >> >
> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson <benjamin at python.org
> >
> >> > wrote:
> >> >>
> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >> > None in Python 3.1 is really useless in terms of its semantics in
> >> >> > relative
> >> >> > imports; importlib doesn't support it and still passes as
> __import__
> >> >> > (at
> >> >> > least last time I ran the test suite that way). I thought we had
> >> >> > agreed
> >> >> > a
> >> >> > while back that supporting None was not warranted in Python 3.0?
> >> >> > Otherwise I
> >> >> > will do whatever work is necessary for this to happen.
> >> >>
> >> >> I think it's still nice for the rare cases where you need to trick a
> >> >> module into thinking another one doesn't exist.
> >> >
> >> > But None does not strictly mean "I don't exist". None is supposed to
> >> > trigger
> >> > an another import attempt for the module with a top-level name. It's
> >> > that
> >> > extra import trigger that has no real use in 3.0 and just complicates
> >> > import
> >> > semantics (IMO) needlessly. If you want a module to not exist then you
> >> > either stick something else in (e.g. '42') or we remove the special
> >> > semantics for None (which I thought we had).
> >>
> >>
> >> I didn't realize None had other semantics attached to it. (Imagine
> >> that dealing with import!) +1 for making it simply indicate an
> >> ImportError.
> >
> > I'm +0 with having import raise ImportError if None is set in sys.modules
> as
> > long as we don't suddenly expect loaders to trigger the same thing if
> they
> > return None (actually, as of right now what loaders return count for
> > nothing, but just want to be clear).
> > -Brett
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090723/ee9c4acd/attachment-0001.htm>

From horos11 at gmail.com  Fri Jul 24 07:31:31 2009
From: horos11 at gmail.com (Edward Peschko)
Date: Thu, 23 Jul 2009 22:31:31 -0700
Subject: [Python-Dev] command line attachable debugger
Message-ID: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>

all,

I'I was wondering if there was a command line python debugger that was
able to attach to an existing process. I'd very much like to be able
to debug over a ssh session using screen.

Ed

(ps - and yes, I know about winpdb, etc... that is not exactly what
I'm looking for..)

From tjreedy at udel.edu  Fri Jul 24 07:49:37 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 24 Jul 2009 01:49:37 -0400
Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile,
	and  WriteFile
In-Reply-To: <e7ba66e40907232018r643fa4b8paa5e4ddd71af47ea@mail.gmail.com>
References: <4A68565C.3020500@cheimes.de>	<20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com>	<e7ba66e40907230913g44125a18j2944bdf27680f0ec@mail.gmail.com>	<h4ahvs$q46$1@ger.gmane.org>
	<e7ba66e40907232018r643fa4b8paa5e4ddd71af47ea@mail.gmail.com>
Message-ID: <h4bi1g$2oo$1@ger.gmane.org>

Lisandro Dalcin wrote:
> On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy<tjreedy at udel.edu> wrote:
>> Lisandro Dalcin wrote:
>>> On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun at divmod.com>
>>> wrote:
>>>
>>>> True, CPython has C infrastructure.  What about the other Python
>>>> runtimes,
>>>> though?
>>>>
>>> Perhaps these other Python runtimes could implement the functionality
>>> of PC/_subprocess.c and use ctypes for that ?
>> Is it sensibly possible to augment a C file with #ifdefs so that it can be
>> compiled either as a directly importable module or as a ctypes-accessible
>> shared library?
>>
> 
> Of course it is posible... Moreover, you could likely reuse 100 % of
> code intended for ctypes in implementing the extension module calls
> intended for Python. Mac OS X could have some issues though (IIUC, .so
> ext modules are not the same as .dylib dynamic libs, though perhaps
> ctypes can still dlopen() .so files?)
> 
> However, how that would help these other Python runtimes without C
> infraestructure ??

I believe someone just posted that PyPy and IronPython have ctypes and 
Jython is working on it. That is what triggered this post.


From kalman.gergely at duodecad.hu  Fri Jul 24 10:32:36 2009
From: kalman.gergely at duodecad.hu (=?ISO-8859-1?Q?K=E1lm=E1n_Gergely?=)
Date: Fri, 24 Jul 2009 10:32:36 +0200
Subject: [Python-Dev] python sendmsg()/recvmsg() implementation
In-Reply-To: <20090723173739.GB565@panix.com>
References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com>	<4A3634EB.2020800@duodecad.hu>
	<4A365B93.7020501@gmail.com>	<4A688317.9060600@duodecad.hu>
	<20090723173739.GB565@panix.com>
Message-ID: <4A6971A4.6030705@duodecad.hu>

Done, it's at:

http://bugs.python.org/issue6560

Kalman Gergely


Aahz wrote:
> On Thu, Jul 23, 2009, K?lm?n Gergely wrote:
>   
>> This is the rewritten-from-scratch implementation of the
>> sendmsg()/recvmsg() methods.  Any comments / suggestions / flames are
>> very welcome. Currently it supports what I need and I'm only releasing
>> it, because I don't have much time to develop it further in the
>> forseeable future (1-2 months). It is rewritten from scratch, using
>> the python c-api documents. I've tried my best, but I wouldn't bet
>> that it works as it's supposed to. I'd be glad if someone could give
>> me a review on what I've done wrong.
>>     
>
> Please post this to bugs.python.org
>   


From bugtrack at roumenpetrov.info  Fri Jul 24 14:16:06 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Fri, 24 Jul 2009 15:16:06 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A675C10.6060200@stackless.com>
References: <4A675C10.6060200@stackless.com>
Message-ID: <4A69A606.1040506@roumenpetrov.info>

Christian Tismer wrote:
> Hi all,
> 
> I was hacking to get mingw32 builds of psyco to work
> and found a pretty weird thing:
> 
> I used mingw32 (stable distro) to build the psyco extension
> on top of standard python2.6, built with Visual Studio,
> and got weird crashes.
> 
> The reason is in objimpl.h:
> 
> typedef union _gc_head {
>     struct {
>         union _gc_head *gc_next;
>         union _gc_head *gc_prev;
>         Py_ssize_t gc_refs;
>     } gc;
>     long double dummy;  /* force worst-case alignment */
> } PyGC_Head;
[SNIP]

Did the crash disappear is you add "__attribute__((aligned(8)))" after 
variable dummy ?

Regards,
Roumen

From bugtrack at roumenpetrov.info  Fri Jul 24 14:45:12 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Fri, 24 Jul 2009 15:45:12 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090723T225426-200@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>	<4A68E0E0.7010508@stackless.com>
	<loom.20090723T225426-200@post.gmane.org>
Message-ID: <4A69ACD8.5040408@roumenpetrov.info>

Antoine Pitrou wrote:
> Christian Tismer <tismer <at> stackless.com> writes:
>> Well, I doubt that a 12 byte long double causes any other
>> alignment but 4.
> 
> In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).

And alignment is ?



From solipsis at pitrou.net  Fri Jul 24 14:55:29 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 24 Jul 2009 12:55:29 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>	<4A68E0E0.7010508@stackless.com>
	<loom.20090723T225426-200@post.gmane.org>
	<4A69ACD8.5040408@roumenpetrov.info>
Message-ID: <loom.20090724T125513-986@post.gmane.org>

Roumen Petrov <bugtrack <at> roumenpetrov.info> writes:
> 
> Antoine Pitrou wrote:
> > 
> > In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).
> 
> And alignment is ?

16 bytes as well.




From solipsis at pitrou.net  Fri Jul 24 15:07:54 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 24 Jul 2009 13:07:54 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>	<4A68E0E0.7010508@stackless.com>
	<loom.20090723T225426-200@post.gmane.org>
	<4A69ACD8.5040408@roumenpetrov.info>
	<loom.20090724T125513-986@post.gmane.org>
Message-ID: <loom.20090724T130617-820@post.gmane.org>

Antoine Pitrou <solipsis <at> pitrou.net> writes:
> 
> Roumen Petrov <bugtrack <at> roumenpetrov.info> writes:
> > 
> > Antoine Pitrou wrote:
> > > 
> > > In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).
> > 
> > And alignment is ?
> 
> 16 bytes as well.

This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit
mode, but 32 bytes in 64-bit mode.



From bugtrack at roumenpetrov.info  Fri Jul 24 15:51:33 2009
From: bugtrack at roumenpetrov.info (Roumen Petrov)
Date: Fri, 24 Jul 2009 16:51:33 +0300
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090724T130617-820@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>	<4A68E0E0.7010508@stackless.com>	<loom.20090723T225426-200@post.gmane.org>	<4A69ACD8.5040408@roumenpetrov.info>	<loom.20090724T125513-986@post.gmane.org>
	<loom.20090724T130617-820@post.gmane.org>
Message-ID: <4A69BC65.4000006@roumenpetrov.info>

Antoine Pitrou wrote:
> Antoine Pitrou <solipsis <at> pitrou.net> writes:
>> Roumen Petrov <bugtrack <at> roumenpetrov.info> writes:
>>> Antoine Pitrou wrote:
>>>> In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!).
>>> And alignment is ?
>> 16 bytes as well.
> 
> This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit
> mode, but 32 bytes in 64-bit mode.

yes, 10x

Roumen

From aahz at pythoncraft.com  Fri Jul 24 16:10:08 2009
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 24 Jul 2009 07:10:08 -0700
Subject: [Python-Dev] REVIEW: PyArg_ParseTuple with "s" format and
	NUL:	Bogus TypeError detail string.
In-Reply-To: <20090724005734.GA20019@tummy.com>
References: <20090724005734.GA20019@tummy.com>
Message-ID: <20090724141008.GA21866@panix.com>

On Thu, Jul 23, 2009, Sean Reifschneider wrote:
>
> Thoughts?

Please file a report at bugs.python.org to make sure it doesn't get lost.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"At Resolver we've found it useful to short-circuit any doubt and just        
refer to comments in code as 'lies'. :-)"
--Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22

From foom at fuhm.net  Fri Jul 24 16:53:42 2009
From: foom at fuhm.net (James Y Knight)
Date: Fri, 24 Jul 2009 10:53:42 -0400
Subject: [Python-Dev] command line attachable debugger
In-Reply-To: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
Message-ID: <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net>


On Jul 24, 2009, at 1:31 AM, Edward Peschko wrote:

> all,
>
> I'I was wondering if there was a command line python debugger that was
> able to attach to an existing process. I'd very much like to be able
> to debug over a ssh session using screen.
>
> Ed
>
> (ps - and yes, I know about winpdb, etc... that is not exactly what
> I'm looking for..)

Winpdb is *exactly* what you asked for, so if it's not what you're  
looking for you'll need to be more specific about what you want that  
it doesn't do...

James

From aahz at pythoncraft.com  Fri Jul 24 17:13:56 2009
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 24 Jul 2009 08:13:56 -0700
Subject: [Python-Dev] command line attachable debugger
In-Reply-To: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
Message-ID: <20090724151356.GA13491@panix.com>

On Thu, Jul 23, 2009, Edward Peschko wrote:
> 
> I'I was wondering if there was a command line python debugger that was
> able to attach to an existing process. I'd very much like to be able
> to debug over a ssh session using screen.

python-dev is not the correct place to ask about this, please use
comp.lang.python (python-dev is for questions about fixing bugs and
adding features).
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"At Resolver we've found it useful to short-circuit any doubt and just        
refer to comments in code as 'lies'. :-)"
--Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22

From g.brandl at gmx.net  Fri Jul 24 17:54:09 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 24 Jul 2009 17:54:09 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <5ecc64238b6e429c72441815bf818f48@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
Message-ID: <h4clgl$cfe$1@ger.gmane.org>

David Lyon schrieb:
> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes <lists at cheimes.de>
> wrote:
>> I'm sorry to inform you that a wxWindows based solution has zero change
>> to get into the Python standard library ever. We are not going to add
>> another GUI toolkit to the core distribution. 
> 
> In executable form, the Package Manager does not require wxWidgets
> to be installed.
> 
> There is no dependency for this to be installed.

What does "in exectuable form" mean?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From status at bugs.python.org  Fri Jul 24 18:07:59 2009
From: status at bugs.python.org (Python tracker)
Date: Fri, 24 Jul 2009 18:07:59 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20090724160759.C0C57785CD@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (07/17/09 - 07/24/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2278 open (+41) / 16106 closed (+16) / 18384 total (+57)

Open issues with patches:   897

Average duration of open issues: 661 days.
Median duration of open issues: 413 days.

Open Issues Breakdown
   open  2245 (+41)
pending    33 ( +0)

Issues Created Or Reopened (58)
_______________________________

warnings.warn segfaults on bad formatted string                  07/18/09
CLOSED http://bugs.python.org/issue6415    reopened r.david.murray                
       patch                                                                   

lib2to3 fails to convert 'thread' when not followed by a period  07/17/09
CLOSED http://bugs.python.org/issue6506    created  ericp                         
                                                                               

Enhance dis.dis to autocompile codestrings                       07/17/09
       http://bugs.python.org/issue6507    created  tjreedy                       
                                                                               

expose setresuid                                                 07/17/09
       http://bugs.python.org/issue6508    created  solinym                       
                                                                               

re.py - <compiled byte-object regular expr> encounter unexpected 07/18/09
       http://bugs.python.org/issue6509    created  kaizhu                        
                                                                               

zipfile: OSError File exists                                     07/18/09
CLOSED http://bugs.python.org/issue6510    created  srid                          
                                                                               

zipfile: Invalid argument when opening zero-sized files          07/18/09
       http://bugs.python.org/issue6511    created  srid                          
                                                                               

UnicodeDecodeError: 'utf8' codec can't decode bytes in position  07/18/09
CLOSED http://bugs.python.org/issue6512    created  wplappert                     
                                                                               

Standard Library, Warnings, 28.5.4, example fails                07/18/09
CLOSED http://bugs.python.org/issue6513    created  jasontiller                   
                                                                               

"python -m unittest <testmodule>" does not run any tests         07/18/09
CLOSED http://bugs.python.org/issue6514    created  exarkun                       
                                                                               

http://docs.python.org/dev/library/unittest.html#load-tests-prot 07/18/09
       http://bugs.python.org/issue6515    created  exarkun                       
                                                                               

reset owner/group to root for distutils tarballs                 07/18/09
       http://bugs.python.org/issue6516    created  till                          
                                                                               

configparser: add possibility to escape formatstrings            07/18/09
       http://bugs.python.org/issue6517    created  till                          
       patch                                                                   

Enable 'with' statement in ossaudiodev module                    07/19/09
       http://bugs.python.org/issue6518    created  jjalocha                      
                                                                               

Reorder 'with' statement for files in Python Tutorial            07/19/09
       http://bugs.python.org/issue6519    created  jjalocha                      
                                                                               

urllib.urlopen does not have timeout parameter where as urllib2. 07/19/09
       http://bugs.python.org/issue6520    created  orsenthil                     
       easy                                                                    

Contradictory documentation for email.mime.text.MIMEText         07/19/09
       http://bugs.python.org/issue6521    created  pitrou                        
                                                                               

docs for unittest.expectedFailure do not syntactically show it's 07/20/09
       http://bugs.python.org/issue6522    created  brett.cannon                  
                                                                               

smtplib exception smtp.connect TypeError encode_plain            07/20/09
       http://bugs.python.org/issue6523    created  kmcguire                      
                                                                               

tarfile module missing decompression support?                    07/20/09
CLOSED http://bugs.python.org/issue6524    created  hravnx                        
                                                                               

Problem with string.lowercase in Windows XP                      07/20/09
CLOSED http://bugs.python.org/issue6525    created  PeterL                        
                                                                               

importlib.import_module affects permissions of .pyc files subseq 07/20/09
       http://bugs.python.org/issue6526    created  r.david.murray                
       patch                                                                   

test_ttk_guionly buildbot test crash: Tcl_FinalizeNotifier: noti 07/20/09
       http://bugs.python.org/issue6527    created  r.david.murray                
                                                                               

builtins colored as keyword at beginning of line                 07/20/09
       http://bugs.python.org/issue6528    created  Freiddie                      
                                                                               

get{pw,gr}{nam,[ug]id}() calls are not re-entrant                07/20/09
       http://bugs.python.org/issue6529    created  dhduvall                      
                                                                               

Regression on "python -Wi" crash                                 07/21/09
CLOSED http://bugs.python.org/issue6530    created  kbk                           
       needs review                                                            

atexit_callfuncs() crashing within Py_Finalize() when using mult 07/21/09
       http://bugs.python.org/issue6531    created  grahamd                       
                                                                               

thread.get_ident() should return unsigned value                  07/21/09
       http://bugs.python.org/issue6532    created  sargo                         
                                                                               

Make test_xmlrpc_net functional in the absence of time.xmlrpc.co 07/21/09
       http://bugs.python.org/issue6533    created  r.david.murray                
       patch, patch, easy                                                      

os.makedirs returns EACCES for "C:\"                             07/21/09
CLOSED http://bugs.python.org/issue6534    created  luke-jr                       
                                                                               

optparse required field for Options                              07/21/09
       http://bugs.python.org/issue6535    created  mindvirus                     
       patch, easy                                                             

urllib2 howto contains typo                                      07/21/09
CLOSED http://bugs.python.org/issue6536    created  kurtmckee                     
                                                                               

string.split shouldn't split on non-breaking spaces              07/21/09
CLOSED http://bugs.python.org/issue6537    reopened ericp                         
                                                                               

MatchObject is not a hyperlink in the 're' module documentation  07/21/09
       http://bugs.python.org/issue6538    created  r.david.murray                
       easy                                                                    

unittest dir not created during install                          07/21/09
CLOSED http://bugs.python.org/issue6539    created  kbk                           
       patch, needs review                                                     

bytearray.translate(): error in error handling                   07/22/09
CLOSED http://bugs.python.org/issue6540    created  haypo                         
       patch                                                                   

SpooledTemporaryFile breakages                                   07/22/09
       http://bugs.python.org/issue6541    created  leonov                        
       patch                                                                   

test_os TestInvalidFD.test_closerange causes test_pipes hang in  07/22/09
CLOSED http://bugs.python.org/issue6542    created  r.david.murray                
       patch                                                                   

traceback presented in wrong encoding                            07/22/09
       http://bugs.python.org/issue6543    created  r_mosaic                      
       patch, needs review                                                     

Fix refleak in kqueue implementation                             07/22/09
       http://bugs.python.org/issue6544    created  georg.brandl                  
       patch                                                                   

test fails in distutils.tests.test_extensions if -O is used      07/22/09
CLOSED http://bugs.python.org/issue6545    created  tarek                         
                                                                               

[Distutils][PATCH] Add bdist_rpm option to select the name of th 07/22/09
       http://bugs.python.org/issue6546    created  OG7                           
       patch                                                                   

shutil.copytree fails on dangling symlinks                       07/22/09
       http://bugs.python.org/issue6547    created  TTimo                         
                                                                               

cmath documentation misleading:  suggests existence of real() an 07/22/09
       http://bugs.python.org/issue6548    created  marketdickinson               
       patch                                                                   

ttk.Style not translating some Tcl options                       07/22/09
       http://bugs.python.org/issue6549    created  markroseman                   
                                                                               

asyncore incorrect failure when connection is refused and using  07/22/09
       http://bugs.python.org/issue6550    created  nirs                          
       patch                                                                   

test_codecs fails when ran after test_zipimport and test_mailbox 07/23/09
       http://bugs.python.org/issue6551    created  alexandre.vassalotti          
                                                                               

Build Applet.app                                                 07/23/09
       http://bugs.python.org/issue6552    created  MartienF                      
                                                                               

cPickle "binunicode" segmentation fault                          07/23/09
CLOSED http://bugs.python.org/issue6553    created  boogenhagn                    
       patch                                                                   

Do we have something like	os.pid_exists()?                       07/23/09
       http://bugs.python.org/issue6554    created  pmolina                       
                                                                               

distutils config file should have the same name on both platform 07/23/09
       http://bugs.python.org/issue6555    created  zooko                         
                                                                               

"HOME" is not a standard environment variable on Windows         07/23/09
       http://bugs.python.org/issue6556    created  zooko                         
                                                                               

urllib.urlopen creates bad requests when location header of 301  07/23/09
       http://bugs.python.org/issue6557    created  till                          
                                                                               

#ifdef linux is incorrect; should be #ifdef __linux__ (preferred 07/24/09
       http://bugs.python.org/issue6558    created  yaneurabeya                   
                                                                               

[PATCH]add pass_fds paramter to subprocess.Popen()               07/24/09
       http://bugs.python.org/issue6559    created  zhigang                       
       patch                                                                   

socket sendmsg(), recvmsg() methods                              07/24/09
       http://bugs.python.org/issue6560    created  synapse                       
       patch                                                                   

Regex '\d' should not match unicode category 'No'.               07/24/09
       http://bugs.python.org/issue6561    created  marketdickinson               
       patch                                                                   

OverflowError in RLock.acquire()                                 07/24/09
       http://bugs.python.org/issue6562    created  davidar                       
                                                                               



Issues Now Closed (37)
______________________

atexit module not safe in Python 3.0 with multiple interpreters   270 days
       http://bugs.python.org/issue4200    grahamd                       
       patch, needs review                                                     

bugs in array.array with exports (buffer protocol)                230 days
       http://bugs.python.org/issue4509    alexandre.vassalotti          
       patch, needs review                                                     

urllib2.py timeouts do not propagate across redirects for 2.6.1   170 days
       http://bugs.python.org/issue5102    orsenthil                     
       patch, easy                                                             

pydoc reports misleading failure if target module raises an Impo  156 days
       http://bugs.python.org/issue5230    r.david.murray                
       patch, easy                                                             

Python 2.6 makes .pyc/.pyo bytecode files executable               60 days
       http://bugs.python.org/issue6070    r.david.murray                
       patch                                                                   

[IDLE] UnicodeDecodeError when invoking force-open-completions     50 days
       http://bugs.python.org/issue6144    ezio.melotti                  
                                                                               

Make PyDescr_COMMON conform to standard C                          52 days
       http://bugs.python.org/issue6151    alexandre.vassalotti          
       patch, patch, needs review                                              

Make io.BytesIO and io.StringIO picklable.                         46 days
       http://bugs.python.org/issue6218    alexandre.vassalotti          
       patch, patch                                                            

Better type checking for the arguments of io.StringIO              43 days
       http://bugs.python.org/issue6241    alexandre.vassalotti          
       patch                                                                   

Fix reference leak in io.StringIO                                  43 days
       http://bugs.python.org/issue6242    alexandre.vassalotti          
       patch                                                                   

Simplify string decoding in xmlrpc.client.                         23 days
       http://bugs.python.org/issue6360    alexandre.vassalotti          
       patch                                                                   

warnings.warn segfaults on bad formatted string                     1 days
       http://bugs.python.org/issue6415    r.david.murray                
       patch                                                                   

The cgi docs should advertize using "in" instead of "has_key"      17 days
       http://bugs.python.org/issue6423    ezio.melotti                  
       patch                                                                   

Fraction fails equality test with a user-defined type              11 days
       http://bugs.python.org/issue6431    marketdickinson               
       patch                                                                   

Improve/update python.org/dev/                                      8 days
       http://bugs.python.org/issue6449    r.david.murray                
       easy                                                                    

Documentation of ElementTree.Element.getiterator implies element    3 days
       http://bugs.python.org/issue6489    georg.brandl                  
                                                                               

2to3 generates "from urllib.parse import pathname2url"              1 days
       http://bugs.python.org/issue6496    benjamin.peterson             
       patch                                                                   

documentation error: missing comma between kwonlyargcount & nloc    1 days
       http://bugs.python.org/issue6502    georg.brandl                  
                                                                               

Minor typos in tutorial (i/o chapter)                               1 days
       http://bugs.python.org/issue6505    georg.brandl                  
                                                                               

lib2to3 fails to convert 'thread' when not followed by a period     0 days
       http://bugs.python.org/issue6506    benjamin.peterson             
                                                                               

zipfile: OSError File exists                                        3 days
       http://bugs.python.org/issue6510    amaury.forgeotdarc            
                                                                               

UnicodeDecodeError: 'utf8' codec can't decode bytes in position     0 days
       http://bugs.python.org/issue6512    ezio.melotti                  
                                                                               

Standard Library, Warnings, 28.5.4, example fails                   0 days
       http://bugs.python.org/issue6513    georg.brandl                  
                                                                               

"python -m unittest <testmodule>" does not run any tests            2 days
       http://bugs.python.org/issue6514    michael.foord                 
                                                                               

tarfile module missing decompression support?                       0 days
       http://bugs.python.org/issue6524    hravnx                        
                                                                               

Problem with string.lowercase in Windows XP                         3 days
       http://bugs.python.org/issue6525    PeterL                        
                                                                               

Regression on "python -Wi" crash                                    0 days
       http://bugs.python.org/issue6530    benjamin.peterson             
       needs review                                                            

os.makedirs returns EACCES for "C:\"                                0 days
       http://bugs.python.org/issue6534    amaury.forgeotdarc            
                                                                               

urllib2 howto contains typo                                         0 days
       http://bugs.python.org/issue6536    ezio.melotti                  
                                                                               

string.split shouldn't split on non-breaking spaces                 0 days
       http://bugs.python.org/issue6537    benjamin.peterson             
                                                                               

unittest dir not created during install                             0 days
       http://bugs.python.org/issue6539    benjamin.peterson             
       patch, needs review                                                     

bytearray.translate(): error in error handling                      0 days
       http://bugs.python.org/issue6540    georg.brandl                  
       patch                                                                   

test_os TestInvalidFD.test_closerange causes test_pipes hang in     1 days
       http://bugs.python.org/issue6542    r.david.murray                
       patch                                                                   

test fails in distutils.tests.test_extensions if -O is used         0 days
       http://bugs.python.org/issue6545    tarek                         
                                                                               

cPickle "binunicode" segmentation fault                             0 days
       http://bugs.python.org/issue6553    amaury.forgeotdarc            
       patch                                                                   

Hang using files named prn.txt, etc                              2810 days
       http://bugs.python.org/issue481171  amaury.forgeotdarc            
                                                                               

copy 2                                                            725 days
       http://bugs.python.org/issue1764044 amaury.forgeotdarc            
                                                                               



Top Issues Most Discussed (10)
______________________________

 12 Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer     8 days
open    http://bugs.python.org/issue6499   

 10 Problem with string.lowercase in Windows XP                        3 days
closed  http://bugs.python.org/issue6525   

  7 Do we have something like	os.pid_exists()?                         1 days
open    http://bugs.python.org/issue6554   

  7 UnicodeDecodeError: 'utf8' codec can't decode bytes in position    0 days
closed  http://bugs.python.org/issue6512   

  6 test_os TestInvalidFD.test_closerange causes test_pipes hang in    1 days
closed  http://bugs.python.org/issue6542   

  5 importlib.import_module affects permissions of .pyc files subse    4 days
open    http://bugs.python.org/issue6526   

  5 xml.sax.saxutils.XMLGenerator should write to io.RawIOBase.      186 days
open    http://bugs.python.org/issue4997   

  4 cmath documentation misleading:  suggests existence of real() a    2 days
open    http://bugs.python.org/issue6548   

  4 optparse required field for Options                                3 days
open    http://bugs.python.org/issue6535   

  4 configparser: add possibility to escape formatstrings              6 days
open    http://bugs.python.org/issue6517   




From p.f.moore at gmail.com  Fri Jul 24 18:08:32 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 24 Jul 2009 17:08:32 +0100
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <h4clgl$cfe$1@ger.gmane.org>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
	<h4clgl$cfe$1@ger.gmane.org>
Message-ID: <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com>

2009/7/24 Georg Brandl <g.brandl at gmx.net>:
> David Lyon schrieb:
>> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes <lists at cheimes.de>
>> wrote:
>>> I'm sorry to inform you that a wxWindows based solution has zero change
>>> to get into the Python standard library ever. We are not going to add
>>> another GUI toolkit to the core distribution.
>>
>> In executable form, the Package Manager does not require wxWidgets
>> to be installed.
>>
>> There is no dependency for this to be installed.
>
> What does "in exectuable form" mean?

I read this as meaning that David was proposing to ship a built
application (on Windows, bundled up with something like py2exe, I
guess) and any supporting DLLs such as the wxWindows ones would be
bundled in, but the wxPython package would *not* be shipped as part of
the standard library.

If my assumption is right, that makes the package manager
inappropriate for shipping with Python, whether or not there are other
issues.

In my view, for an "application" to be shipped with Python, whether it
be Idle, or a package manager, or pydoc, or whatever, it must:
- be written as a Python script
- depend only on packages shipped with the standard library

That is before any considerations of the value of the application itself.

Given that David's package manager uses setuptools to do the installs
(and now it appears it uses wxPython for its GUI) that immediately
disqualifies it for inclusion in my view - regardless of its value as
a package.

Paul.

From guido at python.org  Fri Jul 24 19:39:53 2009
From: guido at python.org (Guido van Rossum)
Date: Fri, 24 Jul 2009 10:39:53 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> 
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com>
Message-ID: <ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>

Still works, at least in some old 3.1 I had lying around:

$ python3.1
Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['string'] = None
>>> import string
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named string
>>>
$ python3.1
Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>>
$

The experiment should be easily repeatable. :-)

--Guido

On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon<brett at python.org> wrote:
>
>
> On Thu, Jul 23, 2009 at 20:18, Guido van Rossum <guido at python.org> wrote:
>>
>> So, I guess, we'll live with it for a while longer. Given that it
>> managed to evade our attention for so long, I think that's fine.
>
> Can someone double-check me that the semantics can even be triggered in 3.1?
> I just tried and couldn't come up with anything. Heck, I quick search for a
> Py_None comparison in 3.1's import.c turned up nothing useful (other than
> mark_miss() is the function used to set None in sys.modules). We might have
> actually already removed it or made it so that the semantics can't be
> triggered.
>
>>
>> I agree that there's no reason for a None result from loaders to be
>> interpreted the same way, assuming that's not how it works ATM.
>>
>> And we can live with import and importlib differing on this in 3.1
>> (though you could call it a bug in importlib and fix it for 3.1.1 --
>> not sure if you were planning on that).
>
> I can if people can trigger the semantics somehow so I have a test to go by.
> -Brett
>
>>
>> --Guido
>>
>> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon<brett at python.org> wrote:
>> >
>> >
>> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org>
>> > wrote:
>> >>
>> >> 2009/7/23 Brett Cannon <brett at python.org>:
>> >> >
>> >> >
>> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson
>> >> > <benjamin at python.org>
>> >> > wrote:
>> >> >>
>> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
>> >> >> > None in Python 3.1 is really useless in terms of its semantics in
>> >> >> > relative
>> >> >> > imports; importlib doesn't support it and still passes as
>> >> >> > __import__
>> >> >> > (at
>> >> >> > least last time I ran the test suite that way). I thought we had
>> >> >> > agreed
>> >> >> > a
>> >> >> > while back that supporting None was not warranted in Python 3.0?
>> >> >> > Otherwise I
>> >> >> > will do whatever work is necessary for this to happen.
>> >> >>
>> >> >> I think it's still nice for the rare cases where you need to trick a
>> >> >> module into thinking another one doesn't exist.
>> >> >
>> >> > But None does not strictly mean "I don't exist". None is supposed to
>> >> > trigger
>> >> > an another import attempt for the module with a top-level name. It's
>> >> > that
>> >> > extra import trigger that has no real use in 3.0 and just complicates
>> >> > import
>> >> > semantics (IMO) needlessly. If you want a module to not exist then
>> >> > you
>> >> > either stick something else in (e.g. '42') or we remove the special
>> >> > semantics for None (which I thought we had).
>> >>
>> >>
>> >> I didn't realize None had other semantics attached to it. (Imagine
>> >> that dealing with import!) +1 for making it simply indicate an
>> >> ImportError.
>> >
>> > I'm +0 with having import raise ImportError if None is set in
>> > sys.modules as
>> > long as we don't suddenly expect loaders to trigger the same thing if
>> > they
>> > return None (actually, as of right now what loaders return count for
>> > nothing, but just want to be clear).
>> > -Brett
>>
>>
>>
>> --
>> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul 24 19:50:22 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 24 Jul 2009 10:50:22 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com> 
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>
Message-ID: <bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com>

On Fri, Jul 24, 2009 at 10:39, Guido van Rossum <guido at python.org> wrote:

> Still works, at least in some old 3.1 I had lying around:
>
> $ python3.1
> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.modules['string'] = None
> >>> import string
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ImportError: No module named string
> >>>
> $ python3.1
> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import string
> >>>
> $
>
> The experiment should be easily repeatable. :-)
>

=) Yes, the None raising ImportError semantics can be added easily and
codified as official import semantics in 3.1. I was talking about the "None
triggers another import as with relative imports" semantics and if there was
some rather convoluted way to trigger that. I am guessing not as that would
require a value of -1 for level which is no longer valid.

I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing, should I
put the failing test in now and decorate it with unittest.expectedFailure
until I get around to fixing it?

-Brett



>
> --Guido
>
> On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon<brett at python.org> wrote:
> >
> >
> > On Thu, Jul 23, 2009 at 20:18, Guido van Rossum <guido at python.org>
> wrote:
> >>
> >> So, I guess, we'll live with it for a while longer. Given that it
> >> managed to evade our attention for so long, I think that's fine.
> >
> > Can someone double-check me that the semantics can even be triggered in
> 3.1?
> > I just tried and couldn't come up with anything. Heck, I quick search for
> a
> > Py_None comparison in 3.1's import.c turned up nothing useful (other than
> > mark_miss() is the function used to set None in sys.modules). We might
> have
> > actually already removed it or made it so that the semantics can't be
> > triggered.
> >
> >>
> >> I agree that there's no reason for a None result from loaders to be
> >> interpreted the same way, assuming that's not how it works ATM.
> >>
> >> And we can live with import and importlib differing on this in 3.1
> >> (though you could call it a bug in importlib and fix it for 3.1.1 --
> >> not sure if you were planning on that).
> >
> > I can if people can trigger the semantics somehow so I have a test to go
> by.
> > -Brett
> >
> >>
> >> --Guido
> >>
> >> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon<brett at python.org> wrote:
> >> >
> >> >
> >> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org
> >
> >> > wrote:
> >> >>
> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >> >
> >> >> >
> >> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson
> >> >> > <benjamin at python.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >> >> > None in Python 3.1 is really useless in terms of its semantics
> in
> >> >> >> > relative
> >> >> >> > imports; importlib doesn't support it and still passes as
> >> >> >> > __import__
> >> >> >> > (at
> >> >> >> > least last time I ran the test suite that way). I thought we had
> >> >> >> > agreed
> >> >> >> > a
> >> >> >> > while back that supporting None was not warranted in Python 3.0?
> >> >> >> > Otherwise I
> >> >> >> > will do whatever work is necessary for this to happen.
> >> >> >>
> >> >> >> I think it's still nice for the rare cases where you need to trick
> a
> >> >> >> module into thinking another one doesn't exist.
> >> >> >
> >> >> > But None does not strictly mean "I don't exist". None is supposed
> to
> >> >> > trigger
> >> >> > an another import attempt for the module with a top-level name.
> It's
> >> >> > that
> >> >> > extra import trigger that has no real use in 3.0 and just
> complicates
> >> >> > import
> >> >> > semantics (IMO) needlessly. If you want a module to not exist then
> >> >> > you
> >> >> > either stick something else in (e.g. '42') or we remove the special
> >> >> > semantics for None (which I thought we had).
> >> >>
> >> >>
> >> >> I didn't realize None had other semantics attached to it. (Imagine
> >> >> that dealing with import!) +1 for making it simply indicate an
> >> >> ImportError.
> >> >
> >> > I'm +0 with having import raise ImportError if None is set in
> >> > sys.modules as
> >> > long as we don't suddenly expect loaders to trigger the same thing if
> >> > they
> >> > return None (actually, as of right now what loaders return count for
> >> > nothing, but just want to be clear).
> >> > -Brett
> >>
> >>
> >>
> >> --
> >> --Guido van Rossum (home page: http://www.python.org/~guido/)
> >
> >
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090724/ee23d5d7/attachment.htm>

From brett at python.org  Fri Jul 24 19:54:55 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 24 Jul 2009 10:54:55 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<ca471dc20907231305h579a61f3p83b53629df1bc380@mail.gmail.com> 
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com> 
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>
Message-ID: <bbaeab100907241054u2330e9b6s76df9c3e2019184a@mail.gmail.com>

On Fri, Jul 24, 2009 at 10:39, Guido van Rossum <guido at python.org> wrote:

> Still works, at least in some old 3.1 I had lying around:
>

Which reminds me, do we want to change the error message for ImportError in
this case to say something like "import of {0} halted; None found in
sys.modules"? Might be a tough exception to debug if you are not aware that
some library shoved None into sys.modules.



>
> $ python3.1
> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.modules['string'] = None
> >>> import string
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ImportError: No module named string
> >>>
> $ python3.1
> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import string
> >>>
> $
>
> The experiment should be easily repeatable. :-)
>
> --Guido
>
> On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon<brett at python.org> wrote:
> >
> >
> > On Thu, Jul 23, 2009 at 20:18, Guido van Rossum <guido at python.org>
> wrote:
> >>
> >> So, I guess, we'll live with it for a while longer. Given that it
> >> managed to evade our attention for so long, I think that's fine.
> >
> > Can someone double-check me that the semantics can even be triggered in
> 3.1?
> > I just tried and couldn't come up with anything. Heck, I quick search for
> a
> > Py_None comparison in 3.1's import.c turned up nothing useful (other than
> > mark_miss() is the function used to set None in sys.modules). We might
> have
> > actually already removed it or made it so that the semantics can't be
> > triggered.
> >
> >>
> >> I agree that there's no reason for a None result from loaders to be
> >> interpreted the same way, assuming that's not how it works ATM.
> >>
> >> And we can live with import and importlib differing on this in 3.1
> >> (though you could call it a bug in importlib and fix it for 3.1.1 --
> >> not sure if you were planning on that).
> >
> > I can if people can trigger the semantics somehow so I have a test to go
> by.
> > -Brett
> >
> >>
> >> --Guido
> >>
> >> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon<brett at python.org> wrote:
> >> >
> >> >
> >> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson <benjamin at python.org
> >
> >> > wrote:
> >> >>
> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >> >
> >> >> >
> >> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson
> >> >> > <benjamin at python.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> 2009/7/23 Brett Cannon <brett at python.org>:
> >> >> >> > None in Python 3.1 is really useless in terms of its semantics
> in
> >> >> >> > relative
> >> >> >> > imports; importlib doesn't support it and still passes as
> >> >> >> > __import__
> >> >> >> > (at
> >> >> >> > least last time I ran the test suite that way). I thought we had
> >> >> >> > agreed
> >> >> >> > a
> >> >> >> > while back that supporting None was not warranted in Python 3.0?
> >> >> >> > Otherwise I
> >> >> >> > will do whatever work is necessary for this to happen.
> >> >> >>
> >> >> >> I think it's still nice for the rare cases where you need to trick
> a
> >> >> >> module into thinking another one doesn't exist.
> >> >> >
> >> >> > But None does not strictly mean "I don't exist". None is supposed
> to
> >> >> > trigger
> >> >> > an another import attempt for the module with a top-level name.
> It's
> >> >> > that
> >> >> > extra import trigger that has no real use in 3.0 and just
> complicates
> >> >> > import
> >> >> > semantics (IMO) needlessly. If you want a module to not exist then
> >> >> > you
> >> >> > either stick something else in (e.g. '42') or we remove the special
> >> >> > semantics for None (which I thought we had).
> >> >>
> >> >>
> >> >> I didn't realize None had other semantics attached to it. (Imagine
> >> >> that dealing with import!) +1 for making it simply indicate an
> >> >> ImportError.
> >> >
> >> > I'm +0 with having import raise ImportError if None is set in
> >> > sys.modules as
> >> > long as we don't suddenly expect loaders to trigger the same thing if
> >> > they
> >> > return None (actually, as of right now what loaders return count for
> >> > nothing, but just want to be clear).
> >> > -Brett
> >>
> >>
> >>
> >> --
> >> --Guido van Rossum (home page: http://www.python.org/~guido/)
> >
> >
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090724/69d6ef54/attachment-0001.htm>

From guido at python.org  Fri Jul 24 19:58:39 2009
From: guido at python.org (Guido van Rossum)
Date: Fri, 24 Jul 2009 10:58:39 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<bbaeab100907231931w3fb62de2lf65e361fe8cea315@mail.gmail.com> 
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com> 
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com> 
	<bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com>
Message-ID: <ca471dc20907241058p394037bfh4b0af79b54edc995@mail.gmail.com>

On Fri, Jul 24, 2009 at 10:50 AM, Brett Cannon<brett at python.org> wrote:
>
>
> On Fri, Jul 24, 2009 at 10:39, Guido van Rossum <guido at python.org> wrote:
>>
>> Still works, at least in some old 3.1 I had lying around:
>>
>> $ python3.1
>> Python 3.1a0 (py3k:70152, Mar ?3 2009, 16:55:42)
>> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import sys
>> >>> sys.modules['string'] = None
>> >>> import string
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> ImportError: No module named string
>> >>>
>> $ python3.1
>> Python 3.1a0 (py3k:70152, Mar ?3 2009, 16:55:42)
>> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import string
>> >>>
>> $
>>
>> The experiment should be easily repeatable. :-)
>
> =) Yes, the None raising ImportError semantics can be added easily and
> codified as official import semantics in 3.1.

I thought it was too simple. :-) Maybe I should go back to bed and
nurse my cough some more instead of playing hookie from being sick and
indulging in email.

> I was talking about the "None
> triggers another import as with relative imports" semantics and if there was
> some rather convoluted way to trigger that.

I can't think of one and doubt you will find one -- it was introduced
(in the earliest of early package import implementations) to optimize
the problem we had with code living inside a package trying to stat()
a toplevel module locally each time an import of it was processed (a
problem if a package contains lots of modules each of which import os,
sys, etc.). Since we no longer to relative import, I don't see how
there could be another use -- if it's not there it's not there.

> I am guessing not as that would
> require a value of -1 for level which is no longer valid.

Ah. Sure.

> I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing, should I
> put the failing test in now and decorate it with unittest.expectedFailure
> until I get around to fixing it?

Oh, I don't know. All this new-fangled technology... Can't you just
leave it in your workspace, unsubmitted, until you get to fixing it?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul 24 20:05:15 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 24 Jul 2009 11:05:15 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <ca471dc20907241058p394037bfh4b0af79b54edc995@mail.gmail.com>
References: <h4ac0g$519$1@ger.gmane.org>
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com> 
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com> 
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com> 
	<bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com> 
	<ca471dc20907241058p394037bfh4b0af79b54edc995@mail.gmail.com>
Message-ID: <bbaeab100907241105h35caaadbw79e985a8f50c8c9b@mail.gmail.com>

On Fri, Jul 24, 2009 at 10:58, Guido van Rossum <guido at python.org> wrote:

> On Fri, Jul 24, 2009 at 10:50 AM, Brett Cannon<brett at python.org> wrote:
> >
> >
> > On Fri, Jul 24, 2009 at 10:39, Guido van Rossum <guido at python.org>
> wrote:
> >>
> >> Still works, at least in some old 3.1 I had lying around:
> >>
> >> $ python3.1
> >> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import sys
> >> >>> sys.modules['string'] = None
> >> >>> import string
> >> Traceback (most recent call last):
> >>  File "<stdin>", line 1, in <module>
> >> ImportError: No module named string
> >> >>>
> >> $ python3.1
> >> Python 3.1a0 (py3k:70152, Mar  3 2009, 16:55:42)
> >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import string
> >> >>>
> >> $
> >>
> >> The experiment should be easily repeatable. :-)
> >
> > =) Yes, the None raising ImportError semantics can be added easily and
> > codified as official import semantics in 3.1.
>
> I thought it was too simple. :-) Maybe I should go back to bed and
> nurse my cough some more instead of playing hookie from being sick and
> indulging in email.
>

I'm glad I can help provide a distraction. And I hope you get over whatever
you have soon.


>
> > I was talking about the "None
> > triggers another import as with relative imports" semantics and if there
> was
> > some rather convoluted way to trigger that.
>
> I can't think of one and doubt you will find one -- it was introduced
> (in the earliest of early package import implementations) to optimize
> the problem we had with code living inside a package trying to stat()
> a toplevel module locally each time an import of it was processed (a
> problem if a package contains lots of modules each of which import os,
> sys, etc.). Since we no longer to relative import, I don't see how
> there could be another use -- if it's not there it's not there.
>
> > I am guessing not as that would
> > require a value of -1 for level which is no longer valid.
>
> Ah. Sure.
>
> > I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing,
> should I
> > put the failing test in now and decorate it with unittest.expectedFailure
> > until I get around to fixing it?
>
> Oh, I don't know. All this new-fangled technology... Can't you just
> leave it in your workspace, unsubmitted, until you get to fixing it?
>

Oh sure, I normally just leave it in my scratch svn checkout until I get
around to it. But since we now have the decorator I was wondering if any
discussion came up while I was gone of using it within our
tests when we happen to get a working (failing) test case before we
have a patch to fix it.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090724/78f974bb/attachment.htm>

From solipsis at pitrou.net  Fri Jul 24 20:13:43 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 24 Jul 2009 18:13:43 +0000 (UTC)
Subject: [Python-Dev] Document None values in sys.modules?
References: <h4ac0g$519$1@ger.gmane.org>
	<1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com>
	<bbaeab100907231942y464409eegad723f8600650137@mail.gmail.com>
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com>
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com>
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com>
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com>
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com>
	<bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com>
	<ca471dc20907241058p394037bfh4b0af79b54edc995@mail.gmail.com>
	<bbaeab100907241105h35caaadbw79e985a8f50c8c9b@mail.gmail.com>
Message-ID: <loom.20090724T181211-234@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:
> > Oh, I don't know. All this new-fangled technology... Can't you just
> > leave it in your workspace, unsubmitted, until you get to fixing it?
> 
> Oh sure, I normally just leave it in my scratch svn checkout until I get
around to it.


What? You're not even using a DVCS?



From sridharr at activestate.com  Fri Jul 24 20:47:32 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Fri, 24 Jul 2009 11:47:32 -0700
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<4A68264C.9060700@gmail.com>
	<36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
	<4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com>
Message-ID: <op.uxk45ezs1q4jlt@homeless.vc.shawcable.net>

On Thu, 23 Jul 2009 06:11:38 -0700, Jesse Noller <jnoller at gmail.com> wrote:

> Then why not include pip, easy_install, and this bash script I use to
> install packages into core? The more the merrier, right?
> Answer: None of these are standards, and as nick points out, there's
> issues with sysadmins, security, and other things. Not to mention
> they're fundamentally not part of the language.
> At the language summit, this was hashed out quite a bit. I think most
> everyone agreed that tools like easy_install, pip, virtualenv,
> zc.buildout, etc simply do not belong in core python.

Once PEP 376 is implemented, my suggestion is to at least link to these  
tools (if not write a paragraph) as way of pointing users to alternative  
package managers that uses the 'uninstall' API. A mention of PyPI would  
also be helpful.

-srid

From brett at python.org  Fri Jul 24 22:50:04 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 24 Jul 2009 13:50:04 -0700
Subject: [Python-Dev] Document None values in sys.modules?
In-Reply-To: <loom.20090724T181211-234@post.gmane.org>
References: <h4ac0g$519$1@ger.gmane.org>
	<1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 
	<bbaeab100907231950h67ff9349vd0695c1b08e26b44@mail.gmail.com> 
	<ca471dc20907232018q42caf1d9w26dd8e589fe1e280@mail.gmail.com> 
	<bbaeab100907232035t49b43fdav422eded4489593d6@mail.gmail.com> 
	<ca471dc20907241039p63a45b94p23b4149c266a2c95@mail.gmail.com> 
	<bbaeab100907241050w3e83e0c8yec76acf58a2b5125@mail.gmail.com> 
	<ca471dc20907241058p394037bfh4b0af79b54edc995@mail.gmail.com> 
	<bbaeab100907241105h35caaadbw79e985a8f50c8c9b@mail.gmail.com> 
	<loom.20090724T181211-234@post.gmane.org>
Message-ID: <bbaeab100907241350x78151794o593f2a6b327aa139@mail.gmail.com>

On Fri, Jul 24, 2009 at 11:13, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Brett Cannon <brett <at> python.org> writes:
> > > Oh, I don't know. All this new-fangled technology... Can't you just
> > > leave it in your workspace, unsubmitted, until you get to fixing it?
> >
> > Oh sure, I normally just leave it in my scratch svn checkout until I get
> around to it.
>
>
> What? You're not even using a DVCS?


Not for Python; waiting for hg switch (all my other code uses hg). Never got
a workflow involving both hg and svn up and going (and don't even mention
gitsvn).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090724/8fa546ea/attachment-0001.htm>

From brett at python.org  Fri Jul 24 23:05:42 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 24 Jul 2009 14:05:42 -0700
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <op.uxk45ezs1q4jlt@homeless.vc.shawcable.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net> 
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<4A68264C.9060700@gmail.com>
	<36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com>
	<4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com> 
	<op.uxk45ezs1q4jlt@homeless.vc.shawcable.net>
Message-ID: <bbaeab100907241405s677cd327h210354480e3cb825@mail.gmail.com>

On Fri, Jul 24, 2009 at 11:47, Sridhar Ratnakumar
<sridharr at activestate.com>wrote:

> On Thu, 23 Jul 2009 06:11:38 -0700, Jesse Noller <jnoller at gmail.com>
> wrote:
>
>  Then why not include pip, easy_install, and this bash script I use to
>> install packages into core? The more the merrier, right?
>> Answer: None of these are standards, and as nick points out, there's
>> issues with sysadmins, security, and other things. Not to mention
>> they're fundamentally not part of the language.
>> At the language summit, this was hashed out quite a bit. I think most
>> everyone agreed that tools like easy_install, pip, virtualenv,
>> zc.buildout, etc simply do not belong in core python.
>>
>
> Once PEP 376 is implemented, my suggestion is to at least link to these
> tools (if not write a paragraph) as way of pointing users to alternative
> package managers that uses the 'uninstall' API. A mention of PyPI would also
> be helpful.


You can put that in the wiki, but it does not belong in the core
documentation (except the PyPI mention as we do control that). That simply
becomes a maintenance nightmare for use as we then have to keep up with the
releases of external tools that we have no direct connection with. Plus it
also give the illusion that Python has blessed these tools when in fact we
have not.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090724/05df68db/attachment.htm>

From david.lyon at preisshare.net  Sat Jul 25 01:33:42 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 24 Jul 2009 19:33:42 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <h4clgl$cfe$1@ger.gmane.org>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
	<h4clgl$cfe$1@ger.gmane.org>
Message-ID: <74d8c1df41dac74a14d451324e0adfc7@preisshare.net>

On Fri, 24 Jul 2009 17:54:09 +0200, Georg Brandl <g.brandl at gmx.net> wrote:
> David Lyon schrieb:
>> In executable form, the Package Manager does not require wxWidgets
>> to be installed.
>> 
>> There is no dependency for this to be installed.
> 
> What does "in exectuable form" mean?

Compiled with py2exe.

An executable ".exe" file is created under windows.

David

From david.lyon at preisshare.net  Sat Jul 25 02:09:28 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 24 Jul 2009 20:09:28 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
	<h4clgl$cfe$1@ger.gmane.org>
	<79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com>
Message-ID: <ea748e1963a55b590c5c69c812f37b7b@preisshare.net>

On Fri, 24 Jul 2009 17:08:32 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> I read this as meaning that David was proposing to ship a built
> application (on Windows, bundled up with something like py2exe, I
> guess) and any supporting DLLs such as the wxWindows ones would be
> bundled in, but the wxPython package would *not* be shipped as part of
> the standard library.

That's correct.

> In my view, for an "application" to be shipped with Python, whether it
> be Idle, or a package manager, or pydoc, or whatever, it must:
> - be written as a Python script
> - depend only on packages shipped with the standard library

This is true for the Package Manager in executable form.

> Given that David's package manager uses setuptools to do the installs
> (and now it appears it uses wxPython for its GUI) that immediately
> disqualifies it for inclusion in my view - regardless of its value as
> a package.

Yes, It offers the choice to install via setuptools.

As well as PIP.

And it will be extended to Enstaller.

Presently it used pkg_resources to read the list of packages installed
which is part of setuptools. I was told it was the "right" and only way
to read a list of packages. If using this package presents a problem, then 
it can be removed and I will replace the functionality with my own code.

If the gui needs to be rewritten in TK this can be accomplished.

I'm in no way pushing any GUI toolkit, and this was the one suggested to
me by many. Redoing to meet the above criteria is achievable.

If there's any other objections, can they be raised now before I
start a rewrite? Thank you.

David



From david.lyon at preisshare.net  Sat Jul 25 02:24:44 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 24 Jul 2009 20:24:44 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <h4b2fd$6uc$1@ger.gmane.org>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
Message-ID: <2e29b19bb5c64ce035f2c2a23f9f3534@preisshare.net>

On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes <lists at cheimes.de>
wrote:
> I'm sorry to inform you that a wxWindows based solution has zero change
> to get into the Python standard library ever. 

Is that a personal preference or is there a software engineering reason
for this?

I wasn't suggesting including it in the standard library as I like others
know how troublesome it can be.

> We are not going to add
> another GUI toolkit to the core distribution. We might even remove TK
> from the core and ship it as a separate package like bsddb3.

So is this a bias against native window applications?

Does the application need to be web based?

or console based?

or TK based... ?

For it to be acceptable..

Thank you

David


From ben+python at benfinney.id.au  Sat Jul 25 02:40:52 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 25 Jul 2009 10:40:52 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
Message-ID: <871vo561d7.fsf@benfinney.id.au>

david.lyon at preisshare.net writes:

> Distutils was once seperate and was then included in the standard
> python.
> 
> So i guess that I am working with the same goal in mind.

I interpret this as expressing your intent to (eventually) have your
application included in standard Python.

David Lyon <david.lyon at preisshare.net> writes:

> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes <lists at cheimes.de>
> wrote:
> > I'm sorry to inform you that a wxWindows based solution has zero
> > change to get into the Python standard library ever.
> 
> I wasn't suggesting including it in the standard library as I like
> others know how troublesome it can be.

This appears to contradict what you said above.

> > We are not going to add another GUI toolkit to the core
> > distribution. We might even remove TK from the core and ship it as a
> > separate package like bsddb3.
> 
> So is this a bias against native window applications?

No, it's a bias against adding things to Python that depend on things
not already in Python.

Are you, or are you not, intending for your application to be in
standard Python?

If that is your intent, I think Christian's answer is valid:
applications, especially ones that depend on third-party widget
libraries, shouldn't get added to Python.

If that is not your intent, then your application shouldn't be mentioned
in standard Python documentation.

-- 
 \                ?Stop ? Drive sideways.? ?detour sign, Kyushu, Japan |
  `\                                                                   |
_o__)                                                                  |
Ben Finney


From exarkun at divmod.com  Sat Jul 25 03:06:10 2009
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Fri, 24 Jul 2009 21:06:10 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <871vo561d7.fsf@benfinney.id.au>
Message-ID: <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>

On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney <ben+python at benfinney.id.au> wrote:
> [snip]
>
>If that is not your intent, then your application shouldn't be mentioned
>in standard Python documentation.
>

Hm.  But docutils isn't part of the standard library, and the documentation
refers to it.  And the docs link to ActivePython and Enthought's Python
distribution.  And they link to things like "Collin Winter?s functional
module" and Django and PyOpenGL and so forth.

I don't think anyone would seriously suggest removing all references to
third-party software from the Python documentation.  Perhaps the criterion
that David is looking to have is explained is that people would probably
prefer for his app to get a bit more popular before Python itself starts
pointing people in its direction.

Jean-Paul

From david.lyon at preisshare.net  Sat Jul 25 03:04:48 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 24 Jul 2009 21:04:48 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <871vo561d7.fsf@benfinney.id.au>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
Message-ID: <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>

On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney <ben+python at benfinney.id.au>
wrote:
> I interpret this as expressing your intent to (eventually) have your
> application included in standard Python.

It's my intention to get a Package Manager included in standard
python - yes.

>> I wasn't suggesting including it in the standard library as I like
>> others know how troublesome it can be.
> 
> This appears to contradict what you said above.

Not at all. In source form the pythonpkgmgr requires wx package. In
executable form it does not.

> No, it's a bias against adding things to Python that depend on things
> not already in Python.

That implies that nothing new can be added then. 

> Are you, or are you not, intending for your application to be in
> standard Python?
> 
> If that is your intent, I think Christian's answer is valid:
> applications, especially ones that depend on third-party widget
> libraries, shouldn't get added to Python.
> 
> If that is not your intent, then your application shouldn't be mentioned
> in standard Python documentation.

The Python Package Manager can be written to work in console mode.

It can be redone to work with the TK that already ships.

That doesn't present any great problem.

If it was the interests of python users that were foremost, being
able to install packages in a modern way - Then I think there would 
be more room to move than this. It's no different than any other 
windows application that ships with dll files.

I fully appreciate that changes could be necessary and I'm prepared to 
accomodate them.

Not much more I can do than that..

David




From ben+python at benfinney.id.au  Sat Jul 25 03:21:12 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 25 Jul 2009 11:21:12 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <871vo561d7.fsf@benfinney.id.au>
	<20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>
Message-ID: <87ws5x4kxj.fsf@benfinney.id.au>

Jean-Paul Calderone <exarkun at divmod.com> writes:

> Hm. But docutils isn't part of the standard library, and the
> documentation refers to it.

Fair enough, because the documentation is generated using Docutils.

> And the docs link to ActivePython and Enthought's Python distribution.

I consider those to be exceptions that prove the rule: they are
?standard Python? in a different form.

> And they link to things like "Collin Winter?s functional module" and
> Django and PyOpenGL and so forth.

I don't know enough about those to comment.

> I don't think anyone would seriously suggest removing all references to
> third-party software from the Python documentation.

No, and nor did I. I'm suggesting that *adding* such a reference has a
high barrier.

> Perhaps the criterion that David is looking to have is explained is
> that people would probably prefer for his app to get a bit more
> popular before Python itself starts pointing people in its direction.

Right. The process is usually in the following order:

* get the code registered on PyPI

* get a community of users and make the code meet their needs well

* discuss adding some or all of the code to Python, by (in recent years)
  raising the topic on ?python-ideas?

* defend the proposal against all comers to ensure it's actually a good
  idea

* await further instructions

-- 
 \      ?It seems intuitively obvious to me, which means that it might |
  `\                                           be wrong.? ?Chris Torek |
_o__)                                                                  |
Ben Finney


From ben+python at benfinney.id.au  Sat Jul 25 03:25:27 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 25 Jul 2009 11:25:27 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
Message-ID: <87skgl4kqg.fsf@benfinney.id.au>

David Lyon <david.lyon at preisshare.net> writes:

> Not at all. In source form the pythonpkgmgr requires wx package. In
> executable form it does not.

The only way it could be added is in source form; that's essential for
free software like Python. So, if it's not suitable for adding to Python
in source form, it's not suitable for adding to Python.

> > No, it's a bias against adding things to Python that depend on things
> > not already in Python.
> 
> That implies that nothing new can be added then. 

You omit the important part: adding a new thing to Python *so long as it
doesn't depend on anything outside Python*.

> The Python Package Manager can be written to work in console mode.

I think this would be best. The functionality you often discuss around
this tool would be best implemented independently of any UI. It would,
in fact, be best to work with the team performing ongoing active
standardisation of distutils functionality.

-- 
 \        ?Consider the daffodil. And while you're doing that, I'll be |
  `\              over here, looking through your stuff.? ?Jack Handey |
_o__)                                                                  |
Ben Finney


From pje at telecommunity.com  Sat Jul 25 03:37:35 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 24 Jul 2009 21:37:35 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <ea748e1963a55b590c5c69c812f37b7b@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
	<h4clgl$cfe$1@ger.gmane.org>
	<79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com>
	<ea748e1963a55b590c5c69c812f37b7b@preisshare.net>
Message-ID: <20090725013432.793043A4099@sparrow.telecommunity.com>

At 08:09 PM 7/24/2009 -0400, David Lyon wrote:
>Presently it used pkg_resources to read the list of packages installed
>which is part of setuptools. I was told it was the "right" and only way
>to read a list of packages.

"Right" is relative, but right now it is certainly the *only* way to 
read a list of installed packages, that's compatible with both 
distutils and setuptools-based ways of installing packages (including 
of course pip, enstaller, and buildout).

PEP 376 proposes a stdlib replacement for a portion of this 
functionality, but the status of whether it's actually going to 
support anything other than distutils, pip, and setuptools' 
backward-compatibility mode is apparently still up for grabs.


From ben+python at benfinney.id.au  Sat Jul 25 03:38:29 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 25 Jul 2009 11:38:29 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
Message-ID: <87ocr94k4q.fsf@benfinney.id.au>

Ben Finney <ben+python at benfinney.id.au> writes:

> The functionality you often discuss around this tool would be best
> implemented independently of any UI. It would, in fact, be best to
> work with the team performing ongoing active standardisation of
> distutils functionality.

Sloppy use of ?in fact?. That should be ?in my opinion?.

-- 
 \        ?Our wines leave you nothing to hope for.? ?restaurant menu, |
  `\                                                       Switzerland |
_o__)                                                                  |
Ben Finney


From horos11 at gmail.com  Sat Jul 25 03:43:59 2009
From: horos11 at gmail.com (Edward Peschko)
Date: Fri, 24 Jul 2009 18:43:59 -0700
Subject: [Python-Dev] command line attachable debugger
In-Reply-To: <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net>
References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
	<82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net>
Message-ID: <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com>

> Winpdb is *exactly* what you asked for, so if it's not what you're looking
> for you'll need to be more specific about what you want that it doesn't
> do...
>
> James
>

Fair enough, I was assuming that winpdb didn't have a command line interface,
but I was mistaken (rpdb2).

In any case, I would argue that attach functionality is something that
should be put
into the core, along with pdb, as a standard. What if I want to use
pudb? or pydb? or the interface which goes with perl?

There should be a standard mechanism for debuggers to talk to a python
process; not one-offs for each debugger, probably at the level of the
python executable (the same way that gcc lets gdb attach with a pid..

Ed

From david.lyon at preisshare.net  Sat Jul 25 04:20:05 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 24 Jul 2009 22:20:05 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <87skgl4kqg.fsf@benfinney.id.au>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
Message-ID: <fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>

On Sat, 25 Jul 2009 11:25:27 +1000, Ben Finney <ben+python at benfinney.id.au>
> You omit the important part: adding a new thing to Python *so long as it
> doesn't depend on anything outside Python*.

I'm signing out on this silly discussion for now....

Any python program is dependant on things "outside" python. For
example an operating system.. a computer... and a user perhaps
for without such it wouldn't be possible to generate the all important
1Di0t errors that we so often have to deal with.

>> The Python Package Manager can be written to work in console mode.
> 
> I think this would be best. 

Haha - I'm glad somebody took this seriously... It was a sort of a joke 
comment but it's a serious possibility.

> The functionality you often discuss around
> this tool would be best implemented independently of any UI. 

It is anyway. That's why doing an entirely different UI isn't any
major issue. I have a class and a possible command line interface
behind the scenes.

> It would, in fact, be best to work with the team performing ongoing 
> active standardisation of distutils functionality.

I am already doing that.

But there is a bias against windows development and a bias
against native applications. That's fine because I know they
are using python on different platforms.

I'm working on a proposal to make setup.py object oriented
and  "modern".

http://wiki.python.org/moin/Distutils/Proposals

So I'm doing as much as I can - really.

David



From orsenthil at gmail.com  Sat Jul 25 06:03:23 2009
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Sat, 25 Jul 2009 09:33:23 +0530
Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2
Message-ID: <20090725040323.GA6260@ubuntu.ubuntu-domain>

Issue: http://bugs.python.org/issue1424152   mentions about HTTPS
Support via proxy in urllib2.

This is fixed in the trunk (Revision 72880), but there has been number
of valid requests to backport it Python 2.6. While I agree and ready
to backport to Python 2.6, I would like to ask here if there are any
objections in this front.

I am ready with the Python 3.x patch as well (with tests passed and
tested under proxy setup). I shall apply that too, which is pending
for a while now. :(


-- 
Senthil
Wad some power the giftie gie us
To see oursels as others see us.
		-- R. Burns

From stephen at xemacs.org  Sat Jul 25 07:18:50 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 25 Jul 2009 14:18:50 +0900
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <ea748e1963a55b590c5c69c812f37b7b@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<h4b2fd$6uc$1@ger.gmane.org>
	<5ecc64238b6e429c72441815bf818f48@preisshare.net>
	<h4clgl$cfe$1@ger.gmane.org>
	<79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com>
	<ea748e1963a55b590c5c69c812f37b7b@preisshare.net>
Message-ID: <873a8l49xh.fsf@uwakimon.sk.tsukuba.ac.jp>

David Lyon writes:
 > On Fri, 24 Jul 2009 17:08:32 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
 > > I read this as meaning that David was proposing to ship a built
 > > application (on Windows, bundled up with something like py2exe, I
 > > guess) and any supporting DLLs such as the wxWindows ones would be
 > > bundled in, but the wxPython package would *not* be shipped as part of
 > > the standard library.
 > 
 > That's correct.

-1, then.  Python is open source.  Anything shipped with the standard
distribution should be buildable with the Pythonic tools in the
standard distribution.  Obviously the C compiler and operating system
services are non-Pythonic, and the line is pretty fuzzy, but I would
not want to extend similar consideration to wxPython.

This kind of thing is what ActiveState and the OS distros are for.

If you say this is discriminating against Windows, well, it probably
is, but it should be addressed with a Windows solution, such as a
script to generate MSIs.

From glyph at twistedmatrix.com  Sat Jul 25 07:50:09 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sat, 25 Jul 2009 01:50:09 -0400
Subject: [Python-Dev] command line attachable debugger
In-Reply-To: <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com>
References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
	<82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net>
	<5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com>
Message-ID: <d9047f780907242250h3d54b54o9be7d7a39e62b719@mail.gmail.com>

On Fri, Jul 24, 2009 at 9:43 PM, Edward Peschko <horos11 at gmail.com> wrote:


> There should be a standard mechanism for debuggers to talk to a python
> process; not one-offs for each debugger, probably at the level of the
> python executable (the same way that gcc lets gdb attach with a pid..
>

Sounds like this is moving into hypothetical territory better-suited to
python-ideas.  (Although I'm sure that if you wanted to contribute polished,
tested code for a standard remote debugger interface, few people would
complain.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090725/5fdc9c39/attachment.htm>

From ben+python at benfinney.id.au  Sat Jul 25 08:00:13 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 25 Jul 2009 16:00:13 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
Message-ID: <87fxcl480i.fsf@benfinney.id.au>

David Lyon <david.lyon at preisshare.net> writes:

> On Sat, 25 Jul 2009 11:25:27 +1000, Ben Finney <ben+python at benfinney.id.au>
> > You omit the important part: adding a new thing to Python *so long
> > as it doesn't depend on anything outside Python*.
> 
> I'm signing out on this silly discussion for now....
> 
> Any python program is dependant on things "outside" python.

I think you know quite well what ?depend on? means in this instance,
so this is taking it to silly extremes.

-- 
 \             ?For your convenience we recommend courteous, efficient |
  `\                            self-service.? ?supermarket, Hong Kong |
_o__)                                                                  |
Ben Finney


From nyamatongwe at gmail.com  Sat Jul 25 08:24:03 2009
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Sat, 25 Jul 2009 16:24:03 +1000
Subject: [Python-Dev] command line attachable debugger
In-Reply-To: <d9047f780907242250h3d54b54o9be7d7a39e62b719@mail.gmail.com>
References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com>
	<82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net>
	<5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com>
	<d9047f780907242250h3d54b54o9be7d7a39e62b719@mail.gmail.com>
Message-ID: <50862ebd0907242324j6d434d2ahd85f7a433af5e4db@mail.gmail.com>

Glyph Lefkowitz:

> Sounds like this is moving into hypothetical territory better-suited to
> python-ideas.? (Although I'm sure that if you wanted to contribute polished,
> tested code for a standard remote debugger interface, few people would
> complain.)

   There is a remote debugger protocol called DBGP for different
languages (including Python) and debuggers (such as Komodo)
http://xdebug.org/docs-dbgp.php

   Neil

From p.f.moore at gmail.com  Sat Jul 25 11:28:51 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 25 Jul 2009 10:28:51 +0100
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
Message-ID: <79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com>

2009/7/25 David Lyon <david.lyon at preisshare.net>:
>> It would, in fact, be best to work with the team performing ongoing
>> active standardisation of distutils functionality.
>
> I am already doing that.
>
> But there is a bias against windows development and a bias
> against native applications. That's fine because I know they
> are using python on different platforms.

??? I see no bias as you describe in the distutils enhancement work.
Everything so far has been carefully platform-neutral (ie, *not*
biased against Windows, but equally not biased against other operating
systems).

Native applications are by definition not platform neutral. How does
your proposal help Linux users? Mac OS? Solaris?

I think you are seeing bias against Windows where non exists. What
*does* exist is
- A concern that tools work for everyone, regardless of OS
- A limited number of people willing to explain Windows issues so that
they get considered

> I'm working on a proposal to make setup.py object oriented
> and ?"modern".
>
> http://wiki.python.org/moin/Distutils/Proposals

And that looks interesting, and potentially useful (although perhaps
more radical than will end up being possible).

> So I'm doing as much as I can - really.

If your concern is to make things easier for Windows users, then your
application is worthwhile, but it should probably remain an external
project. If it gets overwhelming support, maybe providing a
standardised version with a simplified UI in the core would be an
option. How many users do you have currently?

Paul.

PS Does your application work with the large existing base of
bdist_msi and bdist_wininst installers? Unless it will manage pywin32,
cx_Oracle, PIL, wxPython, pyQT, pygame, numpy etc (all of which are
available in binary form but not as eggs as far as I know, and have
too complex a set of dependencies for me to build locally) it's
useless to me.

From martin at v.loewis.de  Sat Jul 25 11:42:13 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 11:42:13 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
Message-ID: <4A6AD375.8000805@v.loewis.de>

> It's my intention to get a Package Manager included in standard
> python - yes.

In addition to the other constraints you'll have to meet for this
to happen, you also have to wait a rather long time (several years)
before inclusion becomes possible. This time is necessary for the
community to accept your tool, and evaluate it. Ideally, the request
for inclusion should not come from you, but from your users.

> Not at all. In source form the pythonpkgmgr requires wx package. In
> executable form it does not.

If we include it, we *only* include it in source form. Inclusion
in executable form is not acceptable. We also require that it works on
all major systems, not just windows.

>> No, it's a bias against adding things to Python that depend on things
>> not already in Python.
> 
> That implies that nothing new can be added then. 

No. It implies that any addition could only depend on Python (and
OS API) - so things *can* be added. For example, setuptools could
be added by this principle. OTOH, if your tool depends on setuptools,
you'll have to wait for setuptools to get included before inclusion
of your tool can be considered.

Regards,
Martin


From martin at v.loewis.de  Sat Jul 25 11:53:38 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 11:53:38 +0200
Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2
In-Reply-To: <20090725040323.GA6260@ubuntu.ubuntu-domain>
References: <20090725040323.GA6260@ubuntu.ubuntu-domain>
Message-ID: <4A6AD622.2050400@v.loewis.de>

> This is fixed in the trunk (Revision 72880), but there has been number
> of valid requests to backport it Python 2.6. While I agree and ready
> to backport to Python 2.6, I would like to ask here if there are any
> objections in this front.

In msg76855 Greg ruled that it is a new feature, and thus cannot be
backported. I trust his judgement - so to convince me, you would need
to convince Greg first :-)

Regards,
Martin

From martin at v.loewis.de  Sat Jul 25 13:05:20 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 13:05:20 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A68D2BB.8060607@stackless.com>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com>
Message-ID: <4A6AE6F0.2060303@v.loewis.de>

> To Martin: So I disagree. The gc header is not responsible for
> alignment in the first place, but to propagate it, correctly.

I think we are in agreement in this respect. That's the whole point
of the long double: to make the gc head's alignment the maximum
alignment on that platform

> And this fails miserably (in principle) since years.

Here I agree. In principle, it works fine - it is just perhaps
incomplete.

> Proposal: We should use a simple construct that makes the
> gc header size simply a multiple of 8 or 16, whatever needed.
> Even a byte array would be ok.

How would you implement that? In particular, how do you determine
what is needed?

> But please no long double :-)

Well, the long double needs to stay, as for some platforms, long
double is indeed the type with the maximum alignment.

I propose to add another (regular) double into the union.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 25 13:09:57 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 13:09:57 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090723T212150-107@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>
	<loom.20090723T212150-107@post.gmane.org>
Message-ID: <4A6AE805.3000005@v.loewis.de>

>> This may not be recognized so far, because there is no builtin
>> GC-enabled type that embeds a double.
>> But if you create such a type, then the double will be correctly
>> aligned in your object's structure, but then, when the object
>> gets allocated, it is misaligned, because the header size is not a
>> multiple of 8.
> 
> I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU.

That's not really the point. The question is whether the double is
misaligned wrt. the compiler's ABI. The compiler (or the operating
system) specifies the alignment for all primitive types, and guarantees
that everything is properly aligned.

> Alignment is primarily important to avoid access violations on CPUs and
> datatypes which don't support arbitrary alignment, although it can also be
> useful for performance reasons.

Alignment may also have propagated into other aspects. For example, if
you have

  double foo;
  intptr_t bar = (intptr_t)&foo;

then it may be guaranteed (by the ABI) that bar&7 == 0. So we really
need to keep all promises wrt. alignment intact.

> In any case, you seem to be right on this particular point: the PyGC_Head union
> should probably contain a "double" alternative in addition to the "long double"
> (and perhaps even a "long long" one).

I agree.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 25 13:13:03 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 13:13:03 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090723T213301-657@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>
	<loom.20090723T213301-657@post.gmane.org>
Message-ID: <4A6AE8BF.8040405@v.loewis.de>

> Antoine Pitrou <solipsis <at> pitrou.net> writes:
>> In any case, you seem to be right on this particular point: the PyGC_Head union
>> should probably contain a "double" alternative in addition to the "long double"
>> (and perhaps even a "long long" one).
> 
> Sorry, I realize that this doesn't really address the point.

I don't realize that. Why is your first proposal bad?

> In addition to that union, we should also have a particular mechanism to compute
> what the proper offset should be between the PyGC_Head and the PyObject.

Why is that difficult? It's sizeof(PyGC_Head).

Regards,
Martin

From martin at v.loewis.de  Sat Jul 25 13:15:39 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 13:15:39 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A68E0E0.7010508@stackless.com>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>
	<4A68E0E0.7010508@stackless.com>
Message-ID: <4A6AE95B.20301@v.loewis.de>

> For that reason, I don't like the addition of the opaque header
> too much. If there were an option to make the header explicit,
> we would not have to round up the object size to a multiple
> of (8, 16), but could arrange embedded doubles as they fit the best.

We could add the gc head *after* the object head. However, this will
require manual changes to all types that support GC, so we can't do
that before Python 4.

>> (I disagree, however, that we should remove the "long double". After
>> all, we
>> also want alignment of PyObjects to allow inclusion of a "long double"
>> in them)
> 
> Well, I doubt that a 12 byte long double causes any other
> alignment but 4.

What you apparently fail to understand is this: On some platforms,
sizeof(long double) == 16, and alignof(long double) == 16. It is these
platforms why we need to keep the long double.

Regards,
Martin

From solipsis at pitrou.net  Sat Jul 25 13:30:13 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 25 Jul 2009 11:30:13 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>
	<loom.20090723T213301-657@post.gmane.org>
	<4A6AE8BF.8040405@v.loewis.de>
Message-ID: <loom.20090725T112025-245@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> > In addition to that union, we should also have a particular mechanism to 
compute
> > what the proper offset should be between the PyGC_Head and the PyObject.
> 
> Why is that difficult? It's sizeof(PyGC_Head).

Is it enough to correctly propagate the alignment for, say, a long double in
the following PyObject? I'm sorry, I'm not enough of a C lawyer.

Regards

Antoine.



From david.lyon at preisshare.net  Sat Jul 25 14:57:22 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 25 Jul 2009 08:57:22 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <87fxcl480i.fsf@benfinney.id.au>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
	<87fxcl480i.fsf@benfinney.id.au>
Message-ID: <ff8daa674b133d5cec4ecdbd8a7d8fdb@preisshare.net>

On Sat, 25 Jul 2009 16:00:13 +1000, Ben Finney <ben+python at benfinney.id.au>
wrote:
> I think you know quite well what ?depend on? means in this instance,
> so this is taking it to silly extremes.

haha - yes - no offence. 

It was just bad humour.

Have a nice weekend

David

From david.lyon at preisshare.net  Sat Jul 25 15:41:24 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 25 Jul 2009 09:41:24 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<87bpnb7ufv.fsf@benfinney.id.au>
	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>
	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
	<79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com>
Message-ID: <56b9b173c5deb6563cce6aa93c317b7c@preisshare.net>

On Sat, 25 Jul 2009 10:28:51 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> ??? I see no bias as you describe in the distutils enhancement work.

ok

> Native applications are by definition not platform neutral. How does
> your proposal help Linux users? Mac OS? Solaris?

I'm doing a Linux/Solaris version. But I find issues every day that
must be addressed.

> If your concern is to make things easier for Windows users, then your
> application is worthwhile, but it should probably remain an external
> project. If it gets overwhelming support, maybe providing a
> standardised version with a simplified UI in the core would be an
> option. 

That sounds ok.

> How many users do you have currently?

Approx 250 downloads this (first) month for the windows version. I need
to extend coverage to Linux and the Mac. Given that I'm relatively new
to all this there's a learning curve.

But so far so good.

David
 





From nyamatongwe at gmail.com  Sat Jul 25 16:11:30 2009
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Sun, 26 Jul 2009 00:11:30 +1000
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A6AE6F0.2060303@v.loewis.de>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de>
Message-ID: <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>

Martin v. L?wis:

> I propose to add another (regular) double into the union.

   Adding a regular double as a second dummy gives the same sizes and
alignments with Mingw or MSVC as the original definition with MSVC:

typedef union _gc_head {
       struct {
               union _gc_head *gc_next;
               union _gc_head *gc_prev;
               Py_ssize_t gc_refs;
       } gc;
       long double dummy;  /* force worst-case alignment */
       double dummy2;  /* in case long double doesn't trigger worst-case */
} PyGC_Head;

   In regard to alignment penalties, a simple copy loop for doubles
runs about 20% slower when misaligned on an my AMD processor. Other
x86 processors can be much worse. As much as 2 to 3.25 times according
to
http://msdn.microsoft.com/en-us/library/aa290049%28VS.71%29.aspx

   Neil

From solipsis at pitrou.net  Sat Jul 25 16:19:36 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 25 Jul 2009 14:19:36 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de>
	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
Message-ID: <loom.20090725T141648-749@post.gmane.org>

Neil Hodgson <nyamatongwe <at> gmail.com> writes:
> 
> typedef union _gc_head {
>        struct {
>                union _gc_head *gc_next;
>                union _gc_head *gc_prev;
>                Py_ssize_t gc_refs;
>        } gc;
>        long double dummy;  /* force worst-case alignment */
>        double dummy2;  /* in case long double doesn't trigger worst-case */
> } PyGC_Head;

We could add a PY_LONG_LONG to the mix just in case.
By the way, will this kind of thing be frozen by the PEP ABI?

Regards

Antoine.



From solipsis at pitrou.net  Sat Jul 25 16:22:05 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 25 Jul 2009 14:22:05 +0000 (UTC)
Subject: [Python-Dev] mingw32 and gc-header weirdness
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de>
	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
	<loom.20090725T141648-749@post.gmane.org>
Message-ID: <loom.20090725T142147-20@post.gmane.org>

Antoine Pitrou <solipsis <at> pitrou.net> writes:
> 
> We could add a PY_LONG_LONG to the mix just in case.
> By the way, will this kind of thing be frozen by the PEP ABI?

(I meant the ABI PEP, sorry)




From martin at v.loewis.de  Sat Jul 25 17:22:06 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 25 Jul 2009 17:22:06 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090725T112025-245@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>	<loom.20090723T212150-107@post.gmane.org>	<loom.20090723T213301-657@post.gmane.org>	<4A6AE8BF.8040405@v.loewis.de>
	<loom.20090725T112025-245@post.gmane.org>
Message-ID: <4A6B231E.5010502@v.loewis.de>

>> Why is that difficult? It's sizeof(PyGC_Head).
> 
> Is it enough to correctly propagate the alignment for, say, a long double in
> the following PyObject? I'm sorry, I'm not enough of a C lawyer.

Yes, sizeof must always be a multiple of the alignment. Otherwise, array
elements would be misaligned.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 25 17:25:01 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Jul 2009 17:25:01 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <loom.20090725T141648-749@post.gmane.org>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>
	<4A6AE6F0.2060303@v.loewis.de>	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
	<loom.20090725T141648-749@post.gmane.org>
Message-ID: <4A6B23CD.1010306@v.loewis.de>

> We could add a PY_LONG_LONG to the mix just in case.
> By the way, will this kind of thing be frozen by the PEP ABI?

Yes: alignof(PyGC_HEAD) would be specified as being the maximum
alignment on a platform; sizeof(PyGC_HEAD) would be frozen. The
actual content would not be frozen, i.e. it could be changed in
principle as long as the change would be guaranteed not to affect
the structure size.

Regards,
Martin

From tjreedy at udel.edu  Sat Jul 25 18:47:21 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 25 Jul 2009 12:47:21 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
Message-ID: <h4fcup$tic$1@ger.gmane.org>

David Lyon wrote:

>>> The Python Package Manager can be written to work in console mode.
>> I think this would be best. 
> 
> Haha - I'm glad somebody took this seriously... It was a sort of a joke 
> comment but it's a serious possibility.

I took it seriously too ;-). It seems to me that you project can have at 
least 3 components. 1) the core library module, which might or might not 
be targeted at eventual stdlib inclusion. 2) a small console driver 
script. 3) one or more GUI scripts. A TK version would be good from my 
viewpoint as it does not require download and installation of any other 
binaries. Removal of TK from the Windows distribution will be extremely 
contentious if seriously proposed, so it could not happen for a while.

tjr


From greg at krypto.org  Sat Jul 25 19:05:09 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 25 Jul 2009 10:05:09 -0700
Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2
In-Reply-To: <4A6AD622.2050400@v.loewis.de>
References: <20090725040323.GA6260@ubuntu.ubuntu-domain>
	<4A6AD622.2050400@v.loewis.de>
Message-ID: <52dc1c820907251005k2f52a3bco6c773a464d96e766@mail.gmail.com>

On Sat, Jul 25, 2009 at 2:53 AM, "Martin v. L?wis"<martin at v.loewis.de> wrote:
>> This is fixed in the trunk (Revision 72880), but there has been number
>> of valid requests to backport it Python 2.6. While I agree and ready
>> to backport to Python 2.6, I would like to ask here if there are any
>> objections in this front.
>
> In msg76855 Greg ruled that it is a new feature, and thus cannot be
> backported. I trust his judgement - so to convince me, you would need
> to convince Greg first :-)
>
> Regards,
> Martin

So long as the new set_tunnel() method is hidden in the backport I'm
fine with the patch.  It fixes urllib2 do the right thing with https
and a proxy without any public API change.

-Greg

From greg at krypto.org  Sat Jul 25 19:25:35 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 25 Jul 2009 10:25:35 -0700
Subject: [Python-Dev] pthreads, fork, import, and execvp
In-Reply-To: <9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com>
References: <d2115f4f0605090639h72570784k1f4ef0802b5a8d51@mail.gmail.com> 
	<446A4202.6070608@v.loewis.de>
	<ee2a432c0605172343h30b1f469ydb5dff9435ca371f@mail.gmail.com> 
	<446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> 
	<446CB69E.4090609@v.loewis.de>
	<9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> 
	<3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> 
	<9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com> 
	<9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com>
Message-ID: <52dc1c820907251025j4db82363yea166be14a012ada@mail.gmail.com>

On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters<thomas at python.org> wrote:
>
> So attached (and at http://codereview.appspot.com/96125/show ) is a
> preliminary fix, correcting the problem with os.fork(), os.forkpty() and
> os.fork1(). This doesn't expose a general API for C code to use, for two
> reasons: it's not easy, and I need this fix more than I need the API change
> :-) (I actually need this fix myself for Python 2.4, but it applies fairly
> cleanly.)

This looks good to me.

Your idea of making this an API called a 'fork lock' or something
sounds good (though I think it needs a better name.  PyBeforeFork &
PyAfterFork?).  The subprocess module, for example, disables garbage
collection before forking and restores it afterwards to avoid
http://bugs.python.org/issue1336.  That type of thing could also be
done in such a function.

Related to the above subprocess fork + gc bug.. It'd be nice for
CPython to have code that does the fork+misc twiddling+exec all in one
C call without needing to execute Python code in the child process
prior to the exec().  Most of the inner body of subprocess's
_execute_child() method could be done that way.  (with the obvious
exception of the preexec_fn)

>
> To fix the mutex-across-fork problem correctly we should really acquire
> three locks (the import lock, the GIL and the TLS lock, in that order.) The
> import lock is re-entrant, and the TLS lock is tightly confined to the
> thread-local-storage lookup functions, but the GIL is neither re-entrant nor
> inspectable. That means we can't use pthread_atfork (we can't tell whether
> we have the GIL already or not, right before the fork), nor can we provide a
> convenient API for extension modules to use, really. The best we can do is
> provide three functions, pthread_atfork-style: a 'prepare' function, an
> 'after-fork-in-child' function, and an 'after-fork-in-parent' function. The
> 'prepare' function would start by releasing the GIL, then acquire the import
> lock, the GIL and the TLS lock in that order. It would also record
> *somewhere* the thread_ident of the current thread. The 'in-parent' function
> would simply release the TLS lock and the import lock, and the 'in-child'
> would release those locks, call the threading._at_fork() function, and fix
> up the TLS data, using the recorded thread ident to do lookups. The
> 'in-child' function would replace the current PyOS_AfterFork() function
> (which blindly reinitializes the GIL and the TLS lock, and calls
> threading._at_fork().)
>
> Alternatively we could do what we do now, which is to ignore the fact that
> thread idents may change by fork() and thus that thread-local data may
> disappear, in which case the 'in-child' function could do a little less
> work. I'm suitably scared and disgusted of the TLS implementation, the
> threading implementations we support (including the pthread one) and the way
> we blindly type-pun a pthread_t to a long, that I'm not sure I want to try
> and build something correct and reliable on top of it.
>
> --
> Thomas Wouters <thomas at python.org>
>
> Hi! I'm a .signature virus! copy me into your .signature file to help me
> spread!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>

From brett at python.org  Sun Jul 26 00:28:39 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 25 Jul 2009 15:28:39 -0700
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>
References: <871vo561d7.fsf@benfinney.id.au>
	<20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>
Message-ID: <bbaeab100907251528t60e463d2ybeefc7e38e95d2d6@mail.gmail.com>

On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone <exarkun at divmod.com>wrote:

> On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney <ben+python at benfinney.id.au<ben%2Bpython at benfinney.id.au>>
> wrote:
>
>> [snip]
>>
>> If that is not your intent, then your application shouldn't be mentioned
>> in standard Python documentation.
>>
>>
> Hm.  But docutils isn't part of the standard library, and the documentation
> refers to it.


Why do you think distutils is not part of the standard library? The official
code location is Lib/distutils within the standard library.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090725/e47befbe/attachment.htm>

From brett at python.org  Sun Jul 26 00:35:22 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 25 Jul 2009 15:35:22 -0700
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <bbaeab100907251528t60e463d2ybeefc7e38e95d2d6@mail.gmail.com>
References: <871vo561d7.fsf@benfinney.id.au>
	<20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>
	<bbaeab100907251528t60e463d2ybeefc7e38e95d2d6@mail.gmail.com>
Message-ID: <bbaeab100907251535n8e7b7a8m29c8cc86dd8e33fb@mail.gmail.com>

On Sat, Jul 25, 2009 at 15:28, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone <exarkun at divmod.com>wrote:
>
>> On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney <
>> ben+python at benfinney.id.au <ben%2Bpython at benfinney.id.au>> wrote:
>>
>>> [snip]
>>>
>>> If that is not your intent, then your application shouldn't be mentioned
>>> in standard Python documentation.
>>>
>>>
>> Hm.  But docutils isn't part of the standard library, and the
>> documentation
>> refers to it.
>
>
> Why do you think distutils is not part of the standard library? The
> official code location is Lib/distutils within the standard library.
>

Oops, I misread the project JP mentioned. Ignore me. =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090725/f81e8e60/attachment.htm>

From robert.kern at gmail.com  Sun Jul 26 00:36:44 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 25 Jul 2009 17:36:44 -0500
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <bbaeab100907251528t60e463d2ybeefc7e38e95d2d6@mail.gmail.com>
References: <871vo561d7.fsf@benfinney.id.au>	<20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com>
	<bbaeab100907251528t60e463d2ybeefc7e38e95d2d6@mail.gmail.com>
Message-ID: <h4g1ds$gut$1@ger.gmane.org>

On 2009-07-25 17:28, Brett Cannon wrote:
>
>
> On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone <exarkun at divmod.com
> <mailto:exarkun at divmod.com>> wrote:
>
>     On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney
>     <ben+python at benfinney.id.au <mailto:ben%2Bpython at benfinney.id.au>>
>     wrote:
>
>         [snip]
>
>
>         If that is not your intent, then your application shouldn't be
>         mentioned
>         in standard Python documentation.
>
>
>     Hm.  But docutils isn't part of the standard library, and the
>     documentation
>     refers to it.
>
>
> Why do you think distutils is not part of the standard library? The
> official code location is Lib/distutils within the standard library.

He said "docutils", not "distutils".

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From nyamatongwe at gmail.com  Sun Jul 26 00:44:07 2009
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Sun, 26 Jul 2009 08:44:07 +1000
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A6B23CD.1010306@v.loewis.de>
References: <4A675C10.6060200@stackless.com>
	<loom.20090722T193713-455@post.gmane.org>
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>
	<4A67AEC3.6070000@stackless.com>
	<loom.20090723T090139-199@post.gmane.org>
	<4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de>
	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
	<loom.20090725T141648-749@post.gmane.org>
	<4A6B23CD.1010306@v.loewis.de>
Message-ID: <50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com>

Martin v. L?wis:

> Yes: alignof(PyGC_HEAD) would be specified as being the maximum
> alignment on a platform; sizeof(PyGC_HEAD) would be frozen.

   Maximum alignment currently on x86 is 16 bytes for SSE vector
types. Next year AVX will add 32 byte types and while they are
supposed to work OK with 16 byte alignment, performance will be better
with 32 byte alignment.

   It is possible that some use could be found for vector instructions
in core Python but it is more likely that they will only be used in
specialized extensions that can take care of alignment issues for
their own cases.

http://en.wikipedia.org/wiki/Advanced_Vector_Extensions
http://software.intel.com/en-us/forums/intel-avx-and-cpu-instructions/topic/61891/

   Neil

From martin at v.loewis.de  Sun Jul 26 01:28:08 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 26 Jul 2009 01:28:08 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>	<loom.20090722T193713-455@post.gmane.org>	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	<4A67AEC3.6070000@stackless.com>	<loom.20090723T090139-199@post.gmane.org>	<4A68D2BB.8060607@stackless.com>
	<4A6AE6F0.2060303@v.loewis.de>	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>	<loom.20090725T141648-749@post.gmane.org>	<4A6B23CD.1010306@v.loewis.de>
	<50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com>
Message-ID: <4A6B9508.1030707@v.loewis.de>

>    Maximum alignment currently on x86 is 16 bytes for SSE vector
> types. Next year AVX will add 32 byte types and while they are
> supposed to work OK with 16 byte alignment, performance will be better
> with 32 byte alignment.

That doesn't really matter. What matters is the platform's ABI, i.e.
what the maximum alignment is that the compiler gives, and what the
alignment of a malloc() result is. That is unlikely to change even
if x86 evolves.

Regards,
Martin

From david.lyon at preisshare.net  Sun Jul 26 03:38:55 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 25 Jul 2009 21:38:55 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <h4fcup$tic$1@ger.gmane.org>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>	<87skgl4kqg.fsf@benfinney.id.au>
	<fda6ac9510b99ec5ed42a2c33961ce1d@preisshare.net>
	<h4fcup$tic$1@ger.gmane.org>
Message-ID: <4a601dcc46d513bfd446cbf141654401@preisshare.net>

On Sat, 25 Jul 2009 12:47:21 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
>>>> The Python Package Manager can be written to work in console mode.
>>> I think this would be best. 
>> 
>> Haha - I'm glad somebody took this seriously... It was a sort of a joke 
>> comment but it's a serious possibility.
> 
> I took it seriously too ;-). It seems to me that you project can have at 
> least 3 components. 

It's logical and plausible.

> 1) the core library module, which might or might not be targeted at 
  eventual stdlib inclusion. 

I'm sure they'll fix distutils someday, and if they do, then I'll use
that.

> 2) a small console driver script. 3) one or more GUI scripts. 

It's possible. I'll give it serious consideration.

> A TK version would be good from my 
> viewpoint as it does not require download and installation of any other 
> binaries.

Sure. Who cares if it's not as slick as something else. I'll try
when I have time.

David





From david.lyon at preisshare.net  Sun Jul 26 06:32:55 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 26 Jul 2009 00:32:55 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6AD375.8000805@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
Message-ID: <f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>

On Sat, 25 Jul 2009 11:42:13 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
>> It's my intention to get a Package Manager included in standard
>> python - yes.
> 
> In addition to the other constraints you'll have to meet for this
> to happen, you also have to wait a rather long time (several years)
> before inclusion becomes possible. This time is necessary for the
> community to accept your tool, and evaluate it. Ideally, the request
> for inclusion should not come from you, but from your users.

Yes. But users have been asking for a tool and complaining loudly
about the lack of such a tool. I know you're an extremely competent
and those emails you perphaps floss over. But the pleas are many and
when we compare python to other languages, python rates towards at the
low end of the spectrum its third party-package management facilities.

You can't seriously expect users to wait for years for an integrated
package management tool. Especially on Windows - that's cruel :-)

To date, there just hasn't been any movement on the building of
such a tool due to whatever reasons.

A Package Manager isn't a frivolous "nice-to-have" tool. It's
essential for a new user.

(if you want I can sift the last twelve months worth of ML
 archives and report on how many times easier package management
 has been asked for. And how difficult people are finding it)

> No. It implies that any addition could only depend on Python (and
> OS API) - so things *can* be added. For example, setuptools could
> be added by this principle. OTOH, if your tool depends on setuptools,
> you'll have to wait for setuptools to get included before inclusion
> of your tool can be considered.

Let's get precise. It doesn't "depend" on setuptools. But it will
install setuptools if the user requests to use setuptools/easy_install.

So we should only be back to the lack of a TK interface and the fact that
the Package Manager Project is a new project, and needs to be working
properly on more platforms.

Thanks for your email. I appreciate the feedback, from everyone.

Take care.

David





From martin at v.loewis.de  Sun Jul 26 08:43:07 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 26 Jul 2009 08:43:07 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
Message-ID: <4A6BFAFB.1030807@v.loewis.de>

> You can't seriously expect users to wait for years for an integrated
> package management tool. Especially on Windows - that's cruel :-)

Hmm. I'm -0 on providing a tool whose only purpose is to download
files from a web server. I always use a web browser for that...

> A Package Manager isn't a frivolous "nice-to-have" tool. It's
> essential for a new user.

Hmm. I would expect that a new user is faced with the challenge
of finding out what packages to install more so than with actually
installing them. If they read examples, they will see import
statements, and then they have to find out how to make those work.
Does your tool help with that?

When the user is not so new anymore, I seriously doubt that they
still ask for a package management tool - except perhaps for
dependencies, and here they use easy_install.

Regards,
Martin

From steve at pearwood.info  Sun Jul 26 10:05:07 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 26 Jul 2009 18:05:07 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
Message-ID: <200907261805.08277.steve@pearwood.info>

On Sun, 26 Jul 2009 02:32:55 pm David Lyon wrote:

> > In addition to the other constraints you'll have to meet for this
> > to happen, you also have to wait a rather long time (several years)
> > before inclusion becomes possible. This time is necessary for the
> > community to accept your tool, and evaluate it. Ideally, the
> > request for inclusion should not come from you, but from your
> > users.
>
> Yes. But users have been asking for a tool and complaining loudly
> about the lack of such a tool. I know you're an extremely competent
> and those emails you perphaps floss over. But the pleas are many and
> when we compare python to other languages, python rates towards at
> the low end of the spectrum its third party-package management
> facilities.
>
> You can't seriously expect users to wait for years for an integrated
> package management tool. Especially on Windows - that's cruel :-)

Life is cruel and hard. Anyone who tells you different is selling 
something.

Nobody is saying that users have to wait for years for such a tool. They 
can download it from wherever you host it. But you shouldn't expect the 
Python dev team to accept an unproven tool into the official library 
before demonstrating both the need and the solution. (Just because 
users say they want something, doesn't mean they'll actually use the 
tool you provide -- perhaps they don't know what they really need, and 
perhaps your tool doesn't match their needs.)

I'm a user, and personally I don't want Yet Another Integrated Package 
Management Tool. What I really want is the ability to install Python 
packages using the PM tool I already use, namely yum. (And I'd like a 
pony.) Failing that, I want the Python standard library to be stable 
and reliable. With the greatest of respect, fast-tracking unproven 
tools into the library based on the author's say-so is the first step 
to instability and unreliability.

Putting your software on the Cheeseshop, and making regular 
announcements to the Python community (e.g. on comp.lang.python) will 
be a good way to publicise the tool, and if does meet a need, people 
will use it, and then, if it's good enough and popular enough and 
supported, it may be blessed by inclusion in the standard library.

Although I'm not a Windows user, let me say thank you for your efforts 
on behalf of Python users. Good luck!



-- 
Steven D'Aprano

From ncoghlan at gmail.com  Sun Jul 26 13:44:38 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 26 Jul 2009 21:44:38 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <200907261805.08277.steve@pearwood.info>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
Message-ID: <4A6C41A6.9010500@gmail.com>

Steven D'Aprano wrote:
> I'm a user, and personally I don't want Yet Another Integrated Package 
> Management Tool. What I really want is the ability to install Python 
> packages using the PM tool I already use, namely yum. (And I'd like a 
> pony.)

Picking up on this point... out of curiousity, I threw a couple of
package names at "apt-get -s" on my main (Ubuntu 8.04) PC to see what
worked.

The first two names I tried (numpy, scipy) didn't work, but the next
four (python-numpy, python-scipy, python-django, python-turbogears) all
had hits. Because Turbo Gears incorporates so many other projects, the
simulated run at installing it provided a nice overview of a variety of
other Python packages in the Ubuntu repositories as well.

No doubt a search in Adept would reveal a whole lot more "python-*"
packages that are just a few clicks away for me.

</tangent>

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From david.lyon at preisshare.net  Sun Jul 26 17:00:06 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 26 Jul 2009 11:00:06 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6BFAFB.1030807@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>
	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<4A6BFAFB.1030807@v.loewis.de>
Message-ID: <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net>

On Sun, 26 Jul 2009 08:43:07 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
> Hmm. I'm -0 on providing a tool whose only purpose is to download
> files from a web server. I always use a web browser for that...

It does a lot more than that. Firstly it shows what packages you
already have installed and lets you manage them. Namely, deinstall,
(manually) upgrade, view documentation and examples.

In any case, even if it were only a tool to download packages
it's still consistant with modern appliance design (mobile phones,
programming languages, dedicated hardware) to have an internal
application where a user can pretty easily download "checked"
apps to their device/system.

Nobody gives people just a browser to do that... and tell them
"go browse". It isn't being done like that - except in python.

> Hmm. I would expect that a new user is faced with the challenge
> of finding out what packages to install more so than with actually
> installing them. 

Yes. That's exactly the point. You're 100% right.

That's why the pythonpkgmgr provides a useful search capability
for pypi packages. It's faster and more natural to use a native
app than do it all in the browser. That's true in Perl (cpan) as 
in Python and I'm honestly saying that I can't find any noteable
faults in pypi the way I find it.

> If they read examples, they will see import
> statements, and then they have to find out how to make those work.
> Does your tool help with that?

Yes. It will open the website or homepage to the project/package
in question. And find any documentation that might be stored in
the package directory.

If you only have "import interbasedb" it's a complicated process
for a new user to located the home page or project documentation 
without pythonpkgmgr.

Using pythonpkgmgr, they look on their list of installed packages,
find the "interbasedb" or whatever module, and can get quick
access to the pypi page or the package homepage. It's much easier.

Much faster.. especially if the new user doesn't know the
internals of package storage, pypi and the like.

> When the user is not so new anymore, I seriously doubt that they
> still ask for a package management tool - except perhaps for
> dependencies, and here they use easy_install.

Well, that might be one use-case.

It's true that many python programmers will just want to stick to
their own "known" packages.

However, this tool aids people with more curiosity. Because users can
so easily (and safely) install and deinstall anything.

It's only a click to install a package, and another to deinstall.

At the moment, pythonpkgmgr doesn't handle dependencies except that
which is provided internally by easy_install/pip.

I'm hoping to change this as my experience and understanding grows.

David


From david.lyon at preisshare.net  Sun Jul 26 17:15:22 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 26 Jul 2009 11:15:22 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <200907261805.08277.steve@pearwood.info>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
Message-ID: <ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>

On Sun, 26 Jul 2009 18:05:07 +1000, Steven D'Aprano <steve at pearwood.info>
wrote:
> But you shouldn't expect the 
> Python dev team to accept an unproven tool into the official library 
> before demonstrating both the need and the solution. 

Of course... that's why I started off by asking what the process is.

I am accepting that the current status of pythonpkgmgr is "unproven" so
it will need some time for things to run their course.

> (Just because 
> users say they want something, doesn't mean they'll actually use the 
> tool you provide -- perhaps they don't know what they really need, and 
> perhaps your tool doesn't match their needs.)

Conversely, the process of satisfying needs is to discuss them. And
people have been ever so helpful in that regard.

> I'm a user, and personally I don't want Yet Another Integrated Package 
> Management Tool. What I really want is the ability to install Python 
> packages using the PM tool I already use, namely yum. 

ok - but no alternative to that is available on windows.

pythonpkgmgr is not so different to that. And the idea behind it is
to bring consistancy in package management across the different 
platforms.

It does have configuration that are specific to python, and will
continue to do so. Like being able to specify where one wants the
package installed.

> Putting your software on the Cheeseshop, and making regular 
> announcements to the Python community (e.g. on comp.lang.python) will 
> be a good way to publicise the tool, and if does meet a need, people 
> will use it, and then, if it's good enough and popular enough and 
> supported, it may be blessed by inclusion in the standard library.

Thanks very much. Take care.

David

From fuzzyman at voidspace.org.uk  Sun Jul 26 18:23:59 2009
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 26 Jul 2009 17:23:59 +0100
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
Message-ID: <4A6C831F.5090106@voidspace.org.uk>

David Lyon wrote:
> On Sun, 26 Jul 2009 18:05:07 +1000, Steven D'Aprano <steve at pearwood.info>
> wrote:
>   
>> But you shouldn't expect the 
>> Python dev team to accept an unproven tool into the official library 
>> before demonstrating both the need and the solution. 
>>     
>
> Of course... that's why I started off by asking what the process is.
>
> I am accepting that the current status of pythonpkgmgr is "unproven" so
> it will need some time for things to run their course.
>
>   

It would be great to have  a decent visual package manager for Python. 
It needs to be built on top of the work that Tarek is doing with 
distutils (and be compatible with his Distribute fork of setuptools) and 
it also needs to be established.

The good news is that there is time for both... Aren't there other tools 
that have similar goals? PyPI browser, Yolk etc

It is a shame that it needs to be built on Tk if it is to stand a chance 
of ever being included in the standard library - but with the right 
expertise it is *possible* to create decent looking UIs with Tk.

Michael

>> (Just because 
>> users say they want something, doesn't mean they'll actually use the 
>> tool you provide -- perhaps they don't know what they really need, and 
>> perhaps your tool doesn't match their needs.)
>>     
>
> Conversely, the process of satisfying needs is to discuss them. And
> people have been ever so helpful in that regard.
>
>   
>> I'm a user, and personally I don't want Yet Another Integrated Package 
>> Management Tool. What I really want is the ability to install Python 
>> packages using the PM tool I already use, namely yum. 
>>     
>
> ok - but no alternative to that is available on windows.
>
> pythonpkgmgr is not so different to that. And the idea behind it is
> to bring consistancy in package management across the different 
> platforms.
>
> It does have configuration that are specific to python, and will
> continue to do so. Like being able to specify where one wants the
> package installed.
>
>   
>> Putting your software on the Cheeseshop, and making regular 
>> announcements to the Python community (e.g. on comp.lang.python) will 
>> be a good way to publicise the tool, and if does meet a need, people 
>> will use it, and then, if it's good enough and popular enough and 
>> supported, it may be blessed by inclusion in the standard library.
>>     
>
> Thanks very much. Take care.
>
> David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From martin at v.loewis.de  Sun Jul 26 19:31:40 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 26 Jul 2009 19:31:40 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<4A6BFAFB.1030807@v.loewis.de>
	<4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net>
Message-ID: <4A6C92FC.4010307@v.loewis.de>

>> If they read examples, they will see import
>> statements, and then they have to find out how to make those work.
>> Does your tool help with that?
> 
> Yes. It will open the website or homepage to the project/package
> in question.

How does it know the project in question?

> Using pythonpkgmgr, they look on their list of installed packages,
> find the "interbasedb" or whatever module, and can get quick
> access to the pypi page or the package homepage. It's much easier.

How do they find interbasedb in the list of installed packages if
they haven't installed interbasedb yet?

Regards,
Martin

From martin at v.loewis.de  Sun Jul 26 19:33:37 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 26 Jul 2009 19:33:37 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
Message-ID: <4A6C9371.6000002@v.loewis.de>

>> I'm a user, and personally I don't want Yet Another Integrated Package 
>> Management Tool. What I really want is the ability to install Python 
>> packages using the PM tool I already use, namely yum. 
> 
> ok - but no alternative to that is available on windows.

For removal of packages, an alternative *is* available: APR.

> pythonpkgmgr is not so different to that. And the idea behind it is
> to bring consistancy in package management across the different 
> platforms.

At the cost of being inconsistent within a platform.

Regards,
Martin

From joshua at reverberate.org  Sun Jul 26 20:01:51 2009
From: joshua at reverberate.org (Joshua Haberman)
Date: Sun, 26 Jul 2009 18:01:51 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?Py=5FTPFLAGS=5FHEAPTYPE_too_overloaded?=
Message-ID: <loom.20090726T171916-534@post.gmane.org>

I'm writing a C Python extension that needs to generate PyTypeObjects
dynamically.  Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded
in a way that makes it difficult to achieve this goal.

The documentation for Pt_TPFLAGS_HEAPTYPE says:

Py_TPFLAGS_HEAPTYPE

  This bit is set when the type object itself is allocated
  on the heap. In this case, the ob_type field of its
  instances is considered a reference to the type, and the
  type object is INCREF?ed when a new instance is created,
  and DECREF?ed when an instance is destroyed (this does not
  apply to instances of subtypes; only the type referenced
  by the instance?s ob_type gets INCREF?ed or DECREF?ed).

This sounds like exactly what I want.  I want my type object INCREF'd
and DECREF'd by its instances so it doesn't leak or get deleted
prematurely.  If this were all that Py_TPFLAGS_HEAPTYPE did, it would
work great for me.

Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean
"user-defined type" (as opposed to a built-in type).  It controls
numerous subtle behaviors such as:

- whether the type's name is module.type or just type.
- whether you're allowed to set __name__, __module__, or __bases__ on the type.
- whether you're allowed to set __class__ on instances of this type.
- whether the module name comes from the type name or the __module__ attribute.
- whether it will use type->tp_doc as the docstring
- whether its repr() calls it a "class" or a "type".
- whether you can set attributes of the type.
- whether someone is attempting the Carlo Verre hack.

So I'm stuck with an unenviable choice.  I think the lesser of two evils
is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will
happen is that my types will leak.  This is not as bad as having someone
set __class__ on one of my instances, or set attributes on my type, etc.

Ideally the interpreter would have a separate flag like
Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but
still make it possible to have dynamically generated built-in types get
garbage collected appropriately.

At the very least, the documentation I cited above should make it clear
that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets
INCREF'd and DECREF'd.  Based on the list of behaviors I discovered
above, it is almost certainly not correct for a C exension type to be
declared with Py_TPFLAGS_HEAPTYPE.

Josh


From greg.ewing at canterbury.ac.nz  Mon Jul 27 03:05:55 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 27 Jul 2009 13:05:55 +1200
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <loom.20090726T171916-534@post.gmane.org>
References: <loom.20090726T171916-534@post.gmane.org>
Message-ID: <4A6CFD73.8050201@canterbury.ac.nz>

Joshua Haberman wrote:
> This is not as bad as having someone
> set __class__ on one of my instances, or set attributes on my type, etc.

Is there any real need to prevent someone from doing
those things?

Note that even when you are allowed to change the
__class__ of an instance, you're still prevented
from changing it to something that has a different
C layout, so you can't crash the intepreter that
way.

Similarly, built-in methods check that they're
given an object of appropriate type at the C
level.

My suggestion is to just let it be a full heap
type and accept whatever consequences follow.

-- 
Greg

From stephen at xemacs.org  Mon Jul 27 04:18:25 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 27 Jul 2009 11:18:25 +0900
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6C9371.6000002@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
Message-ID: <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>

"Martin v. L?wis" writes:

 > > pythonpkgmgr is not so different to that. And the idea behind it is
 > > to bring consistancy in package management across the different 
 > > platforms.
 > 
 > At the cost of being inconsistent within a platform.

Indeed, and that seems to be one of the really big sticking points in
all efforts to improve Python's package management: the conflict
between user goals ("something that works for me without any effort[1] on
my part") and sysadmin goals ("something that works and plays nicely
with the rest of the system").

pythonpkgmgr seems entirely oblivious to the latter issue, and not
particularly compatible with the way package management works in *nix
OSS distros (including Linux distros, but also *BSD, MacPorts, and
Fink).  I think that will probably kill uptake on those platforms --
could be wrong, but caution is suggested.

BTW, I think Tarek's efforts strike a good balance between these
goals, and the whole PEP 376 process has demonstrated how difficult it
is to serve these two masters at the same time.

Footnotes: 
[1]  I'm a professional economist: in my lexicon, effort minimization
is a positive goal, not laziness.  Zero is just an extreme case.

From tismer at stackless.com  Mon Jul 27 09:00:19 2009
From: tismer at stackless.com (Christian Tismer)
Date: Mon, 27 Jul 2009 00:00:19 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
References: <4A675C10.6060200@stackless.com>	
	<loom.20090722T193713-455@post.gmane.org>	
	<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>	
	<4A67AEC3.6070000@stackless.com>	
	<loom.20090723T090139-199@post.gmane.org>	
	<4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de>
	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
Message-ID: <4A6D5083.3050709@stackless.com>

On 7/25/09 7:11 AM, Neil Hodgson wrote:
> Martin v. L?wis:
>
>> I propose to add another (regular) double into the union.
>
>     Adding a regular double as a second dummy gives the same sizes and
> alignments with Mingw or MSVC as the original definition with MSVC:

Great (checked that, too)

This makes pretty much sense, also given that this waste
happens under Windows, already.

>     In regard to alignment penalties, a simple copy loop for doubles
> runs about 20% slower when misaligned on an my AMD processor. Other
> x86 processors can be much worse. As much as 2 to 3.25 times according
> to

I am still unhappy with this waste of memory, just because the
GC header has to be rounded up, regardlwss wether that is needed
or not.
We should keep Martin's hint in mind, that Python 4 could place
the gc header at the end of structures, instead.

cheers - chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From lists at cheimes.de  Mon Jul 27 09:48:43 2009
From: lists at cheimes.de (Christian Heimes)
Date: Mon, 27 Jul 2009 09:48:43 +0200
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A6D5083.3050709@stackless.com>
References: <4A675C10.6060200@stackless.com>		<loom.20090722T193713-455@post.gmane.org>		<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>		<4A67AEC3.6070000@stackless.com>		<loom.20090723T090139-199@post.gmane.org>		<4A68D2BB.8060607@stackless.com>
	<4A6AE6F0.2060303@v.loewis.de>	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
	<4A6D5083.3050709@stackless.com>
Message-ID: <4A6D5BDB.7010809@cheimes.de>

Christian Tismer wrote:
> We should keep Martin's hint in mind, that Python 4 could place
> the gc header at the end of structures, instead.

Wow, 3.1 just came out and we already have the first PEP for Python 4k? :)

Christian

From joshua at reverberate.org  Mon Jul 27 10:14:21 2009
From: joshua at reverberate.org (Joshua Haberman)
Date: Mon, 27 Jul 2009 08:14:21 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?Py=5FTPFLAGS=5FHEAPTYPE_too_overloaded?=
References: <loom.20090726T171916-534@post.gmane.org>
	<4A6CFD73.8050201@canterbury.ac.nz>
Message-ID: <loom.20090727T075642-76@post.gmane.org>

Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
> Joshua Haberman wrote:
> > This is not as bad as having someone
> > set __class__ on one of my instances, or set attributes on my type, etc.
> 
> Is there any real need to prevent someone from doing
> those things?

My ultimate goal is to make my types as much like regular built-in types
as possible.  Python as a language has chosen to "lock down" built-in
objects, even going so far as to specifically check for the "Carlo Verre
hack."  I defer to those decisions to answer the question "is there any
real need to prevent someone from doing these things?"  If it's
important for the built-in types, why should it be less important for
mine?

I don't want my type to be a second-class citizen just because I happen
to be dynamically allocating it.

If I were writing this extension for a language like Ruby, for which it is
convention that built-in classes are "open," then I wouldn't mind allowing
these things.  I'm just trying to make my extension as idiomatic and "native"
as possible.

Josh


From david.lyon at preisshare.net  Mon Jul 27 11:42:31 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 05:42:31 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net>

On Mon, 27 Jul 2009 11:18:25 +0900, "Stephen J. Turnbull"
<stephen at xemacs.org> wrote:
> [1] on
> my part") and sysadmin goals ("something that works and plays nicely
> with the rest of the system").
> 
> pythonpkgmgr seems entirely oblivious to the latter issue, and not
> particularly compatible with the way package management works in *nix
> OSS distros (including Linux distros, but also *BSD, MacPorts, and
> Fink).  

Well I'm a sysadmin on linux in my day job.

I'm entirely wondering what you mean by that ? I'm oblivious how?

What can a developer mode project, like pythonpkgmgr possibly do
except to make it easier to manage local package space?

A developer shouldn't need to do sudo every time they want
to put or try some python package from pypi. imho that's
wrong and I'm helping to reduce that need.

If the sysadmin puts a package in (/usr/lib/pythonX.Y/site-packages)
via the O/S package manager (ie synaptics/yum/apt) then that
should stick over what a developer is doing.

But the developer should be able to stick their own packages
in their own local space.

Correct me if my assertion is wrong.

David


From david.lyon at preisshare.net  Mon Jul 27 12:20:30 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 06:20:30 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6C831F.5090106@voidspace.org.uk>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C831F.5090106@voidspace.org.uk>
Message-ID: <b361e5686a9208448916a434367b918e@preisshare.net>

On Sun, 26 Jul 2009 17:23:59 +0100, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> It would be great to have  a decent visual package manager for Python. 

Hopefully one day we'll have one - haha

> It needs to be built on top of the work that Tarek is doing with 
> distutils (and be compatible with his Distribute fork of setuptools) and 
> it also needs to be established.

I'm planning to support it - like I do for setuptools and pip

> The good news is that there is time for both... Aren't there other tools 
> that have similar goals? PyPI browser, Yolk etc

The fact is that there's way more glamorous things to work on
than a package manager. Nobody has written a package manager for
python imho simply because there's so many more fun things to
do with python than that...

> It is a shame that it needs to be built on Tk if it is to stand a chance 
> of ever being included in the standard library - but with the right 
> expertise it is *possible* to create decent looking UIs with Tk.

I'm working on a TK version.. yes.. a TK version is better than nothing
and the differences between a TK and a WX version is really only pretty 
minor. To me, wx is ancient... and tk is simply prehistoric. 

mr.new-to-python developer wouldn't really cares too much either I
would suspect. All they want is to be up and running quickly.

My only point is that Windows ain't no embedded system. It's not
short on memory or disk space. If a package manager is 5 megabytes
extra say, with it's libraries.. what's the extra download time on
that ? compared to three days+ stuffing around trying to find out
how to install packages for a new user.

If the source needs to go in too.. that's a few kilobytes.

That's like 57 extra bytes to an old-timer... I know.. it's
such a sacrifice.. and we're all working on 300 baud... :-)

David



From cournape at gmail.com  Mon Jul 27 12:29:14 2009
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 27 Jul 2009 19:29:14 +0900
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <b361e5686a9208448916a434367b918e@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C831F.5090106@voidspace.org.uk>
	<b361e5686a9208448916a434367b918e@preisshare.net>
Message-ID: <5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com>

On Mon, Jul 27, 2009 at 7:20 PM, David Lyon<david.lyon at preisshare.net> wrote:

> My only point is that Windows ain't no embedded system. It's not
> short on memory or disk space. If a package manager is 5 megabytes
> extra say, with it's libraries.. what's the extra download time on
> that ? compared to three days+ stuffing around trying to find out
> how to install packages for a new user.

The problem is not so much  the size by itself that more code means
more maintenance burden for python developers. Including new code
means it has to work everywhere where python works currently, and that
other people can understand/support the related code. Adding code to a
project is far from free from python maintainers POV.

cheers,

David

From david.lyon at preisshare.net  Mon Jul 27 12:59:18 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 06:59:18 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C831F.5090106@voidspace.org.uk>
	<b361e5686a9208448916a434367b918e@preisshare.net>
	<5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com>
Message-ID: <d84db37f79af4bdc0e42792b5f9025a4@preisshare.net>

On Mon, 27 Jul 2009 19:29:14 +0900, David Cournapeau <cournape at gmail.com>
wrote:
>> My only point is that Windows ain't no embedded system. It's not
>> short on memory or disk space. If a package manager is 5 megabytes
>> extra say, with it's libraries.. what's the extra download time on
>> that ? compared to three days+ stuffing around trying to find out
>> how to install packages for a new user.
> 
> The problem is not so much  the size by itself that more code means
> more maintenance burden for python developers. Including new code
> means it has to work everywhere where python works currently, and that
> other people can understand/support the related code. Adding code to a
> project is far from free from python maintainers POV.

Well I concede that you might have a point.

And I know there have been some issues in the past with package
management in python and people being able to understand other
peoples code.

In this specific case, I don't think you'll find the same problems.

pythonpkgmgr is not an entirely self inclusive project. It leverages
on setuptools and pip. I want to extend it to enstall. If there is
any preference towards setuptools - it's only by test-time availability.

In windows, it's built with py2exe. There are some supporting
.dll files that need to go with it from wxpython. They're both really
mainstream python(windows) toolkits.

Everything in pythonpkgmgr is transperant - and was designed to
be. Otherwise there's no point.

If any developers want to join the project with 'commit' privaleges
on the source, I'll make sure it will happen.

If not.. I'm very happy just continuing on with my own merry way...

:-)

David




From stephen at xemacs.org  Mon Jul 27 13:12:54 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 27 Jul 2009 20:12:54 +0900
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net>
Message-ID: <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp>

David Lyon writes:
 > On Mon, 27 Jul 2009 11:18:25 +0900, "Stephen J. Turnbull"
 > <stephen at xemacs.org> wrote:
 > > [1] on
 > > my part") and sysadmin goals ("something that works and plays nicely
 > > with the rest of the system").
 > > 
 > > pythonpkgmgr seems entirely oblivious to the latter issue, and not
 > > particularly compatible with the way package management works in *nix
 > > OSS distros (including Linux distros, but also *BSD, MacPorts, and
 > > Fink).  
 > 
 > Well I'm a sysadmin on linux in my day job.
 > 
 > I'm entirely wondering what you mean by that ? I'm oblivious how?

Not you; pythonpkgmgr.  You've said nothing about how pythonpkgmgr is
supposed to deal with multiple installed versions of Python or
individual modules, or how it helps if you need a custom version of a
module, etc., or how it can help the developer's "local" modules
cooperate/not interfere with system versions, etc.

 > What can a developer mode project, like pythonpkgmgr possibly do
 > except to make it easier to manage local package space?

Pave the road to DLL Hell with good intentions, of course.

 > A developer shouldn't need to do sudo every time they want
 > to put or try some python package from pypi.

I don't remember *ever* sudo-ing to try a package from pypi.

 > If the sysadmin puts a package in (/usr/lib/pythonX.Y/site-packages)
 > via the O/S package manager (ie synaptics/yum/apt) then that
 > should stick over what a developer is doing.

But I disagree about that, too.  The last thing I want is to have my
in-development environment hosed by system upgrades or installations.
Eventually I want to deal with that for deployment, but in the early
stages one can make faster progress on demos or initial target
platforms by sticking to particular versions of prerequisite libraries.

 > But the developer should be able to stick their own packages
 > in their own local space.
 > 
 > Correct me if my assertion is wrong.

"Should", yes.  But your assertion is incomplete IME: the developer
*is* able to install packages in their own local space *without*
sudo-ing.

I just don't understand what problem you're claiming to solve.  The
problem of managing local package space is in some sense solved by
virtualenv, but you haven't mentioned that at all.  Is pythonpkgmgr an
alternative?  A complement?  to virtualenv or zc.buildout?


From eric.pruitt at gmail.com  Mon Jul 27 18:01:30 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 27 Jul 2009 11:01:30 -0500
Subject: [Python-Dev] Implementing File Modes
Message-ID: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>

Hello,

Since there was a bit of confusion last time, I'll start by saying I am
working on the subprocess.Popen module for Google Summer of Code. One of the
features I am implementing is a class so that a running process can stand in
in place of a file. For examples, instead of open( "filelist", mode = 'r')
one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide
if I should fully implement the mode argument. Right now, it essentially
ignores everything but a 'U' indicated universal newlines in the mode
argument. Should I leave that as is or make it so that things like "r+",
"w", "a" are handled the way they would be for an actual file?

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090727/1dfe718a/attachment.htm>

From tismer at stackless.com  Mon Jul 27 18:11:45 2009
From: tismer at stackless.com (Christian Tismer)
Date: Mon, 27 Jul 2009 09:11:45 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A6D5BDB.7010809@cheimes.de>
References: <4A675C10.6060200@stackless.com>		<loom.20090722T193713-455@post.gmane.org>		<5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com>		<4A67AEC3.6070000@stackless.com>		<loom.20090723T090139-199@post.gmane.org>		<4A68D2BB.8060607@stackless.com>
	<4A6AE6F0.2060303@v.loewis.de>	<50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com>
	<4A6D5083.3050709@stackless.com> <4A6D5BDB.7010809@cheimes.de>
Message-ID: <4A6DD1C1.1020800@stackless.com>

On 7/27/09 12:48 AM, Christian Heimes wrote:
> Christian Tismer wrote:
>> We should keep Martin's hint in mind, that Python 4 could place
>> the gc header at the end of structures, instead.
>
> Wow, 3.1 just came out and we already have the first PEP for Python 4k? :)
>
> Christian

Maybe it's even possible right now, with a couple of tweaks
that allow to explicitly say where the header is, for a specific
extension type. Might cost a few cycles, tho.

Christian

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From tismer at stackless.com  Mon Jul 27 18:13:51 2009
From: tismer at stackless.com (Christian Tismer)
Date: Mon, 27 Jul 2009 09:13:51 -0700
Subject: [Python-Dev] mingw32 and gc-header weirdness
In-Reply-To: <4A69A606.1040506@roumenpetrov.info>
References: <4A675C10.6060200@stackless.com>
	<4A69A606.1040506@roumenpetrov.info>
Message-ID: <4A6DD23F.1030006@stackless.com>

On 7/24/09 5:16 AM, Roumen Petrov wrote:
> Christian Tismer wrote:
...
> Did the crash disappear is you add "__attribute__((aligned(8)))" after
> variable dummy ?

Did not try. But the proposed addition of a double does it,
see the dev list.

cheers - chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

From p.f.moore at gmail.com  Mon Jul 27 18:46:15 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 27 Jul 2009 17:46:15 +0100
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
Message-ID: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>

2009/7/27 Eric Pruitt <eric.pruitt at gmail.com>:
> Hello,
>
> Since there was a bit of confusion last time, I'll start by saying I am
> working on the subprocess.Popen module for Google Summer of Code. One of the
> features I am implementing is a class so that a running process can stand in
> in place of a file. For examples, instead of open( "filelist", mode = 'r')
> one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide
> if I should fully implement the mode argument. Right now, it essentially
> ignores everything but a 'U' indicated universal newlines in the mode
> argument. Should I leave that as is or make it so that things like "r+",
> "w", "a" are handled the way they would be for an actual file?

I would expect "r" to produce a pipe that reads from stdout of the
subprocess, and "w" to produce a pipe that writes to stdin of the
subprocess. "a" would be the same as "w", and arguably "r+" would be a
bidirectional pipe - read from the subprocess stdout and write to its
stdin.

I'd be OK with "r+" not being implemented (if it's too hard to avoid
deadlocks) but "r" and "w" should be present.

Paul.

From python at mrabarnett.plus.com  Mon Jul 27 19:21:30 2009
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 27 Jul 2009 18:21:30 +0100
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
Message-ID: <4A6DE21A.4040803@mrabarnett.plus.com>

Paul Moore wrote:
> 2009/7/27 Eric Pruitt <eric.pruitt at gmail.com>:
>> Hello,
>>
>> Since there was a bit of confusion last time, I'll start by saying I am
>> working on the subprocess.Popen module for Google Summer of Code. One of the
>> features I am implementing is a class so that a running process can stand in
>> in place of a file. For examples, instead of open( "filelist", mode = 'r')
>> one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide
>> if I should fully implement the mode argument. Right now, it essentially
>> ignores everything but a 'U' indicated universal newlines in the mode
>> argument. Should I leave that as is or make it so that things like "r+",
>> "w", "a" are handled the way they would be for an actual file?
> 
> I would expect "r" to produce a pipe that reads from stdout of the
> subprocess, and "w" to produce a pipe that writes to stdin of the
> subprocess. "a" would be the same as "w", and arguably "r+" would be a
> bidirectional pipe - read from the subprocess stdout and write to its
> stdin.
> 
> I'd be OK with "r+" not being implemented (if it's too hard to avoid
> deadlocks) but "r" and "w" should be present.
> 
What about stderr? You could add "e" if you want to read from it.

From eric.pruitt at gmail.com  Mon Jul 27 20:06:45 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Mon, 27 Jul 2009 12:06:45 -0600
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> 
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
Message-ID: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>

I am implementing the file wrapper using changes to subprocess.Popen that
also make it asynchronous and non-blocking so implementing "r+" should be
trivial to do. How about handling stderr? I have the following ideas: leave
out support for reading from stderr, make it so that there is an optional
additional argument like "outputstderr = False", create another function
that toggles / sets whether stderr or stdout is returned or mix the two
outputs.

Thanks for the input,
Eric

On Mon, Jul 27, 2009 at 10:46, Paul Moore <p.f.moore at gmail.com> wrote:

> 2009/7/27 Eric Pruitt <eric.pruitt at gmail.com>:
> > Hello,
> >
> > Since there was a bit of confusion last time, I'll start by saying I am
> > working on the subprocess.Popen module for Google Summer of Code. One of
> the
> > features I am implementing is a class so that a running process can stand
> in
> > in place of a file. For examples, instead of open( "filelist", mode =
> 'r')
> > one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to
> decide
> > if I should fully implement the mode argument. Right now, it essentially
> > ignores everything but a 'U' indicated universal newlines in the mode
> > argument. Should I leave that as is or make it so that things like "r+",
> > "w", "a" are handled the way they would be for an actual file?
>
> I would expect "r" to produce a pipe that reads from stdout of the
> subprocess, and "w" to produce a pipe that writes to stdin of the
> subprocess. "a" would be the same as "w", and arguably "r+" would be a
> bidirectional pipe - read from the subprocess stdout and write to its
> stdin.
>
> I'd be OK with "r+" not being implemented (if it's too hard to avoid
> deadlocks) but "r" and "w" should be present.
>
> Paul.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090727/3ba61afc/attachment.htm>

From p.f.moore at gmail.com  Mon Jul 27 21:04:08 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 27 Jul 2009 20:04:08 +0100
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
Message-ID: <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com>

2009/7/27 Eric Pruitt <eric.pruitt at gmail.com>:
> I am implementing the file wrapper using changes to subprocess.Popen that
> also make it asynchronous and non-blocking so implementing "r+" should be
> trivial to do. How about handling stderr? I have the following ideas: leave
> out support for reading from stderr, make it so that there is an optional
> additional argument like "outputstderr = False", create another function
> that toggles / sets whether stderr or stdout is returned or mix the two
> outputs.

I like MRAB's idea of using a (non-standard) "e" flag to include
stderr. So "r" reads from stdout, "re" reads from stdout+stderr.

Anything more complicated probably should just use "raw" Popen
objects. Don't overcomplicate the interface.

Paul.

From steve at pearwood.info  Tue Jul 28 01:26:23 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 28 Jul 2009 09:26:23 +1000
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <4A6DE21A.4040803@mrabarnett.plus.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
	<4A6DE21A.4040803@mrabarnett.plus.com>
Message-ID: <200907280926.24081.steve@pearwood.info>

On Tue, 28 Jul 2009 03:21:30 am MRAB wrote:

> What about stderr? You could add "e" if you want to read from it.

"Read from stderr" is just a read. "Write to stderr" is just a write. 
The difference between reading stdout and stderr is not that you have 
different modes, but that you are reading from different files.


-- 
Steven D'Aprano

From python at mrabarnett.plus.com  Tue Jul 28 01:41:53 2009
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 28 Jul 2009 00:41:53 +0100
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <200907280926.24081.steve@pearwood.info>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>	<4A6DE21A.4040803@mrabarnett.plus.com>
	<200907280926.24081.steve@pearwood.info>
Message-ID: <4A6E3B41.60701@mrabarnett.plus.com>

Steven D'Aprano wrote:
> On Tue, 28 Jul 2009 03:21:30 am MRAB wrote:
> 
>> What about stderr? You could add "e" if you want to read from it.
> 
> "Read from stderr" is just a read. "Write to stderr" is just a write. 
> The difference between reading stdout and stderr is not that you have 
> different modes, but that you are reading from different files.
> 
By the same argument, aren't stdin and stdout also different files?

From greg.ewing at canterbury.ac.nz  Tue Jul 28 01:52:43 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Jul 2009 11:52:43 +1200
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <loom.20090727T075642-76@post.gmane.org>
References: <loom.20090726T171916-534@post.gmane.org>
	<4A6CFD73.8050201@canterbury.ac.nz>
	<loom.20090727T075642-76@post.gmane.org>
Message-ID: <4A6E3DCB.7030601@canterbury.ac.nz>

Joshua Haberman wrote:
> Python as a language has chosen to "lock down" built-in
> objects...  If it's
> important for the built-in types, why should it be less important for
> mine?

I'm not really sure why so much trouble is taken to lock
down builtin types -- it seems to go against Python's
general consenting-adults philosophy.

I suppose it's felt that you should be able to rely on
builtin types not changing their behaviour. This is probably
more important for the core types than those in extension
modules. Many of the standard library classes are written in
Python, so this protection doesn't extend to them.

> I don't want my type to be a second-class citizen just because I happen
> to be dynamically allocating it.

I don't think anyone will regard your types as second-class
because they allow you to do *more* with them.

The only real concern would be if it were somehow possible
to crash the interpreter by modifying the type dict.
I don't see how that could happen -- but maybe someone
else on python-dev knows more about this?

-- 
Greg

From steve at pearwood.info  Tue Jul 28 02:01:25 2009
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 28 Jul 2009 10:01:25 +1000
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
Message-ID: <200907281001.26339.steve@pearwood.info>

On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote:
> I am implementing the file wrapper using changes to subprocess.Popen
> that also make it asynchronous and non-blocking so implementing "r+"
> should be trivial to do. How about handling stderr? I have the
> following ideas: leave out support for reading from stderr, make it
> so that there is an optional additional argument like "outputstderr =
> False", create another function that toggles / sets whether stderr or
> stdout is returned or mix the two outputs.

Leaving it out is always an option.

As I see it, fundamentally you can either read from stdout and sterr as 
two different streams, or you can interleave (mix) them. To me, that 
suggests three functions:

ProcessIOWrapper()  # read from stdout (or write to stdin etc.)
ProcessIOWrapperStdErr()  # read/write from stderr
ProcessIOWrapper2()  # read from mixed stdout and stderr

I don't like a function to toggle between one and the other: that smacks 
of relying on a global setting in a bad way. I suppose you could add an 
optional argument to ProcessIOWrapper() to select between stdout, 
stderr, or both together.



-- 
Steven D'Aprano

From glyph at twistedmatrix.com  Tue Jul 28 02:32:51 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Mon, 27 Jul 2009 20:32:51 -0400
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
	<79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com>
Message-ID: <d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com>

On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> I like MRAB's idea of using a (non-standard) "e" flag to include
> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
>
> Anything more complicated probably should just use "raw" Popen
> objects. Don't overcomplicate the interface.
>

In my opinion, mangling stderr and stdout together is already an
overcomplication.  It shouldn't be implemented.

It *seems* like a good idea, until you realize that subtle changes to your
OS, environment, or buffering behavior may result in arbitrary, unparseable
output.

For example, let's say you've got a program whose output is a list of lines,
each one containing a number.  Sometimes it tries to import gtk, and fails
to open its display.

That's fine, and you can still deal with it, as long as the interleaved
output looks like this:

100
200
Gtk-WARNING **: cannot open display:
300
400

but of course the output *might* (although unlikely with such small chunks
of output) end up looking like this, instead:

100
2Gtk-WAR0NING0 **:
can30not 0open display:

400

this is the sort of thing which is much more likely to happen once you start
dealing with large volumes of data, where there are more page-boundaries for
your buffers to get confused around, and you are playing with buffering
options to improve performance.  In other words, it's something that fails
only at scale or under load, and is therefore extremely difficult to debug.

This option *might* be okay if it were allowed only on subprocesses opened
in a *text* mode, and if the buffering logic involved forced stderr and
stdout to be line-delimited, and interleave only lines, rather than
arbitrary chunks of bytes.  Of course then if you use this flag with a
program that outputs binary data with no newlines it will buffer forever and
crash your program with a MemoryError, but at least that's easy to debug
when it happens.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090727/34b5c7ff/attachment.htm>

From david.lyon at preisshare.net  Tue Jul 28 02:32:27 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 20:32:27 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net>
	<87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <68eb1540f5da00887220865daef9daee@preisshare.net>

On Mon, 27 Jul 2009 20:12:54 +0900, "Stephen J. Turnbull"
<stephen at xemacs.org> wrote:
> Not you; pythonpkgmgr.  You've said nothing about how pythonpkgmgr is
> supposed to deal with multiple installed versions of Python

Under windows it can deal with multiple versions of python. You just
go to options and select which version of python you wish to manage.

It's as simple as that.

> or how it helps if you need a custom version of a
> module, etc., or how it can help the developer's "local" modules
> cooperate/not interfere with system versions, etc.

It manages local developer modules for python 2.6+.

As for checks, it doesn't have so many at present. But it makes
it a lot easier to see what you have installed. So if something
needs to be removed and reinstalled - it can do that.

> Pave the road to DLL Hell with good intentions, of course.

Well - there's plenty of that

> 
>  > A developer shouldn't need to do sudo every time they want
>  > to put or try some python package from pypi.
> 
> I don't remember *ever* sudo-ing to try a package from pypi.

ok. But if you were using an O/S level package manager you
would have to enter a su password to start the tool. Because
that is needed to load packages into /usr/lib/pythonX.y

> I just don't understand what problem you're claiming to solve.  

Making it simpler for new users to find and install packages
from pypi. And manage the existing packages that are installed
on their machine.

> The
> problem of managing local package space is in some sense solved by
> virtualenv, but you haven't mentioned that at all.  

I never heard of it till a few months ago. To my knowledge it
doesn't offer a gui. I'm told it's good for certain things and
I believe it.

My problem was installing packages on windows boxes, not in
the development.

> Is pythonpkgmgr an
> alternative?  A complement?  to virtualenv or zc.buildout?

pythonpkgmgr is aimed at featherweight users. It could be
highly complementory to both virtualenv or zc.buildout at
some stage if it were customised to work with those.

For example, it could work with virtualenv by having a
a drop down list of the different 'environments'. I don't
know how to do it right now - but I'm hoping I can figure
it out sometime.

With zc.buildout, there's no reason why it couldn't generate
a package list in that format, or do an import from a
manifest. pythonpkgmgr already can generate a list of
local packages as a manifest - but not in the zc.buildout
format. I'm sure in the future - I'll figure that one out
as well.

David




From david.lyon at preisshare.net  Tue Jul 28 02:44:58 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 20:44:58 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6C92FC.4010307@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<87bpnb7ufv.fsf@benfinney.id.au>	<51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com>	<871vo561d7.fsf@benfinney.id.au>	<245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<4A6BFAFB.1030807@v.loewis.de>
	<4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net>
	<4A6C92FC.4010307@v.loewis.de>
Message-ID: <acf2f0c790ac5ffb3d941d418d06de7b@preisshare.net>

On Sun, 26 Jul 2009 19:31:40 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
>>> If they read examples, they will see import
>>> statements, and then they have to find out how to make those work.
>>> Does your tool help with that?
>> 
>> Yes. It will open the website or homepage to the project/package
>> in question.
> 
> How does it know the project in question?

Because it is either in the pypi search results or list of installed
packages.

> How do they find interbasedb in the list of installed packages if
> they haven't installed interbasedb yet?

They'd already installed it. But perphaps through a .exe installer.


David




From david.lyon at preisshare.net  Tue Jul 28 02:50:57 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 27 Jul 2009 20:50:57 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6C9371.6000002@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
Message-ID: <1b9d4c7557e19faed93300f157d07b79@preisshare.net>

On Sun, 26 Jul 2009 19:33:37 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
>> pythonpkgmgr is not so different to that. And the idea behind it is
>> to bring consistancy in package management across the different 
>> platforms.
> 
> At the cost of being inconsistent within a platform.

It has the most generic of user interfaces.

On Windows, a command line interface is the most out-of-place thing
anybody could ask for.

19/20 kids these days wouldn't even know what a command line interface
is until you show them.

The first thing that they would look for is a GUI tool in the Python X.Y
programs menu...

So I respectfully say that there couldn't be anything less true than
the assertion that a GUI package manager is inconsistent within a
modern GUI desktop environment.

David


From stephen at xemacs.org  Tue Jul 28 04:26:44 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 28 Jul 2009 11:26:44 +0900
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <68eb1540f5da00887220865daef9daee@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<4A6AD375.8000805@v.loewis.de>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net>
	<87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp>
	<68eb1540f5da00887220865daef9daee@preisshare.net>
Message-ID: <87iqhdbl0b.fsf@uwakimon.sk.tsukuba.ac.jp>

David Lyon writes:

 > It manages local developer modules for python 2.6+.

 > pythonpkgmgr is aimed at featherweight users.

You were talking about "developers", but now they're "featherweight
users"?  I'm sorry, but the more you post, the less I like the idea of
including it with Python.  Please do develop it, do post it to pypi,
and do discuss it on python-list.  I'm sure there is an audience for
it, not only for Windows users but for the more GUI-oriented POSIX-
based platforms as well.

But I really don't think that python-dev is the appropriate place any
more.

From aahz at pythoncraft.com  Tue Jul 28 04:48:07 2009
From: aahz at pythoncraft.com (Aahz)
Date: Mon, 27 Jul 2009 19:48:07 -0700
Subject: [Python-Dev] Going nomail
Message-ID: <20090728024807.GA28303@panix.com>

I just started a new job today, and I've got a bunch of other stuff going
on in my life, so I'm setting python-dev and python-ideas to nomail for
a while.  Please feel free to ping me directly if you want.

I'll be back.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer

From martin at v.loewis.de  Tue Jul 28 07:12:25 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 28 Jul 2009 07:12:25 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <1b9d4c7557e19faed93300f157d07b79@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
Message-ID: <4A6E88B9.5010504@v.loewis.de>

>>> pythonpkgmgr is not so different to that. And the idea behind it is
>>> to bring consistancy in package management across the different 
>>> platforms.
>> At the cost of being inconsistent within a platform.
> 
> It has the most generic of user interfaces.
> 
[...]
> 
> So I respectfully say that there couldn't be anything less true than
> the assertion that a GUI package manager is inconsistent within a
> modern GUI desktop environment.

That's not my concern - I'm not worried about how it looks, but what
it does. If it "manages packages", I would expect it to manage them
the same way as the native package manager does, but alas, it doesn't
(IIUC). So there are now two incompatible ways to install a package:
either with the native manager, or with pythonpkgmgr. If you install
them one way, and try to remove them the other way, you lose.

Regards,
Martin

From david.lyon at preisshare.net  Tue Jul 28 07:20:59 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 28 Jul 2009 01:20:59 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6E88B9.5010504@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
Message-ID: <14cf53f0cf73387c177f045fd21151b7@preisshare.net>

On Tue, 28 Jul 2009 07:12:25 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
> So there are now two incompatible ways to install a package:
> either with the native manager, or with pythonpkgmgr. If you install
> them one way, and try to remove them the other way, you lose.

pythonpkgmgr is only a thin wrapper for easy_install/pip.

If there is a problem, then it is already a pre-existing problem
that is not of my doing.

It's certainly true that if a python package is installed with a
windows installer or as a .deb/.rpm, then that package should be
uninstalled using the appropriate and corresponding tool.

The same is true if the package was installed with either pip
or easy_install. 

All I can do is put it in my tracker as a bug and deal with it
later when I have time.

Good point

David


From martin at v.loewis.de  Tue Jul 28 07:55:11 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 28 Jul 2009 07:55:11 +0200
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <14cf53f0cf73387c177f045fd21151b7@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
Message-ID: <4A6E92BF.7080901@v.loewis.de>

>> So there are now two incompatible ways to install a package:
>> either with the native manager, or with pythonpkgmgr. If you install
>> them one way, and try to remove them the other way, you lose.
> 
> pythonpkgmgr is only a thin wrapper for easy_install/pip.
> 
> If there is a problem, then it is already a pre-existing problem
> that is not of my doing.

Yes, eggs have the same problem. That's one of the reasons they
don't get integrated into Python.

Regards,
Martin

From david.lyon at preisshare.net  Tue Jul 28 08:57:22 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 28 Jul 2009 02:57:22 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6E92BF.7080901@v.loewis.de>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<4A6AD375.8000805@v.loewis.de>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
Message-ID: <3fcf944249977ccd966d62a73ac0db5d@preisshare.net>

On Tue, 28 Jul 2009 07:55:11 +0200, "Martin v. L?wis" <martin at v.loewis.de>
wrote:
> Yes, eggs have the same problem. That's one of the reasons they
> don't get integrated into Python.

Yes but egg_info is included in python...

and the egg is not....

Hence, what goes in and what doesn't isn't always that rational. I'm
just accepting that for the moment.

David



From greg at krypto.org  Tue Jul 28 09:53:48 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 28 Jul 2009 00:53:48 -0700
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> 
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> 
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> 
	<79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> 
	<d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com>
Message-ID: <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com>

On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz<glyph at twistedmatrix.com> wrote:
> On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> I like MRAB's idea of using a (non-standard) "e" flag to include
>> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
>>
>> Anything more complicated probably should just use "raw" Popen
>> objects. Don't overcomplicate the interface.
>
> In my opinion, mangling stderr and stdout together is already an
> overcomplication.? It shouldn't be implemented.
>
> It seems like a good idea, until you realize that subtle changes to your OS,
> environment, or buffering behavior may result in arbitrary, unparseable
> output.

Agreed.  Leave stderr support out of this.  People who need stderr
should use the full subprocess.Popen interface.  Quick hack unixy
types will just run their process using a shell (which already seems
to be the default based on the "ls -l" example) and add 2>&1.  This
functionality is basically the equivalent of adding the | symbol on
either or both ends of a filename given to open() in perl.  (but
without being so gross).

I do wonder why you're trying to make it behave exactly like open()
including the mode= syntax.

Why not just define several names based on the behavior?

 ProcessReadWrapper()
 ProcessWriteWrapper()
 ProcessReadWriteWrapper()

-gps

>
> For example, let's say you've got a program whose output is a list of lines,
> each one containing a number.? Sometimes it tries to import gtk, and fails
> to open its display.
>
> That's fine, and you can still deal with it, as long as the interleaved
> output looks like this:
>
> 100
> 200
> Gtk-WARNING **: cannot open display:
> 300
> 400
>
> but of course the output might (although unlikely with such small chunks of
> output) end up looking like this, instead:
>
> 100
> 2Gtk-WAR0NING0 **:
> can30not 0open display:
>
> 400
>
> this is the sort of thing which is much more likely to happen once you start
> dealing with large volumes of data, where there are more page-boundaries for
> your buffers to get confused around, and you are playing with buffering
> options to improve performance.? In other words, it's something that fails
> only at scale or under load, and is therefore extremely difficult to debug.
>
> This option might be okay if it were allowed only on subprocesses opened in
> a text mode, and if the buffering logic involved forced stderr and stdout to
> be line-delimited, and interleave only lines, rather than arbitrary chunks
> of bytes.? Of course then if you use this flag with a program that outputs
> binary data with no newlines it will buffer forever and crash your program
> with a MemoryError, but at least that's easy to debug when it happens.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>

From p.f.moore at gmail.com  Tue Jul 28 12:50:00 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 28 Jul 2009 11:50:00 +0100
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
Message-ID: <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>

2009/7/28 David Lyon <david.lyon at preisshare.net>:
> On Tue, 28 Jul 2009 07:55:11 +0200, "Martin v. L?wis" <martin at v.loewis.de>
> wrote:
>> Yes, eggs have the same problem. That's one of the reasons they
>> don't get integrated into Python.
>
> Yes but egg_info is included in python...
>
> and the egg is not....
>
> Hence, what goes in and what doesn't isn't always that rational. I'm
> just accepting that for the moment.

egg_info data is in to allow "standard" (setup.py install and hence OS
package manager managed) packages to provide metadata in a
discoverable way. Using a format that is (reasonably) compatible with
setuptools is simply a matter of co-operating with existing de facto
standards.

Eggs themselves (as a distribution format) are just zip files with a
funny extension, and as such are supported by Python.

The infrastructure and philosophy around eggs (easy_install, the
various .pth file manipulations, multi-version installs, etc) are
supported by Python (in the trivial sense that they are possible) but
are not "blessed" by standard library inclusion, precisely because of
the issues being mentioned here.

Your package manager has the same issues as the egg infrastructure
(lack of integration with system package managers being the biggest
one) and so is not suitable for the standard library in precisely the
same way.

Paul.

From ncoghlan at gmail.com  Tue Jul 28 13:26:58 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 28 Jul 2009 21:26:58 +1000
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <4A6E3DCB.7030601@canterbury.ac.nz>
References: <loom.20090726T171916-534@post.gmane.org>	<4A6CFD73.8050201@canterbury.ac.nz>	<loom.20090727T075642-76@post.gmane.org>
	<4A6E3DCB.7030601@canterbury.ac.nz>
Message-ID: <4A6EE082.3010509@gmail.com>

Greg Ewing wrote:
> The only real concern would be if it were somehow possible to crash
> the interpreter by modifying the type dict. I don't see how that
> could happen -- but maybe someone else on python-dev knows more about
> this?

I believe a major part of the issue is that attempting to answer the
question "Will allowing mutating operation *X* on
object/type/float/int/etc create any interpreter stability or security
problems?" is awfully close to trying to prove a negative.

Certainly, at least some of the "generic" operations involving types
include additional sanity checks that are bypassed for the builtin types.

One specific example I can think of is that object.__hash__ is special
cased in a few places due to the way its definition interacts with the
definition of comparison operations. Allowing changes to the contents of
object's tp_hash slot could lead to much weirdness when it came to
__hash__ inheritance.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Tue Jul 28 13:29:54 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 28 Jul 2009 21:29:54 +1000
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <4A6EE082.3010509@gmail.com>
References: <loom.20090726T171916-534@post.gmane.org>	<4A6CFD73.8050201@canterbury.ac.nz>	<loom.20090727T075642-76@post.gmane.org>
	<4A6E3DCB.7030601@canterbury.ac.nz> <4A6EE082.3010509@gmail.com>
Message-ID: <4A6EE132.8020507@gmail.com>

Nick Coghlan wrote:
> One specific example I can think of is that object.__hash__ is special
> cased in a few places due to the way its definition interacts with the
> definition of comparison operations. Allowing changes to the contents of
> object's tp_hash slot could lead to much weirdness when it came to
> __hash__ inheritance.

Just thought of a much better example as I clicked send: the basic
numeric types (especially int) are locked down because they are
special-cased all over the place (including in the main interpreter eval
loop) in order to speed up simple arithmetic.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From rocky at gnu.org  Tue Jul 28 14:10:45 2009
From: rocky at gnu.org (Rocky Bernstein)
Date: Tue, 28 Jul 2009 08:10:45 -0400
Subject: [Python-Dev] License for code extracted from a Python library?
Message-ID: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com>

Hi -

I'm not sure if this is the right place to ask this, but I don't know of a
more appropriate place.

Sometime around 2007 I first extracted a function that was in Python's Cmd
class and used that. Since then, I have modified and generalized it a bit
more and turned it into a Python egg called
columnize<http://pypi.python.org/pypi/columnize>.
I also ported it to Ruby and made Ruby package or gem. A couple of
people/companies have asked about redistributing this and have asked for a
more MIT-like and less GPL-like license. I guess they will bundle this with
their proprietary code.

For the Python package, I used PSL Version 2, but is this legitimate?
code.google.com doesn't seem to recognize in their list of licenses so that
suggests what I did might not be the right thing. (So instead,
code.google.com I selected the MIT License.)

Can a MIT License be used for code extracted from Python's standard library?
Other comments or suggestions?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090728/c3a38809/attachment.htm>

From ncoghlan at gmail.com  Tue Jul 28 14:43:55 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 28 Jul 2009 22:43:55 +1000
Subject: [Python-Dev] License for code extracted from a Python library?
In-Reply-To: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com>
References: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com>
Message-ID: <4A6EF28B.4050304@gmail.com>

Rocky Bernstein wrote:
> Can a MIT License be used for code extracted from Python's standard
> library? Other comments or suggestions?

The extracted code itself would stay under the PSF license since you
don't have the rights to change the license on that. However, as the PSL
itself is a very permissive license, you can bundle it with code under
whatever other license you or users of your package choose to use (even
proprietary ones).

That said, while I know a fair bit about software licensing I'm not a
lawyer, and even if I was one, I still wouldn't be *your* lawyer, so
take what I have to say with a fairly large grain of salt! :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From david.lyon at preisshare.net  Tue Jul 28 16:14:52 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 28 Jul 2009 10:14:52 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>
	<200907261805.08277.steve@pearwood.info>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
Message-ID: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>

On Tue, 28 Jul 2009 11:50:00 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> egg_info data is in to allow "standard" (setup.py install and hence OS
> package manager managed) packages to provide metadata in a
> discoverable way. Using a format that is (reasonably) compatible with
> setuptools is simply a matter of co-operating with existing de facto
> standards.
> 
> Eggs themselves (as a distribution format) are just zip files with a
> funny extension, and as such are supported by Python.

ok - I get it.....

It's all one big monty python comedy thing where things are quirky
and inconsistent - there's little continuety from one scene to the
next but you have to stop sometimes and have a laugh... because
to word quote from you - "funny"

Your whole email whilst perphaps technically correct is terribly
difficult for a software engineering person to follow.

Monty python was only meant to be a funny film for entertainment,
not a philosophy for building software from....

Why we can have the egg_info without allowing the egg is kind of
like having a chicken and egg story without allowing the chicken..
or is it the egg.. oh I lost track...

I think I need to watch the movie again to understand what's
happening here...

Let me go away confused... don't ask me any more questions or
elaborate with more answers. Thanks. :-)

David






From p.f.moore at gmail.com  Tue Jul 28 20:23:01 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 28 Jul 2009 19:23:01 +0100
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>
	<4A6C9371.6000002@v.loewis.de>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
Message-ID: <79990c6b0907281123j586f698ek896e7512c53a6845@mail.gmail.com>

2009/7/28 David Lyon <david.lyon at preisshare.net>:
> ok - I get it.....
[...]
> Your whole email whilst perphaps technically correct is terribly
> difficult for a software engineering person to follow.

OK, I'm sorry if my attempts to help you didn't do so.

> Let me go away confused... don't ask me any more questions or
> elaborate with more answers. Thanks. :-)

Don't worry, I'll do my best not to from now on :-)

Paul.

From ncoghlan at gmail.com  Wed Jul 29 02:56:20 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 29 Jul 2009 10:56:20 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>	<4A6C9371.6000002@v.loewis.de>	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>	<4A6E88B9.5010504@v.loewis.de>	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>	<4A6E92BF.7080901@v.loewis.de>	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
Message-ID: <4A6F9E34.5010708@gmail.com>

David Lyon wrote:
> Your whole email whilst perphaps technically correct is terribly
> difficult for a software engineering person to follow.

It made perfect sense to me.

The words "eggs" brings with it a whole lot more baggage than just the
sum of the technical parts in the language core that support them
(primarily distutils and zipimport). I find it unfortunate that the name
for the distutils metadata format contains the word "egg" because it
implies much greater consensus around the philosophy behind eggs than
actually exists.

A lot of the baggage associated with the "eggs" concept is related to
the inherent conflict between different approaches to dependency management:
1. Use the system package management system for everything (preferred by
many, perhaps even most, *nix sysadmins, but not an option on Windows)
2. Create a Python specific package management system independent of the
system package manager (an area dominated by setuptools, including both
eggs and non-zipped package distributions)
3. Bundle everything into a monolithic application or framework (the
typical approach on Windows with py2exe, but also the philosophy behind
tools like virtualenv)

All 3 of those philosophies have good arguments in their favour and they
all have good arguments against them as well. Your comments about your
package management system suggest that it is just yet another entrant in
category 2 and you have said nothing to allay the concerns of those that
despise that philosophy with a passion because of the problems it causes
them when trying to manage their systems using the first philosophy.

Since the Python constituency includes developers and system
administrators that favour all 3 philosophies (and probably other
variants that I haven't thought to describe), anything that makes it
into the standard library will need to adequately balance the interests
of all parties (e.g. as has occurred in the PEP 376 metadata enhancement
discussions).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From tjreedy at udel.edu  Wed Jul 29 07:42:43 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 29 Jul 2009 01:42:43 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6F9E34.5010708@gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>	<4A6C9371.6000002@v.loewis.de>	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>	<4A6E88B9.5010504@v.loewis.de>	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>	<4A6E92BF.7080901@v.loewis.de>	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
	<4A6F9E34.5010708@gmail.com>
Message-ID: <h4ongh$bb2$1@ger.gmane.org>

Nick Coghlan wrote:
> David Lyon wrote:
>> Your whole email whilst perphaps technically correct is terribly
>> difficult for a software engineering person to follow.
> 
> It made perfect sense to me.

Like David, I found it a bit disjointed too.
> 
> The words "eggs" brings with it a whole lot more baggage than just the
> sum of the technical parts in the language core that support them
> (primarily distutils and zipimport). I find it unfortunate that the name
> for the distutils metadata format contains the word "egg" because it
> implies much greater consensus around the philosophy behind eggs than
> actually exists.
> 
> A lot of the baggage associated with the "eggs" concept is related to
> the inherent conflict between different approaches to dependency management:
> 1. Use the system package management system for everything (preferred by
> many, perhaps even most, *nix sysadmins, but not an option on Windows)
> 2. Create a Python specific package management system independent of the
> system package manager (an area dominated by setuptools, including both
> eggs and non-zipped package distributions)
> 3. Bundle everything into a monolithic application or framework (the
> typical approach on Windows with py2exe, but also the philosophy behind
> tools like virtualenv)
> 
> All 3 of those philosophies have good arguments in their favour and they
> all have good arguments against them as well. Your comments about your
> package management system suggest that it is just yet another entrant in
> category 2 and you have said nothing to allay the concerns of those that
> despise that philosophy with a passion because of the problems it causes
> them when trying to manage their systems using the first philosophy.
> 
> Since the Python constituency includes developers and system
> administrators that favour all 3 philosophies (and probably other
> variants that I haven't thought to describe), anything that makes it
> into the standard library will need to adequately balance the interests
> of all parties (e.g. as has occurred in the PEP 376 metadata enhancement
> discussions).

However, the above clarifies for me what the alternatives and issues 
are. Thank you for posting.

Terry Jan Reedy


From david.lyon at preisshare.net  Wed Jul 29 08:36:30 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 29 Jul 2009 02:36:30 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <4A6F9E34.5010708@gmail.com>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>	<4A6C9371.6000002@v.loewis.de>	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>	<4A6E88B9.5010504@v.loewis.de>	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>	<4A6E92BF.7080901@v.loewis.de>	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
	<4A6F9E34.5010708@gmail.com>
Message-ID: <e6e3479bf6a207bd108baac38e173ab9@preisshare.net>

On Wed, 29 Jul 2009 10:56:20 +1000, Nick Coghlan <ncoghlan at gmail.com>
wrote:
> The words "eggs" brings with it a whole lot more baggage than just the
> sum of the technical parts in the language core that support them
> (primarily distutils and zipimport). 

Well, in this case, (talking metaphorically) we have the ability
to roll the eggs, crack a whole in them and suck out the contents
(deal with a package in a zipped egg)

So metaphorically we could say that python can work with egg shells..

As for what's in the egg... well lets just say that it's a bit floaty..

And perphaps that is the best way for things to be for a while.

> I find it unfortunate that the name
> for the distutils metadata format contains the word "egg" because it
> implies much greater consensus around the philosophy behind eggs than
> actually exists.

I see it a different way. I thinks eggs are simple - as in like a java
bean. A simple way to move a package from one place to another. 

What seems to have gone wrong is that there is a lot of pioneering
and interconnected and interdependent technology within them. They're
an egg.. but in the past they've had little monsters that bite your
fingers when you try to put them in the pan... :-)

I think "Egg" term is very good. But maybe we are best not trying
to over-specify their contents.

Maybe we should say it has certain things (EGG_INFO, PACKAGE DATA)
as in (YOLK, WHITE) and pretty much leave it at that. If more detail
is required - rtm.

> A lot of the baggage associated with the "eggs" concept is related to
> the inherent conflict between different approaches to dependency
> management:

That's not an egg problem. That's a packaging/metadata problem.

It's the package dependency issue that's the problem. That's a distutils
problem.

> 1. Use the system package management system for everything (preferred by
> many, perhaps even most, *nix sysadmins, but not an option on Windows)

Yes, because the package dependency issues are just so great.

> 2. Create a Python specific package management system independent of the
> system package manager (an area dominated by setuptools, including both
> eggs and non-zipped package distributions)

More work needs to go into distutils. Not taking away from any existing
work - but setuptools has relied on the deficiencies of distutils to
gain a foothold.

Fix distutils (give me time to think..) and the need for setuptools and
any further fork is probably negated.

> 3. Bundle everything into a monolithic application or framework (the
> typical approach on Windows with py2exe, but also the philosophy behind
> tools like virtualenv)

Windows users are using py2exe and other products over distutils. To a
normal windows developer, distutils makes imho no sense in the way that
it goes about things now.

For example, very simple concepts like "program directories", (where
you stick the program) isn't an available option in distutils. But
it is the first thing that you need to know in a windows program.

So it's very hard to get to step 1...

> Your comments about your
> package management system suggest that it is just yet another entrant in
> category 2 and you have said nothing to allay the concerns of those that
> despise that philosophy with a passion because of the problems it causes
> them when trying to manage their systems using the first philosophy.

I'm a windows user.. I can't be in category #2..

I'm in category #1, have nothing... (except now my own tool)

> Since the Python constituency includes developers and system
> administrators that favour all 3 philosophies (and probably other
> variants that I haven't thought to describe), anything that makes it
> into the standard library will need to adequately balance the interests
> of all parties (e.g. as has occurred in the PEP 376 metadata enhancement
> discussions).

Well at least you are saying that there is some way of reconciling
all the different options...

There's an awful lot to take in, and there must be 20,000 lines of
emails for every 1 line of python code that is required to fix this
thing.

Take care

David 


From ncoghlan at gmail.com  Wed Jul 29 10:10:26 2009
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 29 Jul 2009 18:10:26 +1000
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <e6e3479bf6a207bd108baac38e173ab9@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>	<f4e10cc3f24f99037c3de34ac657a6ca@preisshare.net>	<200907261805.08277.steve@pearwood.info>	<ac0891ad066cc0fe48ae64c72cd29674@preisshare.net>	<4A6C9371.6000002@v.loewis.de>	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>	<4A6E88B9.5010504@v.loewis.de>	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>	<4A6E92BF.7080901@v.loewis.de>	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
	<4A6F9E34.5010708@gmail.com>
	<e6e3479bf6a207bd108baac38e173ab9@preisshare.net>
Message-ID: <4A7003F2.4080308@gmail.com>

David Lyon wrote:
> There's an awful lot to take in, and there must be 20,000 lines of
> emails for every 1 line of python code that is required to fix this
> thing.

Yep, which goes way back to one of my first emails in this thread:
compared to the social aspects, the technical aspects of packaging tools
are relatively straightforward :)

It's great to see some energy directed towards solving these issues, but
I think it's important for those trying to contribute to realise what
they're dealing with. I just hope I don't kill their enthusiasm in the
process :P

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From jnoller at gmail.com  Wed Jul 29 15:11:24 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 29 Jul 2009 09:11:24 -0400
Subject: [Python-Dev] Update to Python Documentation Website Request
In-Reply-To: <e6e3479bf6a207bd108baac38e173ab9@preisshare.net>
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
	<4A6F9E34.5010708@gmail.com>
	<e6e3479bf6a207bd108baac38e173ab9@preisshare.net>
Message-ID: <4222a8490907290611l1b31c3acr71c2baae3913af3e@mail.gmail.com>

On Wed, Jul 29, 2009 at 2:36 AM, David Lyon<david.lyon at preisshare.net> wrote:
> On Wed, 29 Jul 2009 10:56:20 +1000, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
>> The words "eggs" brings with it a whole lot more baggage than just the
>> sum of the technical parts in the language core that support them
>> (primarily distutils and zipimport).
>
> Well, in this case, (talking metaphorically) we have the ability
> to roll the eggs, crack a whole in them and suck out the contents
> (deal with a package in a zipped egg)
>
> So metaphorically we could say that python can work with egg shells..
>
> As for what's in the egg... well lets just say that it's a bit floaty..
>
> And perphaps that is the best way for things to be for a while.
>
>> I find it unfortunate that the name
>> for the distutils metadata format contains the word "egg" because it
>> implies much greater consensus around the philosophy behind eggs than
>> actually exists.
>
> I see it a different way. I thinks eggs are simple - as in like a java
> bean. A simple way to move a package from one place to another.
>
> What seems to have gone wrong is that there is a lot of pioneering
> and interconnected and interdependent technology within them. They're
> an egg.. but in the past they've had little monsters that bite your
> fingers when you try to put them in the pan... :-)
>
> I think "Egg" term is very good. But maybe we are best not trying
> to over-specify their contents.
>
> Maybe we should say it has certain things (EGG_INFO, PACKAGE DATA)
> as in (YOLK, WHITE) and pretty much leave it at that. If more detail
> is required - rtm.
>
>> A lot of the baggage associated with the "eggs" concept is related to
>> the inherent conflict between different approaches to dependency
>> management:
>
> That's not an egg problem. That's a packaging/metadata problem.
>
> It's the package dependency issue that's the problem. That's a distutils
> problem.
>
>> 1. Use the system package management system for everything (preferred by
>> many, perhaps even most, *nix sysadmins, but not an option on Windows)
>
> Yes, because the package dependency issues are just so great.
>
>> 2. Create a Python specific package management system independent of the
>> system package manager (an area dominated by setuptools, including both
>> eggs and non-zipped package distributions)
>
> More work needs to go into distutils. Not taking away from any existing
> work - but setuptools has relied on the deficiencies of distutils to
> gain a foothold.
>
> Fix distutils (give me time to think..) and the need for setuptools and
> any further fork is probably negated.
>
>> 3. Bundle everything into a monolithic application or framework (the
>> typical approach on Windows with py2exe, but also the philosophy behind
>> tools like virtualenv)
>
> Windows users are using py2exe and other products over distutils. To a
> normal windows developer, distutils makes imho no sense in the way that
> it goes about things now.
>
> For example, very simple concepts like "program directories", (where
> you stick the program) isn't an available option in distutils. But
> it is the first thing that you need to know in a windows program.
>
> So it's very hard to get to step 1...
>
>> Your comments about your
>> package management system suggest that it is just yet another entrant in
>> category 2 and you have said nothing to allay the concerns of those that
>> despise that philosophy with a passion because of the problems it causes
>> them when trying to manage their systems using the first philosophy.
>
> I'm a windows user.. I can't be in category #2..
>
> I'm in category #1, have nothing... (except now my own tool)
>
>> Since the Python constituency includes developers and system
>> administrators that favour all 3 philosophies (and probably other
>> variants that I haven't thought to describe), anything that makes it
>> into the standard library will need to adequately balance the interests
>> of all parties (e.g. as has occurred in the PEP 376 metadata enhancement
>> discussions).
>
> Well at least you are saying that there is some way of reconciling
> all the different options...
>
> There's an awful lot to take in, and there must be 20,000 lines of
> emails for every 1 line of python code that is required to fix this
> thing.
>
> Take care
>
> David
>

I really do think this mail thread needs to move to disutils-sig or
python-ideas.

jesse

From solipsis at pitrou.net  Wed Jul 29 15:45:36 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 29 Jul 2009 13:45:36 +0000 (UTC)
Subject: [Python-Dev] Update to Python Documentation Website Request
References: <eb4dbdc278e338fb0611c200b990cb8f@preisshare.net>
	<1b9d4c7557e19faed93300f157d07b79@preisshare.net>
	<4A6E88B9.5010504@v.loewis.de>
	<14cf53f0cf73387c177f045fd21151b7@preisshare.net>
	<4A6E92BF.7080901@v.loewis.de>
	<3fcf944249977ccd966d62a73ac0db5d@preisshare.net>
	<79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com>
	<2e8c8bc947207c89e75c74cbbcba8920@preisshare.net>
	<4A6F9E34.5010708@gmail.com>
	<e6e3479bf6a207bd108baac38e173ab9@preisshare.net>
	<4222a8490907290611l1b31c3acr71c2baae3913af3e@mail.gmail.com>
Message-ID: <loom.20090729T134527-720@post.gmane.org>

Jesse Noller <jnoller <at> gmail.com> writes:
> 
> I really do think this mail thread needs to move to disutils-sig or
> python-ideas.

+1



From ideasman42 at gmail.com  Wed Jul 29 23:41:49 2009
From: ideasman42 at gmail.com (Campbell Barton)
Date: Wed, 29 Jul 2009 14:41:49 -0700
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <loom.20090726T171916-534@post.gmane.org>
References: <loom.20090726T171916-534@post.gmane.org>
Message-ID: <7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com>

On Sun, Jul 26, 2009 at 11:01 AM, Joshua Haberman<joshua at reverberate.org> wrote:
> I'm writing a C Python extension that needs to generate PyTypeObjects
> dynamically. ?Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded
> in a way that makes it difficult to achieve this goal.
>
> The documentation for Pt_TPFLAGS_HEAPTYPE says:
>
> Py_TPFLAGS_HEAPTYPE
>
> ?This bit is set when the type object itself is allocated
> ?on the heap. In this case, the ob_type field of its
> ?instances is considered a reference to the type, and the
> ?type object is INCREF?ed when a new instance is created,
> ?and DECREF?ed when an instance is destroyed (this does not
> ?apply to instances of subtypes; only the type referenced
> ?by the instance?s ob_type gets INCREF?ed or DECREF?ed).
>
> This sounds like exactly what I want. ?I want my type object INCREF'd
> and DECREF'd by its instances so it doesn't leak or get deleted
> prematurely. ?If this were all that Py_TPFLAGS_HEAPTYPE did, it would
> work great for me.
>
> Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean
> "user-defined type" (as opposed to a built-in type). ?It controls
> numerous subtle behaviors such as:
>
> - whether the type's name is module.type or just type.
> - whether you're allowed to set __name__, __module__, or __bases__ on the type.
> - whether you're allowed to set __class__ on instances of this type.
> - whether the module name comes from the type name or the __module__ attribute.
> - whether it will use type->tp_doc as the docstring
> - whether its repr() calls it a "class" or a "type".
> - whether you can set attributes of the type.
> - whether someone is attempting the Carlo Verre hack.
>
> So I'm stuck with an unenviable choice. ?I think the lesser of two evils
> is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will
> happen is that my types will leak. ?This is not as bad as having someone
> set __class__ on one of my instances, or set attributes on my type, etc.
>
> Ideally the interpreter would have a separate flag like
> Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but
> still make it possible to have dynamically generated built-in types get
> garbage collected appropriately.
>
> At the very least, the documentation I cited above should make it clear
> that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets
> INCREF'd and DECREF'd. ?Based on the list of behaviors I discovered
> above, it is almost certainly not correct for a C exension type to be
> declared with Py_TPFLAGS_HEAPTYPE.
>
> Josh

Hi Joshua, recently I also needed to dynamically make subtypes from C,

I tried 2 ways of doing this, one is to do the C equivalent of calling
type("name",(bases,...), dict)  and the other is to malloc()
PyTypeObject's, fill in the slots and run PyType_Ready on them to
initialize them.

It seems the first is the expected way to make your own types so I
assume thats what your doing?, Just wondering because if you do it the
second way I think youll have more control and the types will be more
limited (like internal types).

I'm not expert enough in this area to know if malloc'ing PyTypeObject
and initializing has some other problems.

- Campbell

From eric.pruitt at gmail.com  Thu Jul 30 01:41:05 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Wed, 29 Jul 2009 18:41:05 -0500
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> 
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> 
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> 
	<79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> 
	<d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com> 
	<52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com>
Message-ID: <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com>

My motivation came from an instance when I was using subprocess.Popen for a
Linux / Windows cross platform program. In part of the program, I was
writing and reading to a cron like object. On Windows, it was a text file
and on Linux it would be the crontab executable. Had I been able to
substitute the "open()" function with my wrapper, it would have been the
only change I had to make for cross platform compatibility; instead of
having to change numerous lines because Linux would need Popen and Windows
would need a regular file open(), I could simply make it so that if the
platform was Linux, my wrapper is used in place of that. Just another
example would be having an external program decrypt a file that can be in
plain text or encrypted that might go something like this:

if encryptionEnabled:
    fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
supersecret.html.gpg")
else:
    fileobj = open("notsosecret.html")

>From there, the functions would not have to be modified despite using a
running process versus a file object.

On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith <greg at krypto.org> wrote:

> On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz<glyph at twistedmatrix.com>
> wrote:
> > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> >>
> >> I like MRAB's idea of using a (non-standard) "e" flag to include
> >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
> >>
> >> Anything more complicated probably should just use "raw" Popen
> >> objects. Don't overcomplicate the interface.
> >
> > In my opinion, mangling stderr and stdout together is already an
> > overcomplication.  It shouldn't be implemented.
> >
> > It seems like a good idea, until you realize that subtle changes to your
> OS,
> > environment, or buffering behavior may result in arbitrary, unparseable
> > output.
>
> Agreed.  Leave stderr support out of this.  People who need stderr
> should use the full subprocess.Popen interface.  Quick hack unixy
> types will just run their process using a shell (which already seems
> to be the default based on the "ls -l" example) and add 2>&1.  This
> functionality is basically the equivalent of adding the | symbol on
> either or both ends of a filename given to open() in perl.  (but
> without being so gross).
>
> I do wonder why you're trying to make it behave exactly like open()
> including the mode= syntax.
>
> Why not just define several names based on the behavior?
>
>  ProcessReadWrapper()
>  ProcessWriteWrapper()
>  ProcessReadWriteWrapper()
>
> -gps
>
> >
> > For example, let's say you've got a program whose output is a list of
> lines,
> > each one containing a number.  Sometimes it tries to import gtk, and
> fails
> > to open its display.
> >
> > That's fine, and you can still deal with it, as long as the interleaved
> > output looks like this:
> >
> > 100
> > 200
> > Gtk-WARNING **: cannot open display:
> > 300
> > 400
> >
> > but of course the output might (although unlikely with such small chunks
> of
> > output) end up looking like this, instead:
> >
> > 100
> > 2Gtk-WAR0NING0 **:
> > can30not 0open display:
> >
> > 400
> >
> > this is the sort of thing which is much more likely to happen once you
> start
> > dealing with large volumes of data, where there are more page-boundaries
> for
> > your buffers to get confused around, and you are playing with buffering
> > options to improve performance.  In other words, it's something that
> fails
> > only at scale or under load, and is therefore extremely difficult to
> debug.
> >
> > This option might be okay if it were allowed only on subprocesses opened
> in
> > a text mode, and if the buffering logic involved forced stderr and stdout
> to
> > be line-delimited, and interleave only lines, rather than arbitrary
> chunks
> > of bytes.  Of course then if you use this flag with a program that
> outputs
> > binary data with no newlines it will buffer forever and crash your
> program
> > with a MemoryError, but at least that's easy to debug when it happens.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090729/8cd94d64/attachment-0001.htm>

From devin.c.cook at gmail.com  Thu Jul 30 03:20:36 2009
From: devin.c.cook at gmail.com (Devin Cook)
Date: Wed, 29 Jul 2009 21:20:36 -0400
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com>
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com>
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com>
	<79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com>
	<d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com>
	<52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com>
	<171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com>
Message-ID: <8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com>

Hmm... can't you do this?

if encryptionEnabled:
    p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"],
stdin = subprocess.PIPE)
    fileobj = p.stdin
else:
    fileobj = open("notsosecret.html")

I think that works. Is there something this way won't work for? You
can also do the same thing to get stdout and stderr file objects. I
guess a wrapper would simplify this process.

-Devin

On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt<eric.pruitt at gmail.com> wrote:
> My motivation came from an instance when I was using subprocess.Popen for a
> Linux / Windows cross platform program. In part of the program, I was
> writing and reading to a cron like object. On Windows, it was a text file
> and on Linux it would be the crontab executable. Had I been able to
> substitute the "open()" function with my wrapper, it would have been the
> only change I had to make for cross platform compatibility; instead of
> having to change numerous lines because Linux would need Popen and Windows
> would need a regular file open(), I could simply make it so that if the
> platform was Linux, my wrapper is used in place of that. Just another
> example would be having an external program decrypt a file that can be in
> plain text or encrypted that might go something like this:
>
> if encryptionEnabled:
> ??? fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
> supersecret.html.gpg")
> else:
> ??? fileobj = open("notsosecret.html")
>

From eric.pruitt at gmail.com  Thu Jul 30 05:02:22 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Wed, 29 Jul 2009 22:02:22 -0500
Subject: [Python-Dev] Implementing File Modes
In-Reply-To: <8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com>
References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> 
	<79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> 
	<171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> 
	<79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> 
	<d9047f780907271732w50ac1462ra2043878605d39b8@mail.gmail.com> 
	<52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> 
	<171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com> 
	<8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com>
Message-ID: <171e8a410907292002w66d26d91h5879d92b9fcb1735@mail.gmail.com>

Well, with a few changes to your code, that would indeed work (you are using
stdin as your pipe. Correct me if I'm wrong but if you intend to read from
it, you need to change it to "stdout = subprocess.PIPE" and the other lines
as well to reflect this change). My Google Summer of Code modifications to
subprocess.Popen provide non-blocking, asynchronous I/O support and my file
wrapper is built upon that augmented functionality. If I remember correctly,
when I was working on the program where I first thought a file wrapper for
subprocess.Popen would be rather handy, I also ran into blocking I/O as
well.

On Wed, Jul 29, 2009 at 20:20, Devin Cook <devin.c.cook at gmail.com> wrote:

> Hmm... can't you do this?
>
> if encryptionEnabled:
>    p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"],
> stdin = subprocess.PIPE)
>    fileobj = p.stdin
> else:
>    fileobj = open("notsosecret.html")
>
> I think that works. Is there something this way won't work for? You
> can also do the same thing to get stdout and stderr file objects. I
> guess a wrapper would simplify this process.
>
> -Devin
>
> On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt<eric.pruitt at gmail.com> wrote:
> > My motivation came from an instance when I was using subprocess.Popen for
> a
> > Linux / Windows cross platform program. In part of the program, I was
> > writing and reading to a cron like object. On Windows, it was a text file
> > and on Linux it would be the crontab executable. Had I been able to
> > substitute the "open()" function with my wrapper, it would have been the
> > only change I had to make for cross platform compatibility; instead of
> > having to change numerous lines because Linux would need Popen and
> Windows
> > would need a regular file open(), I could simply make it so that if the
> > platform was Linux, my wrapper is used in place of that. Just another
> > example would be having an external program decrypt a file that can be in
> > plain text or encrypted that might go something like this:
> >
> > if encryptionEnabled:
> >     fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
> > supersecret.html.gpg")
> > else:
> >     fileobj = open("notsosecret.html")
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090729/06079ff7/attachment.htm>

From hrvoje.niksic at avl.com  Thu Jul 30 10:13:38 2009
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Thu, 30 Jul 2009 10:13:38 +0200
Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
In-Reply-To: <7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com>
References: <loom.20090726T171916-534@post.gmane.org>
	<7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com>
Message-ID: <4A715632.6010604@avl.com>

Campbell Barton wrote:
> I'm not expert enough in this area to know if malloc'ing PyTypeObject
> and initializing has some other problems.

The only problem is that such types will be expected to be around 
forever - they are not reference-counted like heap types, so there is no 
mechanism to free them once they are no longer needed.

From eric.pruitt at gmail.com  Thu Jul 30 18:37:29 2009
From: eric.pruitt at gmail.com (Eric Pruitt)
Date: Thu, 30 Jul 2009 11:37:29 -0500
Subject: [Python-Dev] Implementing File Modes
Message-ID: <171e8a410907300937x655e77c1h188d743d5a267cbf@mail.gmail.com>

> On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote:
> > I am implementing the file wrapper using changes to subprocess.Popen
> > that also make it asynchronous and non-blocking so implementing "r+"
> > should be trivial to do. How about handling stderr? I have the
> > following ideas: leave out support for reading from stderr, make it
> > so that there is an optional additional argument like "outputstderr =
> > False", create another function that toggles / sets whether stderr or
> > stdout is returned or mix the two outputs.
>
> Leaving it out is always an option.
>
> As I see it, fundamentally you can either read from stdout and sterr as
> two different streams, or you can interleave (mix) them. To me, that
> suggests three functions:
>
> ProcessIOWrapper() ?# read from stdout (or write to stdin etc.)
> ProcessIOWrapperStdErr() ?# read/write from stderr
> ProcessIOWrapper2() ?# read from mixed stdout and stderr
>
> I don't like a function to toggle between one and the other: that smacks
> of relying on a global setting in a bad way. I suppose you could add an
> optional argument to ProcessIOWrapper() to select between stdout,
> stderr, or both together.
>
>
>
> --
> Steven D'Aprano

How would having a toggle function rely on a global setting? Each class would
simply have its own member variable like "self.readsstderr." It's a moot point
though as I've decided to use separate functions as you suggested. With
separate functions, the user doesn't have to worry about modifying the mode
keyword if stderr is needed and as an added bonus, it also makes the code
a lot more readable.

Eric

From status at bugs.python.org  Fri Jul 31 18:07:24 2009
From: status at bugs.python.org (Python tracker)
Date: Fri, 31 Jul 2009 18:07:24 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20090731160724.190F8785FE@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (07/24/09 - 07/31/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2292 open (+33) / 16143 closed (+18) / 18435 total (+51)

Open issues with patches:   899

Average duration of open issues: 660 days.
Median duration of open issues: 414 days.

Open Issues Breakdown
   open  2261 (+32)
pending    30 ( +0)

Issues Created Or Reopened (52)
_______________________________

string.lowercase/uppercase/letters not affected by locale change 07/25/09
CLOSED http://bugs.python.org/issue6525    reopened ezio.melotti                  
                                                                               

inserting None into sys.modules does not raise ImportError with  07/24/09
       http://bugs.python.org/issue6563    created  brett.cannon                  
                                                                               

Error in Sec. 8.4 of Tutorial                                    07/24/09
CLOSED http://bugs.python.org/issue6564    created  gmarsden                      
                                                                               

improper use of __setitem__ in ElementTree for Python 3.1        07/24/09
       http://bugs.python.org/issue6565    created  aroberge                      
                                                                               

json.dumps converts None to "null" (not null)                    07/24/09
       http://bugs.python.org/issue6566    created  srid                          
                                                                               

Make inf be almost equal to inf                                  07/25/09
       http://bugs.python.org/issue6567    reopened lucaspmelo                    
                                                                               

unittest test discovery improvements                             07/24/09
       http://bugs.python.org/issue6568    created  michael.foord                 
                                                                               

unittest document bug (random.shuffle sequence)                  07/25/09
CLOSED http://bugs.python.org/issue6569    created  hsmtkk                        
       patch                                                                   

Documentation Clarity                                            07/25/09
       http://bugs.python.org/issue6570    created  StMark                        
                                                                               

Help index                                                       07/25/09
CLOSED http://bugs.python.org/issue6571    created  cjw                           
                                                                               

Manas Thapliyal wants to chat                                    07/25/09
CLOSED http://bugs.python.org/issue6572    created  gravitywarrior1               
                                                                               

set union method ignores arguments appearing after the original  07/25/09
CLOSED http://bugs.python.org/issue6573    created  ssmout                        
       patch                                                                   

List the __future__ features in a table                          07/26/09
       http://bugs.python.org/issue6574    created  ezio.melotti                  
       patch, easy                                                             

Can't download docs                                              07/26/09
CLOSED http://bugs.python.org/issue6575    created  zuo                           
                                                                               

re docs: wrong link targets                                      07/26/09
CLOSED http://bugs.python.org/issue6576    created  zuo                           
                                                                               

Links wrongly targeting to builtin functions' instead of module  07/26/09
CLOSED http://bugs.python.org/issue6577    created  zuo                           
                                                                               

2 problems with 'Docs for other versions' section on left HTML d 07/26/09
CLOSED http://bugs.python.org/issue6578    created  zuo                           
                                                                               

No update about automatic numbering of fields in format strings  07/26/09
       http://bugs.python.org/issue6579    reopened georg.brandl                  
                                                                               

No deprecation warning for list comprehension leak conflict      07/26/09
       http://bugs.python.org/issue6580    created  mrbax                         
                                                                               

inspect.py sys._getframe patch for Python 2.6                    07/26/09
CLOSED http://bugs.python.org/issue6581    created  michael.foord                 
       patch, patch, needs review                                              

test_telnetlib doesn't test Telnet.write                         07/26/09
       http://bugs.python.org/issue6582    created  jackdied                      
       easy                                                                    

2to3 fails to fix test.test_support                              07/27/09
       http://bugs.python.org/issue6583    created  joe.amenta                    
                                                                               

gzip module has no custom exception                              07/27/09
       http://bugs.python.org/issue6584    created  srid                          
                                                                               

configure.in forces specific autoconf version                    07/27/09
CLOSED http://bugs.python.org/issue6585    created  cavallo71                     
                                                                               

Documentation of os.write and os.read are inaccurate.            07/27/09
CLOSED http://bugs.python.org/issue6586    created  cliechti                      
                                                                               

interrupts during long writes cause connection corruption with S 07/28/09
       http://bugs.python.org/issue6587    created  janssen                       
                                                                               

insert cookies into cookie jar - cookielib.py                    07/28/09
       http://bugs.python.org/issue6588    created  jondebonis                    
       patch                                                                   

smtpd.SMTPServer can cause asyncore.loop to enter infinite event 07/28/09
       http://bugs.python.org/issue6589    created  cmcginty                      
                                                                               

Py3K can't set the browser's character encoding! (CGI)           07/28/09
CLOSED http://bugs.python.org/issue6590    created  [PhysiC]                      
                                                                               

add reference to fcntl.ioctl in the socket module                07/28/09
CLOSED http://bugs.python.org/issue6591    created  gagenellina                   
       patch                                                                   

otparse: processing variable number of short option arguments    07/28/09
       http://bugs.python.org/issue6592    created  tanvalley                     
                                                                               

Documentation: gettext install link                              07/28/09
CLOSED http://bugs.python.org/issue6593    created  efrerich                      
                                                                               

json C serializer performance tied to structure depth on some sy 07/28/09
       http://bugs.python.org/issue6594    created  swalker                       
                                                                               

Make Decimal constructor accept all unicode decimal digits in in 07/29/09
       http://bugs.python.org/issue6595    created  marketdickinson               
       patch                                                                   

urllib2 bug on CentOS                                            07/29/09
       http://bugs.python.org/issue6596    created  rk3dov                        
                                                                               

Deprecate iterable.next in Python > 2.6.x when called with -3 op 07/29/09
CLOSED http://bugs.python.org/issue6597    created  mattrussell                   
                                                                               

calling email.utils.make_msgid frequently has a non-trivial prob 07/30/09
       http://bugs.python.org/issue6598    created  mwh                           
                                                                               

2to3 test_print_function_option fails on Windows                 07/30/09
       http://bugs.python.org/issue6599    created  abbeyj                        
       patch                                                                   

MemoryError in AiX 64-bit build - PyMem_MALLOC failed            07/30/09
       http://bugs.python.org/issue6600    created  srid                          
                                                                               

Fractions do not support other Fractions as numerators or denomi 07/30/09
CLOSED http://bugs.python.org/issue6601    created  honeyman                      
                                                                               

BaseHTTPServer log_message should log to sys.stdout              07/30/09
       http://bugs.python.org/issue6602    created  tomkm                         
                                                                               

Compilation error if configuref --with-computed-gotos            07/30/09
       http://bugs.python.org/issue6603    created  wiget                         
                                                                               

test_distutils subtest test_get_exe_bytes fails depending on exe 07/30/09
       http://bugs.python.org/issue6604    created  matejcik                      
       patch                                                                   

smtplib.SMTP.sendmail() rejected after quit(),connect() sequence 07/30/09
       http://bugs.python.org/issue6605    created  cmcginty                      
                                                                               

csv.Sniffer.sniff on data with doublequotes doesn't set up the d 07/30/09
       http://bugs.python.org/issue6606    created  jtate                         
       patch                                                                   

asctime causing python to crash                                  07/30/09
CLOSED http://bugs.python.org/issue6607    created  AmirHabibi                    
                                                                               

asctime causing python to crash                                  07/30/09
       http://bugs.python.org/issue6608    created  AmirHabibi                    
                                                                               

zipfile: WindowsError [267] The directory name is invalid        07/31/09
       http://bugs.python.org/issue6609    created  srid                          
                                                                               

Subprocess descriptor debacle                                    07/31/09
       http://bugs.python.org/issue6610    created  christian.heimes              
       easy, needs review                                                      

HTMLParser cannot deal with mixture of arbitrary data and charac 07/31/09
       http://bugs.python.org/issue6611    created  liudongmiao at gmail.com         
                                                                               

'import site' fails when called from an unlinked directory       07/31/09
       http://bugs.python.org/issue6612    created  labrat                        
                                                                               

ctypes.PyThreadState_SetAsyncExc does not work on linux 64bit ma 07/31/09
       http://bugs.python.org/issue6613    created  coutinhotiago                 
                                                                               



Issues Now Closed (40)
______________________

The ValueError raised by failing to unpack sequence should have   412 days
       http://bugs.python.org/issue3071    jackdied                      
       patch                                                                   

telnetlib process_rawq buffer handling is confused                168 days
       http://bugs.python.org/issue5188    jackdied                      
       patch                                                                   

Subprocess.getstatusoutput Fails Executing 'dir' Command on Wind   98 days
       http://bugs.python.org/issue5808    sorin                         
                                                                               

spurious space after opening parenthesis when auto-completing      78 days
       http://bugs.python.org/issue5992    pitrou                        
                                                                               

Document and slightly simplify lnotab tracing                      75 days
       http://bugs.python.org/issue6042    alexandre.vassalotti          
       patch, needs review                                                     

read_until                                                         63 days
       http://bugs.python.org/issue6106    jackdied                      
                                                                               

Make logging configuration files easier to use                     63 days
       http://bugs.python.org/issue6136    vsajip                        
                                                                               

logging.basicConfig(level='DEBUG', ... and setLevel("DEBUG") res   24 days
       http://bugs.python.org/issue6314    vsajip                        
       patch                                                                   

trunk does not build with --enable-unicode=ucs4                    37 days
       http://bugs.python.org/issue6330    eric.smith                    
       easy                                                                    

nb_divide missing in docs                                          35 days
       http://bugs.python.org/issue6336    georg.brandl                  
                                                                               

TimedRotatingFileHandler permission denied rename failure on Win   35 days
       http://bugs.python.org/issue6343    vsajip                        
                                                                               

zipfile: Invalid argument when opening zero-sized files            11 days
       http://bugs.python.org/issue6511    amaury.forgeotdarc            
                                                                               

string.lowercase/uppercase/letters not affected by locale change    0 days
       http://bugs.python.org/issue6525    georg.brandl                  
                                                                               

cmath documentation misleading:  suggests existence of real() an    6 days
       http://bugs.python.org/issue6548    marketdickinson               
       patch                                                                   

Regex '\d' should not match unicode category 'No'.                  4 days
       http://bugs.python.org/issue6561    marketdickinson               
       patch, needs review                                                     

Error in Sec. 8.4 of Tutorial                                       0 days
       http://bugs.python.org/issue6564    georg.brandl                  
                                                                               

unittest document bug (random.shuffle sequence)                     0 days
       http://bugs.python.org/issue6569    benjamin.peterson             
       patch                                                                   

Help index                                                          0 days
       http://bugs.python.org/issue6571    georg.brandl                  
                                                                               

Manas Thapliyal wants to chat                                       0 days
       http://bugs.python.org/issue6572    benjamin.peterson             
                                                                               

set union method ignores arguments appearing after the original     2 days
       http://bugs.python.org/issue6573    rhettinger                    
       patch                                                                   

Can't download docs                                                 0 days
       http://bugs.python.org/issue6575    georg.brandl                  
                                                                               

re docs: wrong link targets                                         0 days
       http://bugs.python.org/issue6576    georg.brandl                  
                                                                               

Links wrongly targeting to builtin functions' instead of module     0 days
       http://bugs.python.org/issue6577    georg.brandl                  
                                                                               

2 problems with 'Docs for other versions' section on left HTML d    0 days
       http://bugs.python.org/issue6578    georg.brandl                  
                                                                               

inspect.py sys._getframe patch for Python 2.6                       0 days
       http://bugs.python.org/issue6581    michael.foord                 
       patch, patch, needs review                                              

configure.in forces specific autoconf version                       1 days
       http://bugs.python.org/issue6585    r.david.murray                
                                                                               

Documentation of os.write and os.read are inaccurate.               2 days
       http://bugs.python.org/issue6586    georg.brandl                  
                                                                               

Py3K can't set the browser's character encoding! (CGI)              0 days
       http://bugs.python.org/issue6590    [PhysiC]                      
                                                                               

add reference to fcntl.ioctl in the socket module                   1 days
       http://bugs.python.org/issue6591    georg.brandl                  
       patch                                                                   

Documentation: gettext install link                                 1 days
       http://bugs.python.org/issue6593    georg.brandl                  
                                                                               

Deprecate iterable.next in Python > 2.6.x when called with -3 op    1 days
       http://bugs.python.org/issue6597    brett.cannon                  
                                                                               

Fractions do not support other Fractions as numerators or denomi    0 days
       http://bugs.python.org/issue6601    marketdickinson               
                                                                               

asctime causing python to crash                                     0 days
       http://bugs.python.org/issue6607    alexandre.vassalotti          
                                                                               

Hang using files named prn.txt, etc                              2812 days
       http://bugs.python.org/issue481171  ezio.melotti                  
                                                                               

Allow set swig include dirs in setup.py                          2108 days
       http://bugs.python.org/issue828336  wiget                         
       patch                                                                   

sendmsg() and recvmsg() for C socket module                      1546 days
       http://bugs.python.org/issue1194378 ezio.melotti                  
       patch                                                                   

telnetlib expect() and read_until() do not time out properly     1346 days
       http://bugs.python.org/issue1360221 jackdied                      
       patch                                                                   

urllib/urllib2: HTTPS over (Squid) Proxy fails                   1268 days
       http://bugs.python.org/issue1424152 orsenthil                     
       patch                                                                   

urllib2+https+proxy not working                                  1231 days
       http://bugs.python.org/issue1448934 orsenthil                     
                                                                               

Telnetlib dosn't accept u'only ascii'                             714 days
       http://bugs.python.org/issue1772794 jackdied                      
                                                                               



Top Issues Most Discussed (10)
______________________________

 28 base64 module ignores non-alphabet characters                   1212 days
open    http://bugs.python.org/issue1466065

  8 asyncore incorrect failure when connection is refused and using    9 days
open    http://bugs.python.org/issue6550   

  8 Regexp 2.7 (modifications to current re 2.2.2)                   472 days
open    http://bugs.python.org/issue2636   

  7 MemoryError in AiX 64-bit build - PyMem_MALLOC failed              2 days
open    http://bugs.python.org/issue6600   

  7 Make inf be almost equal to inf                                    6 days
open    http://bugs.python.org/issue6567   

  7 OverflowError in RLock.acquire()                                   7 days
open    http://bugs.python.org/issue6562   

  6 csv.Sniffer.sniff on data with doublequotes doesn't set up the     1 days
open    http://bugs.python.org/issue6606   

  6 No update about automatic numbering of fields in format strings    5 days
open    http://bugs.python.org/issue6579   

  6 Regex '\d' should not match unicode category 'No'.                 4 days
closed  http://bugs.python.org/issue6561   

  6 email.FeedParser.BufferedSubFile improperly handles "\r\n"       804 days
open    http://bugs.python.org/issue1721862




From jacobolus at gmail.com  Fri Jul 31 23:16:02 2009
From: jacobolus at gmail.com (Jacob Rus)
Date: Fri, 31 Jul 2009 21:16:02 +0000 (UTC)
Subject: [Python-Dev] standard library mimetypes module pathologically
	broken?
Message-ID: <loom.20090731T211507-952@post.gmane.org>

Hi all,

In an attempt to figure out some twisted.web code, I was reading
through the Python Standard Library?s mimetypes module today, and
was shocked at the poor quality of the code. I wonder how the
mimetypes code made it into the standard library, and whether anyone
has ever bothered to read it or update it: it is an embarrassment. 
Much of the code is redundant, portions fail to execute, control 
flow is routed through a horribly confusing mess of spaghetti, and 
most of the complexity has no clear benefit as far as I can tell. I 
probably should drop the subject and get back to work, but as a good 
citizen, it?s hard to just ignore this sort of thing.

mimetypes.py stores its types in a pair of dictionaries, one for
"strict" use, and the other for "non-standard types". It creates the
strict dictionary by default out of apache's mime.types file, and
then overrides the entries it finds with a set of exceptions. Then
it creates the non-standard dictionary, which is set to match if the
strict parameter is set to False when guessing types. Just in this
basic design, and in the list of types in the file, there are
several problems:

  * Various apache mime types files are read, if found, but the 
    ordering of the files is such that older versions of apache are
    sometimes read after newer ones, overriding updated mime types
    with out-of-date versions if multiple versions of apache are
    installed on the system.
    
  * The vast majority of types declared in mimetypes.py are 
    duplicates of types already declared by Apache. In a few cases
    this is to change the apache default (make an exception, that
    is), but in most cases the mime type and extension are
    completely identical. This huge number of redundant types makes
    the file substantially harder to follow. No comments are
    provided to explain why various sets of exceptions are made to
    Apache's default mime types, and in several cases mimetypes.py
    seems to just be out of date as compared to recent versions of
    Apache, for instance not knowing about the 'text/troff' type
    which was registered in January 2006 in RFC 4263.
    
  * The 'non-standard' type dictionary is nearly useless, because 
    all of the types it declares are already in apache's mime.types
    file, meaning that types are, as far as I can tell trying to
    follow ugly program flow, *never* drawn from the non-strict
    dictionary, except in the improbable situation where the 
    mimetypes module is initialized with a custom set of 
    apache-mime.types?like files, which does not include those 
    'non-standard' types. I personally cannot see a use case for 
    initializing the module with a custom set of mime types, but 
    then leaving the very few types included as non-strict to the 
    defaults: this seems like a fragile and pathological use case.
    Given this, I don?t see any benefit to dragging the 'strict'
    parameter along all the way through the code, and would advise 
    getting rid of it altogether. Does anyone know of any code that 
    uses the mimetypes module with strict set to False, where the 
    non-strict code path ever *actually* is executed?

But though these problems, which affect actual use of the code and
are therefore probably most important, are significant, they really
pale in comparison to the awful quality of implementation. I'll try
to briefly outline my understanding of how code flows in
mimetypes.py, and what the problems are. I haven't stepped through 
the code in a debugger, this is just from reading it, so I apologize 
in advance if I get something wrong. This is, however, some of the 
worst code I?ve seen in the standard library or anywhere else.

  * It defines __all__: I didn?t even realize __all__ could be used 
    for single-file modules (w/o submodules), but it definitely 
    shouldn?t be here. This specific __all__ oddly does not include 
    all of the documented variables and functions in the mimetypes 
    class. It?s not clear why someone calling import * here wouldn?t 
    want the bits not included.
        
  * It creates a _default_mime_types() function which declares a 
    bunch of global variables, and then immediately calls 
    _default_mime_types() below the definition. There is literally 
    no difference in result between this and just putting those 
    variables at the top level of the file, so I have no idea why 
    this function exists, except to make the code more confusing.
    
  * It allows command line usage: I don?t think this is necessary
    for a part of the standard library like this. There are better 
    tools for finding mime types from the command line which ship 
    with most operating systems.
    
  * Its API is pretty poorly designed. It offers 6 functions when 
    about 3 are needed, and it takes a couple reads-through of the 
    code to figure out exactly what any of them are supposed to do.
    
  * The operation is crazy: It defines a MimeTypes class which 
    actually stores the type mappings, but this class is designed to 
    be a singleton. The way that such a design is enforced is 
    through the use of the module-global 'init' function, which 
    makes an instance of the class, and then maps all of the 
    functions in the module global namespace to instance methods. 
    But confusingly, all such functions are also defined 
    independently of the init function, with definitions such as:
  
        def guess_type(url, strict=True):
            if not inited:
                init()
            return guess_type(url, strict)
    
    I?d be amazed if anyone could guess what that code was trying to 
    do. I did a double-take when I saw it.
    
    Of course, that return call is only ever reached the first time 
    this function is called, if init() has not happened yet. This 
    was all presumably done for lazy initialization, so that the 
    type information would only be loaded when needed. Needless to 
    say, there are more pythonic ways to accomplish such a goal.
    
    Oh, also, the other good one here is that it means that someone 
    who writes `from mimetypes import guess_types` gets something 
    different than someone who writes:
    `import mimetypes; guess_types = mimetypes.guess_types`. In the 
    former case, this wrapper function is saved as guess_type, which 
    each time just calls the (changed after init()) 
    mimetypes.guess_types function. This caused a performance 
    nightmare before March of this year, when there was no check for 
    `if not inited` before running init() (amazing!?).
    
  * Because the type datastore is set up to be a singleton, any time 
    init() is called in one section of code, it resets any types 
    which have been added manually: this means that if init() is 
    called by different pieces of code in the same python program, 
    they will interfere with each-others? type databases, and break 
    each-other. This is extremely fragile and, in my opinion, crazy. 
    It is hard for me to imagine any use case that would benefit 
    from this ability to clobber custom type mappings, and I very
    much doubt that any code calling the mimetypes module realizes 
    that the contract of the API is so flimsy by definition. In 
    practice, I would not advise consumers of this API to ever call 
    init() manually, or to ever add custom mime type mappings, 
    because they are setting themselves up for hard-to-track bugs 
    down the line.
    
  * The 'inited' flag is a documented part of the interface, in the
    standard library documentation. I cannot imagine any reason to 
    set this flag manually: setting it to false when it was true 
    will have no effect, because the top-level functions have 
    already been replaced by instance methods of the 'db' MimeTypes 
    instance. Setting it to true when it was false will make the 
    code just break outright.
      
  * In python 3, this has been changed a bit. There?s still an 
    inited flag, and it still in the docs, but now awful code from 
    above has been changed slightly, to:
      
        def guess_type(url, strict=True):
            if _db is None:
                init()
                return _db.guess_type(url, strict)
    
    Which is still embarrassingly confusing. On the upside, the 
    inited flag now does literally nothing, but remains defined, and 
    in the docs.
    
  * The 'types_map' and 'common_types' (for 'strict' and 
    'common' types, respectively) dictionaries are also a documented 
    part of the interface. When init() is called, a new MimeTypes 
    instance makes a (different) types_map which is a tuple of two 
    dictionaries, for 'strict' and 'common' types. Then this 
    instance reads the apache mime.types files and adds the types to 
    its pair of self.types_map dictionaries, and then after that 
    looks at the global types_map and common_types dictionaries and 
    adds *those* types to its self.types_map. Then at the end it 
    replaces the global types_map with self.types_map[True] and 
    replaces common_types with self.types_map[False]. Unfortunately, 
    while changing these dictionaries will have an effect on the 
    operation of the library, it will not update the types_map_inv 
    mapping, so inverse lookups will not behave as the changer 
    expects. If these dictionaries are going to remain documented, 
    the documentation should be clear to describe them as read only 
    to avoid very confusing bugs.
    
  * Speaking of these dictionaries, .copy() is called on those two 
    and a few other inside MimeTypes.__init__(), which happens every 
    time the global init() function is called, but then init() puts 
    the copies back in the global namespace, meaning that the 
    original is discarded. Basically the only reason for the .copy() 
    is to make sure that the correct updates are applied to the 
    apache mimetype defaults, but the code will gladly re-read all 
    of the apache files even after its mapped types are already in 
    these dictionaries, essentially making re-initializing a (very 
    expensive) no-op. All we?re doing is a lot of unnecessary extra 
    disk reads and memory allocations and deallocations. The only 
    time this has any effect is when a non-singleton MimeTypes 
    instance is created, as in the read_mime_types function.
    
  * And that read_mime_types function is a doozy. It tries to open a 
    filename, spits back None if there?s an IOError (instead of 
    raising the exception as it should), and then creates a new 
    MimeTypes instance (remember, this is identical to the singleton 
    MimeTypes instance because it starts itself from that one?s 
    mappings), adds any new types it finds in the file with that 
    name, and then returns the 'strict' types_map from it. I?m not 
    sure whether any sane user of this API would expect it to return 
    the existing type mappings *plus* the extra ones in the provided 
    filename, but I really can?t imagine this function ever being 
    particularly useful: it requires you are reading mime types in 
    apache format, but not the apache mime type files you already 
    looked at, and then the only way to find out what new mappings 
    were defined is to take the difference of the default mappings 
    with the result of the function.
    
  * The code itself, on a line-by-line basis, is unpythonic and 
    unnecessarily verbose, confusing, and slow. The code should be 
    rewritten to use python 2.3?2.6 features: even leaving its 
    functionality identical it could be cut to about half the number 
    of lines, and made clearer.

In case the above doesn?t make this clear: this code is extremely
confusing. Trying to read it has caused all the people around me to
look up as I shout "what the fuck??!" at the screen every few
minutes, as each new revelation gives another surprise. I?m not
convinced that I completely understand what the code does, because
it has been quite effectively obfuscated, but I understand enough to
want to throw the whole thing out, and start essentially from 
scratch.

So the question is, what should be done about this? I?d like to hear
how people use the mimetypes module, and how they expect it to work,
to figure out the sanest possible mostly-backwards-compatible
replacement which could be dropped in (ideally this would just allow
the use of default mimetypes and rip out the ability to alter the
default datastore: or is there some easy way to change this away
from a singleton without breaking code which calls these methods?),
and then extend that replacement to support a somewhat saner model
for anyone who actually wants to extend the set of mappings. My 
guess is that replacement code could actually fix subtle bugs in 
existing uses of this module, by people who had a sane expectation 
of how it was supposed to work.

At the very least, the parts about figuring out exactly which
exceptions to Apache?s set of default types are useful would be a
good idea, and I?d maybe even recommend including an up-to-date copy
of Apache?s mime.types file in the Python Standard Library, and then
only overriding its definitions for future versions of Apache (and
then overriding the combination of both of those with further
exceptions deemed useful for python, with comments explaining why
each exception), so that we?re not bothering to look up horribly
out-of-date types in multiple locations from Apache 1, 1.2, 1.3,
etc. I?d also recommend making the API for overriding definitions be
the same as the code used to declare the default overrides, because
as it is there are three ways do define types: a) in a mime.types
formatted file, b) in a python dictionary that gets initialized with
a confusing bit of code, and c) through the add_type function.

Does anyone else have thoughts about this, or maybe some good (it
had better be *really* good) explanations why this code is the way
it is? I'd be happy to try to rewrite it, but I think I?d need a bit
of help figuring out how to make the rewrite backwards-compatible.

Note: someone else has had fun with this module:
<http://lucumr.pocoo.org/2009/3/1/the-1000-speedup-or-the-stdlib-sucks>
<http://lucumr.pocoo.org/2009/7/24/singletons-and-their-problems-in-python>

Cheers,
Jacob Rus