Mailman 3 PEP 384 accepted - Python-Dev

PEP 384 accepted

Benjamin Peterson

2 Dec 2010 2 Dec '10

7:59 p.m.

Hi, Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday. -- Regards, Benjamin

Show replies by date

"Martin v. Löwis"

2 Dec 2 Dec

8:24 p.m.

...

Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday.

Thanks! will do (I'll also take into consideration the proposed changes). Regards, Martin

Tarek Ziadé

8:48 p.m.

On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. Löwis" wrote:

...

...
Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday.

Thanks! will do (I'll also take into consideration the proposed changes).

I did not get an answer to my last mail about distutils / distutils2

...

Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

-- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

9:02 p.m.

Am 02.12.2010 21:48, schrieb Tarek Ziadé:

...

On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. Löwis" wrote:

...
...
Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday.

Thanks! will do (I'll also take into consideration the proposed changes).

I did not get an answer to my last mail about distutils / distutils2

What was the question again, and whom did you want an answer from? Regards, Martin

Tarek Ziadé

9:11 p.m.

2010/12/2 "Martin v. Löwis" :

...

Am 02.12.2010 21:48, schrieb Tarek Ziadé:

...
On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. Löwis" wrote:

...
...
Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday.

Thanks! will do (I'll also take into consideration the proposed changes).

I did not get an answer to my last mail about distutils / distutils2

What was the question again, and whom did you want an answer from?

You can read it in the archives here: http://mail.python.org/pipermail/python-dev/2010-November/106138.html tldr: The question was "Why not implementing this in Distutils2 ?" Your answer was "No, PEP 3149 was accepted, I will do this in Distutils1" My answer was "Having an accepted PEP does not imply your code lands in the sdtlib (like PEP 376 and 345)" So the question still stands: "Why not implementing this in Distutils2 ?" Regards Tarek

...

Regards, Martin

-- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

9:19 p.m.

...

So the question still stands: "Why not implementing this in Distutils2 ?"

Because it then wouldn't be available in Python 3.2, which is the target release of the PEP. If that really causes too much pain, I'll refrain from making any changes to distutils; PEP 384 doesn't specify any changes, anyway. Regards, Martin

Tarek Ziadé

9:33 p.m.

2010/12/2 "Martin v. Löwis" :

...

...
So the question still stands: "Why not implementing this in Distutils2 ?"

Because it then wouldn't be available in Python 3.2, which is the target release of the PEP.

The exact feature I am mentioning is the ability to compile extensions with new options, so I am not sure which PEP is involved since distutils changes refer to PEP 384 in the other PEP. I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. Grouping all new features in the new version and keep Distutils1 in maintenance mode seems to make more sense to me, if we want to make Distutils die and push forward Distutils2 for its new features etc. Or we might get back into backward hell again :) So, I am +1 on a patch on distutils2 and -1 on de-freezing Distutils for any new feature.

...

If that really causes too much pain, I'll refrain from making any changes to distutils; PEP 384 doesn't specify any changes, anyway.

That would be awesome, and we can work on a patch for distutils2 to provide that abi option.

...

Regards, Martin

-- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

9:39 p.m.

...

I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me.

I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse. IIRC, it was really the incompatible changes that made people ask you to stop changing distutils. Regards, Martin

Tarek Ziadé

9:54 p.m.

2010/12/2 "Martin v. Löwis" :

...

...
I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me.

I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is).

"We" is the people at the last language summit. Sorry if I used such a vague word.

...

This freeze made the situation worse.

Can you extend on this and explains why it makes it worse ? If we (as you included) don't agree it's the best solution, I would not want to be pushed back to square one at the next summit.. I happily reverted all my changes last year when asked, and started to work on Distutils2. But I'll get out of steam if the direction changes again, with you stating that it makes the situation worse.

...

IIRC, it was really the incompatible changes that made people ask you to stop changing distutils.

Who is "people" ? Are you suggesting that we could have added all the new features in Distutils in the stdlib ? The decision was because we had a mix of: - incompatible changes in private parts -- and some packages where patching distutils internals - changes on public APIs behavior, whith a behavior that was not clearly documented and suggest to interpretation - some mistakes I made as well But that's what you would expect for a project that needs to evolve a lot. Thus the freezing. So how would you make the situation better, if not by doing the work in distutils2 ?

...

Regards, Martin

-- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

10:17 p.m.

...

...
This freeze made the situation worse.

Can you extend on this and explains why it makes it worse ?

Before the freeze, distutils was unmaintained (i.e. before you started maintaining it), but people who want to improve it gradually atleast could. Now gradual improvements are also banned, so it's not only unmaintained, but I can't even provide support for the PEP in Python that was just accepted.

...

...
IIRC, it was really the incompatible changes that made people ask you to stop changing distutils.

Who is "people" ? Are you suggesting that we could have added all the new features in Distutils in the stdlib ?

No, only the ones that didn't cause backwards incompatibilities, and broke existing packages.

...

But that's what you would expect for a project that needs to evolve a lot. Thus the freezing.

Instead of evolving a lot, and instead of freezing, I would have preferred "evolve a little".

...

So how would you make the situation better, if not by doing the work in distutils2 ?

Lift the freeze. I'm all for replacing distutils with distutils2, but I'm not sure whether you will declare distutils2 ready tomorrow, next year, or ten years from now. Regards, Martin

Tarek Ziadé

10:44 p.m.

2010/12/2 "Martin v. Löwis" :

...

...
...
This freeze made the situation worse.

Can you extend on this and explains why it makes it worse ?

Before the freeze, distutils was unmaintained (i.e. before you started maintaining it), but people who want to improve it gradually atleast could. Now gradual improvements are also banned, so it's not only unmaintained, but I can't even provide support for the PEP in Python that was just accepted.

...
...
IIRC, it was really the incompatible changes that made people ask you to stop changing distutils.

Who is "people" ? Are you suggesting that we could have added all the new features in Distutils in the stdlib ?

No, only the ones that didn't cause backwards incompatibilities, and broke existing packages.

This is impossible. I can point you to some third party project that can break if you touch some distutils internals, like setuptools. Setuptools also uses some privates global variables in some other modules in the stdlib FYI. The right answer was maybe back then: make setuptools and other projects evolve with distutils. But it did not happen. So we left the status quo and moved forward in distutils2. Because we knew distutils needed deeper changes anyways, and we knew setuptools was used everywhere and unfortunately not evolving at the same pace. (note: I am not blaming PJE or anyone when I say this -- the way distutils worked and was poorly maintained was the main reason)

...

...
But that's what you would expect for a project that needs to evolve a lot. Thus the freezing.

Instead of evolving a lot, and instead of freezing, I would have preferred "evolve a little".

...
So how would you make the situation better, if not by doing the work in distutils2 ?

Lift the freeze. I'm all for replacing distutils with distutils2, but I'm not sure whether you will declare distutils2 ready tomorrow, next year, or ten years from now.

Depends on what "ready" means. If by ready you mean it can be used to replace Distutils1 in a project, I declare Distutils2 ready for usage NOW. It's in alpha stage. I want a solid beta before Pycon. I would even remove Distutils from 3.x altogether at some point since setuptools is not Python 3 compatible, and just put distutils2. 3.3 sounds like a good target. Regards Tarek -- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

10:54 p.m.

...

...
No, only the ones that didn't cause backwards incompatibilities, and broke existing packages.

This is impossible. I can point you to some third party project that can break if you touch some distutils internals, like setuptools. Setuptools also uses some privates global variables in some other modules in the stdlib FYI.

So what would break if Extension accepted an abi= keyword parameter?

...

...
Lift the freeze. I'm all for replacing distutils with distutils2, but I'm not sure whether you will declare distutils2 ready tomorrow, next year, or ten years from now.

Depends on what "ready" means.

Included in Python, so that changes become possible again.

...

If by ready you mean it can be used to replace Distutils1 in a project, I declare Distutils2 ready for usage NOW. It's in alpha stage. I want a solid beta before Pycon.

I would even remove Distutils from 3.x altogether at some point since setuptools is not Python 3 compatible, and just put distutils2.

3.3 sounds like a good target.

So will distuils2 be released before that? If so, when? Regards, Martin

Tarek Ziadé

11:25 p.m.

2010/12/2 "Martin v. Löwis" :

...

...
...
No, only the ones that didn't cause backwards incompatibilities, and broke existing packages.

This is impossible. I can point you to some third party project that can break if you touch some distutils internals, like setuptools. Setuptools also uses some privates global variables in some other modules in the stdlib FYI.

So what would break if Extension accepted an abi= keyword parameter?

I suppose you have code behind this, that will be in build_ext and in the compilers. So you will need to try out ALL projects out there that customize build_ext, like numpy or setuptools, etc, But you won't be able to try out all projects because they are not listed somewhere. For starters, the Extension class is replaced by another one in setuptools, that patches the constructor if Pyrex is installed, which is unlikely I guess, so no big deal. But you will also get a replaced version of the Distribution class that uses a private method from distutils, and another version of build_ext with custom compiling flags. Now depending on how you do your thing it could work if you are careful at doing things on the top of setuptools. And then, if numpy.distutils is installed, it relies on distutils build_ext and tries to rely on setuptools one's too, so it gets in the mix of the patched classes, and you get an horrible mix and possible bad interactions. So I am not saying it's impossible to add the feature, but it is impossible to be sure nothing gets broken in third party. So the freeze seems wise indeed

...

...
...
Lift the freeze. I'm all for replacing distutils with distutils2, but I'm not sure whether you will declare distutils2 ready tomorrow, next year, or ten years from now.

Depends on what "ready" means.

Included in Python, so that changes become possible again.

...
If by ready you mean it can be used to replace Distutils1 in a project, I declare Distutils2 ready for usage NOW. It's in alpha stage. I want a solid beta before Pycon.

I would even remove Distutils from 3.x altogether at some point since setuptools is not Python 3 compatible, and just put distutils2.

3.3 sounds like a good target.

So will distuils2 be released before that? If so, when?

An alpha is already released. A beta will be released for Pycon (I need it for my talk :) ) Then hopefully the final before 3.2

...

Regards, Martin

-- Tarek Ziadé | http://ziade.org

"Martin v. Löwis"

11:42 p.m.

...

An alpha is already released. A beta will be released for Pycon (I need it for my talk :) ) Then hopefully the final before 3.2

Ok, that's promising. Regards, Martin

Matthias Klose

3 Dec 3 Dec

1:48 a.m.

On 03.12.2010 00:25, Tarek Ziadé wrote:

...

2010/12/2 "Martin v. Löwis":

...
...
...
No, only the ones that didn't cause backwards incompatibilities, and broke existing packages.

This is impossible. I can point you to some third party project that can break if you touch some distutils internals, like setuptools. Setuptools also uses some privates global variables in some other modules in the stdlib FYI.

So what would break if Extension accepted an abi= keyword parameter?

I suppose you have code behind this, that will be in build_ext and in the compilers. So you will need to try out ALL projects out there that customize build_ext, like numpy or setuptools, etc, But you won't be able to try out all projects because they are not listed somewhere.

is this necessary? are all these projects known to work with 3.2, without having changes compared to 3.1 *without* this pep? hardly ... how many extensions will use this restricted api at all? Is it a legitimate solution to back up building an extension in the "default" mode? even without having any changes in distutils it would make sense to know if an extension can be built with the restricted ABI, so maybe it is better to defer any changes to the extension soname, and provide a check for an extension if it conforms to the restricted ABI, even if the extension still uses the python version specific soname. I did not mean to block this pep by choosing any installation names. Matthias

Éric Araujo

2:14 a.m.

...

even without having any changes in distutils it would make sense to know if an extension can be built with the restricted ABI, so maybe it is better to defer any changes to the extension soname, and provide a check for an extension if it conforms to the restricted ABI, even if the extension still uses the python version specific soname.

Python’s setup.py has an example in Martin’s branch: ext = Extension('xxlimited', ['xxlimited.c'], define_macros=[('Py_LIMITED_API', 1)]) http://codereview.appspot.com/3262043/patch/1/68 This is possible with today’s distutils. I don’t know if it’s enough to build stable-ABI-conformant extension modules. Regards

"Martin v. Löwis"

7:16 a.m.

...

Python’s setup.py has an example in Martin’s branch:

ext = Extension('xxlimited', ['xxlimited.c'], define_macros=[('Py_LIMITED_API', 1)])

http://codereview.appspot.com/3262043/patch/1/68

This is possible with today’s distutils. I don’t know if it’s enough to build stable-ABI-conformant extension modules.

It is. However, there is also the proposal that they use an ABI tag in the SO name; having that generated automatically would require a distutils change. Regards, Martin

Éric Araujo

5:32 a.m.

Le 02/12/2010 23:17, Martin v. Löwis a écrit :

...

Before the freeze, distutils was unmaintained (i.e. before you started maintaining it), but people who want to improve it gradually atleast could. Now gradual improvements are also banned, so it's not only unmaintained, but I can't even provide support for the PEP in Python that was just accepted.

I wonder what your definition of “unmaintained” is. Tarek has been fixing bugs for two years, and recently I have been made a committer to assist him. It’s true that I’ve not been as active as I would have liked*, but I did fix some bugs, as I think you know, given that you’ve helped me in some reports. Sure, distutils is not as well-maintained as other modules, but a dozen bugs have been fixed by five or six of us since the revert. I do feel responsible for all 116 remaining bugs, and intend to address all of them. * This is partly normal, since I had warned before I was accepted as a committer that my time would be scarce for a year, partly due to the fact that I also do bug triage, doc work and patch reviews, and partly due to some personal problems with focusing. On the matter of freeze exceptions, there have been two: - reading the makefile with surogateescape error handler so that python can build with an ASCII locale in a non-ASCII path (haypo, #6011) - handle soabiflags (barry, #9807). I took part in the discussion before those changes and did not object to them: they are very small changes that enable a new feature of Python 3.2. Maybe I should have requested Tarek’s approval for those changes; he knows better than me how third parties may break because of changes that don’t seem to break anything. Regards

"Martin v. Löwis"

7:31 a.m.

...

I wonder what your definition of “unmaintained” is.

In this specific case: doesn't get feature requests acted upon. I'm well aware that you are fixing bugs, and that is appreciated.

...

Sure, distutils is not as well-maintained as other modules, but a dozen bugs have been fixed by five or six of us since the revert. I do feel responsible for all 116 remaining bugs, and intend to address all of them.

But if the resolution of the bug would require a new feature, your answer will be "this is going to be fixed in distutils2 (if at all), it's out of scope for distutils". Before, if the submitter contributed a patch, the patch was just unreviewed for a long time, unless one of the committers picked it up. Now, the patch will be rejected, which I consider worse - because the patch is not being rejected on its own merits, but just because of a policy decision to not improve distutils anymore. For example, I keep running into the issue that distutils doesn't currently support parallel builds. I have been pondering supporting "-j" for building extensions, using both unbounded "-j" and the GNU make style -jN build server. However, I know that the patch will be rejected, so I don't even start working on it.

...

On the matter of freeze exceptions, there have been two: - reading the makefile with surogateescape error handler so that python can build with an ASCII locale in a non-ASCII path (haypo, #6011) - handle soabiflags (barry, #9807). I took part in the discussion before those changes and did not object to them: they are very small changes that enable a new feature of Python 3.2. Maybe I should have requested Tarek’s approval for those changes; he knows better than me how third parties may break because of changes that don’t seem to break anything.

I see. Now, I'd claim that the reasoning as to why an abi= parameter on Extension may break things also applies to the soabiflags: to support soabiflags, the INSTALL_SCHEMES syntax was modified. If the install command is subclassed, that could lead to funny interactions, e.g. where the subclass fails to put abiflags into config_vars. IIUC, subst_vars will then eventually raise a ValueError. I'm not saying that this is a likely scenario - only that the reasoning "if a change can possibly affect existing code, it should not be made" applies to essentially any change. So if you want to avoid breaking things with certainty, not even bug fixes would be acceptable. Regards, Martin

Éric Araujo

12:27 p.m.

Le 03/12/2010 08:31, Martin v. Löwis a écrit :

...

...
I wonder what your definition of “unmaintained” is. In this specific case: doesn't get feature requests acted upon. Thanks for clarifying. I think that’s a stretch, but I see your meaning now.

...

...
Sure, distutils is not as well-maintained as other modules, but a dozen bugs have been fixed by five or six of us since the revert. I do feel responsible for all 116 remaining bugs, and intend to address all of them.

But if the resolution of the bug would require a new feature, your answer will be "this is going to be fixed in distutils2 (if at all), it's out of scope for distutils". Before, if the submitter contributed a patch, the patch was just unreviewed for a long time, unless one of the committers picked it up. Now, the patch will be rejected, which I consider worse - because the patch is not being rejected on its own merits, but just because of a policy decision to not improve distutils anymore. The patch would not be rejected, but assigned to a different version. It‘s not different than replying to an old bug with a patch for Python 2.5 and requesting that it be updated for py3k. It’s also not uncommon to have another contributor or a core dev updating the patch if the original poster does not reply.

...

For example, I keep running into the issue that distutils doesn't currently support parallel builds. I have been pondering supporting "-j" for building extensions, using both unbounded "-j" and the GNU make style -jN build server. However, I know that the patch will be rejected, so I don't even start working on it. This would be a very useful feature for distutils2.

...

...
On the matter of freeze exceptions, there have been two: [snip] I see. Now, I'd claim that the reasoning as to why an abi= parameter on Extension may break things also applies to the soabiflags: to support soabiflags, the INSTALL_SCHEMES syntax was modified. If the install command is subclassed, that could lead to funny interactions, e.g. where the subclass fails to put abiflags into config_vars. IIUC, subst_vars will then eventually raise a ValueError. This is a concern.

...

I'm not saying that this is a likely scenario - only that the reasoning "if a change can possibly affect existing code, it should not be made" applies to essentially any change. So if you want to avoid breaking things with certainty, not even bug fixes would be acceptable. If we wanted 100% certainty, yes.

Regards

"Martin v. Löwis"

10:39 p.m.

...

...
For example, I keep running into the issue that distutils doesn't currently support parallel builds. I have been pondering supporting "-j" for building extensions, using both unbounded "-j" and the GNU make style -jN build server. However, I know that the patch will be rejected, so I don't even start working on it. This would be a very useful feature for distutils2.

But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. Regards, Martin

Éric Araujo

10:48 p.m.

...

But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

(BTW: congratulations on having PEP 384 accepted and merged.) Regards

"Martin v. Löwis"

10:52 p.m.

Am 03.12.2010 23:48, schrieb Éric Araujo:

...

...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it. Regards, Martin

James Y Knight

11:11 p.m.

On Dec 3, 2010, at 5:52 PM, Martin v. Löwis wrote:

...

Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. James

Antoine Pitrou

4 Dec 4 Dec

11:27 a.m.

On Fri, 3 Dec 2010 18:11:57 -0500 James Y Knight wrote:

...

On Dec 3, 2010, at 5:52 PM, Martin v. Löwis wrote:

...
Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then.

+1 from me. Antoine.

Tarek Ziadé

4:55 p.m.

On Sat, Dec 4, 2010 at 12:27 PM, Antoine Pitrou wrote:

...

On Fri, 3 Dec 2010 18:11:57 -0500 James Y Knight wrote:

...
On Dec 3, 2010, at 5:52 PM, Martin v. Löwis wrote:

...
Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then.

+1 from me.

+1 too. Just to clarify my position in a few sentences: - I was told not to progressively change distutils - the PEPs for my changes were accepted under the condition that the changes would be made in distutils2 - PEP 384 is yet another accepted PEP, why the rule would change - because it's a small feature ? - I have tons of small features I had to revert and push back in distutils2 If Martin changes make it into Distutils1, I am going to ask that a collection of small features that I have should go there too, because they "should not break things" Regards Tarek -- Tarek Ziadé | http://ziade.org

Éric Araujo

5:49 p.m.

Le 04/12/2010 17:55, Tarek Ziadé a écrit :

...

On Sat, Dec 4, 2010 at 12:27 PM, Antoine Pitrou wrote:

...
...
It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then.

+1 from me.

+1 too.

+1, and I take responsibility to propose a merge plan to python-dev after 3.2 is released and carry it out. Regards

Michael Foord

11:10 p.m.

On 04/12/2010 11:27, Antoine Pitrou wrote:

...

On Fri, 3 Dec 2010 18:11:57 -0500 James Y Knight wrote:

...
On Dec 3, 2010, at 5:52 PM, Martin v. Löwis wrote:

...
Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win? It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it. It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. +1 from me.

And from me. Michael

...

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Terry Reedy

3 Dec 3 Dec

11:35 p.m.

On 12/3/2010 5:52 PM, "Martin v. Löwis" wrote:

...

Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

Éric, I have the impression from Tarek and you together that D2 is still in alpha only because it is not feature frozen and that it is as capable and stable as D1. I do not know what Martin means by 'integrate' (other than that he be able to use it to build Python), but if my first sentence is true, I cannot help but wonder whether a snapshot of D2 could be included with 3.2, perhaps as '_distribute2' (note leading underscore) at least for Python's use. The doc, if any, could just say 'Development snapshot of D2.a4 (or whatever) for building Python. Other uses should get the latest public release from PyPI." -- Terry Jan Reedy

"Martin v. Löwis"

11:46 p.m.

Am 04.12.2010 00:35, schrieb Terry Reedy:

...

On 12/3/2010 5:52 PM, "Martin v. Löwis" wrote:

...
Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

Éric, I have the impression from Tarek and you together that D2 is still in alpha only because it is not feature frozen and that it is as capable and stable as D1. I do not know what Martin means by 'integrate' (other than that he be able to use it to build Python)

That the master copy of the source code is in the Python source repository. Regards, Martin

Terry Reedy

4 Dec 4 Dec

midnight

On 12/3/2010 6:46 PM, "Martin v. Löwis" wrote:

...

...
and stable as D1. I do not know what Martin means by 'integrate' (other than that he be able to use it to build Python)

That the master copy of the source code is in the Python source repository.

Is a separate branch acceptible, as long as you can commit changes? Does the meaning of 'repository' change any with the hg switch? -- Terry Jan Reedy

"Martin v. Löwis"

12:41 a.m.

Am 04.12.2010 01:00, schrieb Terry Reedy:

...

On 12/3/2010 6:46 PM, "Martin v. Löwis" wrote:

...
...
and stable as D1. I do not know what Martin means by 'integrate' (other than that he be able to use it to build Python)

That the master copy of the source code is in the Python source repository.

Is a separate branch acceptible, as long as you can commit changes?

No. I want the buildbots be able to use the code, in the Python trunk.

...

Does the meaning of 'repository' change any with the hg switch?

No. The code would need to be pushed to the master repository, so that the buildbots can fetch it from there. In short, I'm not very interested in contributing to a tool that has no users (and is the fifths of its kind). Regards, Martin

Toshio Kuratomi

4:42 p.m.

On Fri, Dec 03, 2010 at 11:52:41PM +0100, "Martin v. Löwis" wrote:

...

Am 03.12.2010 23:48, schrieb Éric Araujo:

...
...
But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. You can’t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn’t that a win?

It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it.

At the language summit it was proposed and seemed generally accepted (maybe I took silence as consent... it's been almost a year now) that bold new modules (and bold rewrites of existing modules since it fell out of the distutils/2 discussion) should get implemented in a module on pypi before being merged into the python stdlib. If you wouldn't want to work on any of those modules until they were actually integrated into Python, it sounds like you disagree with that as a general practice? -Toshio

"Martin v. Löwis"

7:43 p.m.

...

At the language summit it was proposed and seemed generally accepted (maybe I took silence as consent... it's been almost a year now) that bold new modules (and bold rewrites of existing modules since it fell out of the distutils/2 discussion) should get implemented in a module on pypi before being merged into the python stdlib. If you wouldn't want to work on any of those modules until they were actually integrated into Python, it sounds like you disagree with that as a general practice?

No - I just don't feel responsible for projects outside of CPython (unless I have an interest in the project as a user, i.e. to scratch my own itches). I have no use for a distutils2 package whatsoever - to install Python packages, I would either use the builtin distutils (preferred), or setuptools, or distribute (if the package author requires setuptools). I agree that new functionality should stand on its own merits first. And if it fails that test (i.e. it doesn't get much attraction from users when stand-alone), it shouldn't be included into Python at all. Now, distutils2 is different: it's *not* new functionality. So perhaps yes: I disagree with the principle that bold rewrites should be developed independently. Such work would be much better carried out in a branch - it will never stand on its own. Regards, Martin

Tarek Ziadé

6 Dec 6 Dec

9:42 a.m.

On Sat, Dec 4, 2010 at 8:43 PM, "Martin v. Löwis" wrote: ...

...

Now, distutils2 is different: it's *not* new functionality. So perhaps yes: I disagree with the principle that bold rewrites should be developed independently. Such work would be much better carried out in a branch - it will never stand on its own.

So we had several +1 on James' idea to bring back distutils2 to the trunk after 3.2 final, and it seems that you are not against this idea. So we will do this unless you object Regards Tarek -- Tarek Ziadé | http://ziade.org

Michael Foord

2 Dec 2 Dec

9:54 p.m.

On 02/12/2010 21:39, "Martin v. Löwis" wrote:

...

...
I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse.

What situation worse? "We" certainly did ask Tarek to become bdfl of distutils and fix/improve it (at a language summit 2 years ago). "We" then asked him to revert distutils and do the work in a new package instead of inside distutils (at the language summit this year). I would perhaps argue for a case by case exception on PEPs that *required* distutils support that are being accepted and implemented prior to distutils2 moving into the standard library. It doesn't sound like your changes are *required* by the PEP though. As I recall Tarek thought it was a bad idea to freeze distutils as well, but "we" insisted. :-)

...

IIRC, it was really the incompatible changes that made people ask you to stop changing distutils.

Which included virtually any change to even private APIs. Given the issues freezing the distutils APIs except for essential bugfixes is a reasonable response. I don't know of any situation it has made worse. Things are getting very much better, but happening in distutils2 not distutils. All the best, Michael Foord

...

Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

"Martin v. Löwis"

10:21 p.m.

Am 02.12.2010 22:54, schrieb Michael Foord:

...

On 02/12/2010 21:39, "Martin v. Löwis" wrote:

...
...
I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse.

What situation worse?

The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes.

...

I would perhaps argue for a case by case exception on PEPs that *required* distutils support that are being accepted and implemented prior to distutils2 moving into the standard library. It doesn't sound like your changes are *required* by the PEP though.

Well, the PEP 384 text in PEP 3149 specifies a change. It's not clear whether this change was accepted when PEP 3149 was accepted, or whether it was accepted when PEP 384 was accepted, or whether it was not accepted at all, or whether it was just proposed. In any case, without the change, you won't naturally get extension modules that use the abi3 tag proposed in 3149. Regards, Martin

Tarek Ziadé

10:23 p.m.

2010/12/2 "Martin v. Löwis" :

...

Am 02.12.2010 22:54, schrieb Michael Foord:

...
On 02/12/2010 21:39, "Martin v. Löwis" wrote:

...
...
I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse.

What situation worse?

The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes.

I welcome those changes in Distutils2. That's the whole point.

"Martin v. Löwis"

10:29 p.m.

...

...
The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes.

I welcome those changes in Distutils2. That's the whole point.

That would be useful if there was a clear vision of when distutils2 will be released. Please understand that I'm not blaming you for not releasing it (it *is* too much for a single person), but please understand that it's also not helpful to submit changes to a codebase that is not going to be released in a foreseeable future. Regards, Martin

Tarek Ziadé

10:53 p.m.

2010/12/2 "Martin v. Löwis" :

...

...
...
The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes.

I welcome those changes in Distutils2. That's the whole point.

That would be useful if there was a clear vision of when distutils2 will be released. Please understand that I'm not blaming you for not releasing it (it *is* too much for a single person), but please understand that it's also not helpful to submit changes to a codebase that is not going to be released in a foreseeable future.

I know you're not blaming me. Distutils 2 alpha3 is currently released and available at PyPI. I use it in some of my professional projects FWIW. alpha4 was postponed but should be out this month. It contains major features, people from the GSOC worked on. The initial roadmap was to have a final by the time 3.2 final is out, but that'll be too short. So the target is to have a beta release for Pycon, and to sync the final release with 3.3, with lots of feedback in the meantime hopefully, and people using it from 2.4 onward.

...

Regards, Martin

-- Tarek Ziadé | http://ziade.org

Antoine Pitrou

10:51 p.m.

On Thu, 02 Dec 2010 23:21:25 +0100 "Martin v. Löwis" wrote:

...

Am 02.12.2010 22:54, schrieb Michael Foord:

...
On 02/12/2010 21:39, "Martin v. Löwis" wrote:

...
...
I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse.

What situation worse?

The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes.

I think distutils is simply a bugfix branch for distutils2. Similarly as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we commit improvements to distutils. (and I think that's how Guido wanted it anyway) Regards Antoine.

"Martin v. Löwis"

11:01 p.m.

...

I think distutils is simply a bugfix branch for distutils2. Similarly as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we commit improvements to distutils.

It's different, though, in the sense that Python has a release schedule and multiple committers working on it, and that it normally gets released even if some changes don't get included in a specific release yet. All this seems not to be true for distutils2. So my motivation to contribute changes to it is *much* lower than my desire to contribute to distutils, and it is also provably lower than my motivation to contribute to distribute (say). I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community. Regards, Martin

Michael Foord

11:23 p.m.

On 02/12/2010 23:01, "Martin v. Löwis" wrote:

...

...
[snip...] I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community.

The very best hope of resolving that particular problem is distutils2. :-) distutils2 is *already* available to the Python community, and whether or not there is a fixed release date it will have betas and then a 1.0 release in the foreseeable future. The team working on it has made an enormous amount of progress. We're much better off as a development community putting our support and energy into distutils2 rather than pining for evolution of distutils. Michael

...

Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

Amaury Forgeot d'Arc

11:51 p.m.

Hi, 2010/12/3 Michael Foord

...

On 02/12/2010 23:01, "Martin v. Löwis" wrote:

...
[snip...]

...
I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community.

The very best hope of resolving that particular problem is distutils2. :-)

distutils2 is *already* available to the Python community, and whether or not there is a fixed release date it will have betas and then a 1.0 release in the foreseeable future. The team working on it has made an enormous amount of progress. We're much better off as a development community putting our support and energy into distutils2 rather than pining for evolution of distutils.

Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; they change the paths and filenames used by python. Either we modify distutils to comply with the new names, or defer these PEPs until distutils2 is ready. -- Amaury Forgeot d'Arc

Michael Foord

3 Dec 3 Dec

12:35 a.m.

On 02/12/2010 23:51, Amaury Forgeot d'Arc wrote:

...

Hi,

2010/12/3 Michael Foord mailto:fuzzyman@voidspace.org.uk>

On 02/12/2010 23:01, "Martin v. Löwis" wrote:

[snip...]

I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community.

The very best hope of resolving that particular problem is distutils2. :-)

distutils2 is *already* available to the Python community, and whether or not there is a fixed release date it will have betas and then a 1.0 release in the foreseeable future. The team working on it has made an enormous amount of progress. We're much better off as a development community putting our support and energy into distutils2 rather than pining for evolution of distutils.

Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; they change the paths and filenames used by python. Either we modify distutils to comply with the new names, or defer these PEPs until distutils2 is ready.

Or put support for them into distutils2 now? Michael

...

-- Amaury Forgeot d'Arc

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Barry Warsaw

1:07 a.m.

On Dec 03, 2010, at 12:51 AM, Amaury Forgeot d'Arc wrote:

...

Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; they change the paths and filenames used by python. Either we modify distutils to comply with the new names, or defer these PEPs until distutils2 is ready.

I do not think it would be a good idea to revert PEP 3149. -Barry

Tarek Ziadé

2 Dec 2 Dec

11:50 p.m.

On Fri, Dec 3, 2010 at 12:01 AM, "Martin v. Löwis" wrote:

...

...
I think distutils is simply a bugfix branch for distutils2. Similarly as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we commit improvements to distutils.

It's different, though, in the sense that Python has a release schedule and multiple committers working on it, and that it normally gets released even if some changes don't get included in a specific release yet.

All this seems not to be true for distutils2.

We have 3 or 4 regular contributors. That's not a lot for sure.

...

So my motivation to contribute changes to it is *much* lower than my desire to contribute to distutils, and it is also provably lower than my motivation to contribute to distribute (say). I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community.

I am not trying to motivate you to contribute to Distutils2. I am trying to make sure we are all on the same page for what's good for Python. So if we work in Distutils2 and you work in Distutils saying publicly that you don't want to contribute to Distutils2, that's a total nonsense. We took some decisions, and you want to go against them. So I want to have a consensus here for the packaging eco-system and make sure we are still on track. I am sorry if you get tired of it, but I don't want to be told at the next summit: sorry Tarek, now we need to do changes little by little in distutils1

Barry Warsaw

10:54 p.m.

On Dec 02, 2010, at 11:21 PM, Martin v. Löwis wrote:

...

Well, the PEP 384 text in PEP 3149 specifies a change. It's not clear whether this change was accepted when PEP 3149 was accepted, or whether it was accepted when PEP 384 was accepted, or whether it was not accepted at all, or whether it was just proposed.

From my point of view, the PEP 3149 text is just a proposal. It leaves the final decision to PEP 384, but tries to address some of the issues raised during the PEP 3149 discussion. I think it is within PEP 384's scope to make the final decisions about it.

...

In any case, without the change, you won't naturally get extension modules that use the abi3 tag proposed in 3149.

I would favor changing distutils, if it can be done in a way that reasonably preserves backward compatibility. I suppose it's impossible to know all the ways 3rd party code has reached into distutils, but I think you can make fairly good judgements about whether a change is backward compatible or not. -Barry

"Martin v. Löwis"

11:38 p.m.

...

From my point of view, the PEP 3149 text is just a proposal. It leaves the final decision to PEP 384, but tries to address some of the issues raised during the PEP 3149 discussion. I think it is within PEP 384's scope to make the final decisions about it.

Ok, then it looks like there just won't be any support for module tagging of ABI-conforming modules. It might be possible to support something like this in the import code, but I would consider this pointless without accompanying distutils support. Then, by default, the modules just use the ABI tag that distutils assigns to them by default. It's interesting to note that #9807 got into distutils despite it being frozen (but this is not about ABI tags, right - so does distutils in 3.2 actually assign any ABI tag at all?)

...

I would favor changing distutils, if it can be done in a way that reasonably preserves backward compatibility.

It seems this is right out for policy reasons. Regards, Martin

Stefan Behnel

4 Dec 4 Dec

8:35 a.m.

"Martin v. Löwis", 02.12.2010 21:24:

...

...
Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday.

Thanks! will do (I'll also take into consideration the proposed changes).

Note that the PEP states under "Signature Changes": """ A number of functions currently expect a specific struct, even though callers typically have PyObject* available. These have been changed to expect PyObject* as the parameter; this will cause warnings in applications that currently explicitly cast to the parameter type. """ That's not true for C++ compilation where I now get an error - 3 failing tests in Cython's test suite tonight due to the new PyEval_EvalCode() signature. I'm not really complaining (the API fixes are long overdue), just leaving a comment that what a compiler considers a warning or error pretty much depends on compiler, platform and configuration. So the above statement can be considered euphemistic. Stefan

"Martin v. Löwis"

8:49 a.m.

...

I'm not really complaining (the API fixes are long overdue), just leaving a comment that what a compiler considers a warning or error pretty much depends on compiler, platform and configuration.

No no no. It does *not* depend on compiler, platform, or configuration. It *only* depends on the programming language. So for C++, it's an API change; for C, it's not. Still, I can point that out. Regards, Martin

4883

Age (days ago)

4887

Last active (days ago)

List overview

Download

50 comments

13 participants

participants (13)

"Martin v. Löwis"
Amaury Forgeot d'Arc
Antoine Pitrou
Barry Warsaw
Benjamin Peterson
James Y Knight
Matthias Klose
Michael Foord
Stefan Behnel
Tarek Ziadé
Terry Reedy
Toshio Kuratomi
Éric Araujo