Mailman 3 PEP 385 progress report - Python-Dev

PEP 385 progress report

older
3.1 and 2.7 break format() when...

Dirkjan Ochtman

7 Feb 2010 7 Feb '10

1:35 p.m.

It's been a long time! So for the past few weeks, Mercurial crew member Patrick Mezard has been hunting for the ugly bug in hgsubversion that I'd previously been looking at, and it finally got fixed. A new bug popped up, but then we managed to fix that, too (thanks to the PSF for partially funding our sprint, it was very succesful!). In a joyous moment, I nagged Augie Fackler to actually put a hgsubversion release out there so hopefully more people can start using it, so we now have that, too. Another sponsor for our sprint was Logilab (who provided their brand new office for us to work in), and one of their employees, Andre Espaze, fortunately wanted to help out and managed to write up a patch for the sys.mercurial attribute (now in the pymigr repo). In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP. As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are). The current version of the repository (latest SVN revision is 78055, clone it from hg.python.org) weighs in at about 1.4G, but still needs branch pruning (which will be my primary focus for the coming few weeks). The good part about it now being a year later than, well, last year is that named branches are much more solid than before, and so I feel much better about using those for Python's release branches. Any questions and/or concerns? I will also be at PyCon; I'll be doing a more advanced talk on Mercurial internals on Sunday but I'd also be happy to do some handholding or introductory stuff in an open space. If there's anyone who'd like help converting their SVN repository, I might be able to help there too (during the sprints). For other conversions, I know for a fact that an expert in CVS conversions will be there. Cheers, Dirkjan

Show replies by thread

Benjamin Peterson

7 Feb 7 Feb

1:51 p.m.

2010/2/7 Dirkjan Ochtman :

...

It's been a long time!

Thank you very much for staying on this task! I'm still excited.

...

In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP.

How about a week after, so we have more time to adjust release procedures?

...

Any questions and/or concerns?

Will you do test conversions of the sandbox projects, too? Also I think we should have some document (perhaps the dev FAQ) explaining exactly how to do common tasks in mercurial. For example - A bug fix, which needs to be in 4 branches. - A bug fix, which only belongs in 2.7 and 2.6 or 3.2 and 3.1. - Which way do we merge (What's a subset of what?) -- Regards, Benjamin

Mark Hammond

1:58 p.m.

Hi Dirkjan, On 8/02/2010 8:35 AM, Dirkjan Ochtman wrote: ...

...

In fact, a few weeks ago I talked to Brett and we figured that we should probably pin down a deadline. We discussed aiming at May 1, and at this time I think that should be feasible. That also seems to coincide with the release of 2.7b2, though, so maybe we need to do it one week later (or sooner?). Anyway, we figured that a weekend would probably be a good time. If we manage to find a good date, I'll put it in the PEP.

Isn't setting a date premature while outstanding issues remain without a timetable for their resolution?

...

As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are).

See http://mercurial.selenic.com/wiki/EOLTranslationPlan#TODO - of particular note: * There are transient errors in the tests which Martin is yet to identify. These tests do not require windows to reproduce or fix. * The mercurial tests do not run on Windows. Given the above, most sane Windows developers would hold off on "live" testing of the extension until at least the first issue is resolved - but the second issue makes it very difficult for them to help resolve that. Cheers, Mark

Dirkjan Ochtman

8 Feb 8 Feb

9:14 a.m.

On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...

How about a week after, so we have more time to adjust release procedures?

Sounds fine to me.

...

Will you do test conversions of the sandbox projects, too?

Got any particular projects in mind?

...

Also I think we should have some document (perhaps the dev FAQ) explaining exactly how to do common tasks in mercurial. For example - A bug fix, which needs to be in 4 branches. - A bug fix, which only belongs in 2.7 and 2.6 or 3.2 and 3.1. - Which way do we merge (What's a subset of what?)

Yes, writing lots of docs is part of the plan. Cheers, Dirkjan

Dirkjan Ochtman

9:24 a.m.

On Sun, Feb 7, 2010 at 22:58, Mark Hammond wrote:

...

Isn't setting a date premature while outstanding issues remain without a timetable for their resolution?

If we set a date, that would imply a timetable for their resolution.

...

See http://mercurial.selenic.com/wiki/EOLTranslationPlan#TODO - of particular note:

* There are transient errors in the tests which Martin is yet to identify. These tests do not require windows to reproduce or fix.

* The mercurial tests do not run on Windows.

Given the above, most sane Windows developers would hold off on "live" testing of the extension until at least the first issue is resolved - but the second issue makes it very difficult for them to help resolve that.

The Mercurial tests can actually run on Windows -- and I've updated the page to that effect. They require something called pysh, though. I've also asked Patrick Mezard to include the eol extension in his nightly test run on Windows. I guess since some of the test errors do not require Windows to reproduce or fix, I'd invite anyone to jump in and help fix these issues. Cheers, Dirkjan

Benjamin Peterson

5:11 p.m.

2010/2/8 Dirkjan Ochtman :

...

On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...
Will you do test conversions of the sandbox projects, too?

Got any particular projects in mind?

2to3. -- Regards, Benjamin

"Martin v. Löwis"

7:39 p.m.

Benjamin Peterson wrote:

...

2010/2/8 Dirkjan Ochtman :

...
On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...
Will you do test conversions of the sandbox projects, too? Got any particular projects in mind?

2to3.

Does Mercurial even support merge tracking the way we are doing it for 2to3 right now? Regards, Martin

Benjamin Peterson

7:47 p.m.

2010/2/8 "Martin v. Löwis" :

...

Benjamin Peterson wrote:

...
2010/2/8 Dirkjan Ochtman :

...
On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...
Will you do test conversions of the sandbox projects, too? Got any particular projects in mind?

2to3.

Does Mercurial even support merge tracking the way we are doing it for 2to3 right now?

I don't believe so. My plan was to manually sync updates or use subrepos. -- Regards, Benjamin

Dirkjan Ochtman

9 Feb 9 Feb

2:26 a.m.

On Tue, Feb 9, 2010 at 04:47, Benjamin Peterson wrote:

...

I don't believe so. My plan was to manually sync updates or use subrepos.

Using subrepos should work well for this. It turned out that my local copy of the Subversion repository contained the Python dir only, so I'm now syncing a full copy so that I can convert other parts. I believe 2to3 might be a little tricky because it was moved at some point, but I can look at getting that right (and this will help in converting other parts of the larger Python repository). Cheers, Dirkjan

Benjamin Peterson

5:03 p.m.

2010/2/9 Dirkjan Ochtman :

...

On Tue, Feb 9, 2010 at 04:47, Benjamin Peterson wrote:

...
I don't believe so. My plan was to manually sync updates or use subrepos.

Using subrepos should work well for this.

Excellent.

...

It turned out that my local copy of the Subversion repository contained the Python dir only, so I'm now syncing a full copy so that I can convert other parts. I believe 2to3 might be a little tricky because it was moved at some point, but I can look at getting that right (and this will help in converting other parts of the larger Python repository).

What do you mean by moved? I don't it has ever moved around in the sandbox. -- Regards, Benjamin

Dirkjan Ochtman

10 Feb 10 Feb

12:08 a.m.

On Wed, Feb 10, 2010 at 02:03, Benjamin Peterson wrote:

...

What do you mean by moved? I don't it has ever moved around in the sandbox.

IIRC it was moved into the sandbox from some other location at some point? Cheers, Dirkjan

Benjamin Peterson

4:59 a.m.

2010/2/10 Dirkjan Ochtman :

...

On Wed, Feb 10, 2010 at 02:03, Benjamin Peterson wrote:

...
What do you mean by moved? I don't it has ever moved around in the sandbox.

IIRC it was moved into the sandbox from some other location at some point?

r52858 | guido.van.rossum | 2006-11-29 11:38:40 -0600 (Wed, 29 Nov 2006) | 4 lines Changed paths: A /sandbox/trunk/2to3 A /sandbox/trunk/2to3/Grammar.pickle A /sandbox/trunk/2to3/Grammar.txt A /sandbox/trunk/2to3/pgen2 A /sandbox/trunk/2to3/pgen2/__init__.py A /sandbox/trunk/2to3/pgen2/__init__.pyc A /sandbox/trunk/2to3/pgen2/astnode.py A /sandbox/trunk/2to3/pgen2/conv.py A /sandbox/trunk/2to3/pgen2/driver.py A /sandbox/trunk/2to3/pgen2/grammar.py A /sandbox/trunk/2to3/pgen2/literals.py A /sandbox/trunk/2to3/pgen2/parse.py A /sandbox/trunk/2to3/pgen2/pgen.py A /sandbox/trunk/2to3/pgen2/python.py A /sandbox/trunk/2to3/pgen2/test.py A /sandbox/trunk/2to3/play.py A /sandbox/trunk/2to3/pynode.py Checkpoint of alternative Python 2.x-to-3.0 conversion tool. This contains a modified copy of pgen2 which was open-sourced by Elemental Security through a contributor's agreement with the PSF. ------------------------------------------------------------------------ The only moving was moving a lot of the files into a lib2to3 directory. It would be nice if the hg history could be preserved for those files. -- Regards, Benjamin

Dirkjan Ochtman

10:03 a.m.

On Wed, Feb 10, 2010 at 13:59, Benjamin Peterson wrote:

...

The only moving was moving a lot of the files into a lib2to3 directory. It would be nice if the hg history could be preserved for those files.

Please see if hg.python.org/2to3 would satisfy your needs. Cheers, Dirkjan

Georg Brandl

12 Feb 12 Feb

12:34 a.m.

Am 09.02.2010 04:47, schrieb Benjamin Peterson:

...

2010/2/8 "Martin v. Löwis" :

...
Benjamin Peterson wrote:

...
2010/2/8 Dirkjan Ochtman :

...
On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...
Will you do test conversions of the sandbox projects, too? Got any particular projects in mind?

2to3.

Does Mercurial even support merge tracking the way we are doing it for 2to3 right now?

I don't believe so. My plan was to manually sync updates or use subrepos.

Why even keep 2to3 in the sandbox? It should be mature enough now to be maintained directly in the tree. Also, using a subrepo is fine for the Python 2 version, but what about the Python 3 version? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Georg Brandl

12:39 a.m.

Am 09.02.2010 04:39, schrieb "Martin v. Löwis":

...

Benjamin Peterson wrote:

...
2010/2/8 Dirkjan Ochtman :

...
On Sun, Feb 7, 2010 at 22:51, Benjamin Peterson wrote:

...
Will you do test conversions of the sandbox projects, too? Got any particular projects in mind?

2to3.

Does Mercurial even support merge tracking the way we are doing it for 2to3 right now?

No, it does not. This is also a concern for the Python 2 -> Python 3 merging, where (I think) we decided not to have shared history. Transplant already does most of the job (including recording the source hash of transplanted changesets), but it lacks blocking and consistent rejection of already-merged changesets (it does not read the source hashes it records, but keeps a local cache of such hashes instead, which obviously doesn't do anything across repositories.) I think it should be possible to have transplant regenerate and update that cache automatically on clone/pull/etc. I guess this is a relatively simple task for a Mercurial hacker, and if it's decided to use this workflow "someone" ;) could address it at the PyCon sprint. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Dirkjan Ochtman

1:36 a.m.

On Fri, Feb 12, 2010 at 09:39, Georg Brandl wrote:

...

No, it does not. This is also a concern for the Python 2 -> Python 3 merging, where (I think) we decided not to have shared history. Transplant already

I don't think this is similar to 2 vs. 3, because 2 vs. 3 are full branching (so you could still use "normal" hg merge tracking there). Since hg doesn't do merge tracking on the directory level, you couldn't use Mercurial merges (or transplant, AFAICS) to do what you want here.

...

does most of the job (including recording the source hash of transplanted changesets), but it lacks blocking and consistent rejection of already-merged changesets (it does not read the source hashes it records, but keeps a local cache of such hashes instead, which obviously doesn't do anything across repositories.) I think it should be possible to have transplant regenerate and update that cache automatically on clone/pull/etc.

I guess this is a relatively simple task for a Mercurial hacker, and if it's decided to use this workflow "someone" ;) could address it at the PyCon sprint.

Yes, we should figure out some workflow issues soon. Cheers, Dirkjan

"Martin v. Löwis"

11:17 a.m.

...

Why even keep 2to3 in the sandbox? It should be mature enough now to be maintained directly in the tree.

I think the original plan was to make standalone releases, so that people could upgrade their installation from a newer release of 2to3. IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well. Regards, Martin

Brett Cannon

11:34 a.m.

On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote:

...

...
Why even keep 2to3 in the sandbox? It should be mature enough now to be maintained directly in the tree.

I think the original plan was to make standalone releases, so that people could upgrade their installation from a newer release of 2to3.

That's what I remember as well.

...

IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well.

I vote on giving up the 2to3 sandbox. -Brett

Nick Coghlan

5:51 p.m.

Brett Cannon wrote:

...

On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote:

...
IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well.

I vote on giving up the 2to3 sandbox.

Besides, if we're using hg, it should make it much easier for someone else to branch that part of the stdlib and create a standalone 2to3 release from it if they really want to. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Nick Coghlan

5:53 p.m.

Brett Cannon wrote:

...

On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote: I vote on giving up the 2to3 sandbox.

One other point - is there a Python 2.6 backwards compatibility restriction on 2to3 at the moment? If there isn't, should there be? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Benjamin Peterson

6:02 p.m.

2010/2/12 Nick Coghlan :

...

Brett Cannon wrote:

...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote:

...
IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well.

I vote on giving up the 2to3 sandbox.

Besides, if we're using hg, it should make it much easier for someone else to branch that part of the stdlib and create a standalone 2to3 release from it if they really want to.

I personally like 2to3 in a separate repo because it fits well with my view that 2to3 is an extra application that happens to also be distributed with python. -- Regards, Benjamin

Benjamin Peterson

6:02 p.m.

2010/2/12 Nick Coghlan :

...

Brett Cannon wrote:

...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote: I vote on giving up the 2to3 sandbox.

One other point - is there a Python 2.6 backwards compatibility restriction on 2to3 at the moment? If there isn't, should there be?

I try to keep it compatible with 2.6, since we have to backport changes. -- Regards, Benjamin

Nick Coghlan

6:23 p.m.

Benjamin Peterson wrote:

...

2010/2/12 Nick Coghlan :

...
Brett Cannon wrote:

...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote: I vote on giving up the 2to3 sandbox. One other point - is there a Python 2.6 backwards compatibility restriction on 2to3 at the moment? If there isn't, should there be?

I try to keep it compatible with 2.6, since we have to backport changes.

With 2.7 just around the corner, it should probably be listed in PEP 291 on that basis. Of course, PEP 291 could do with a list of 2.5 and 2.6 specific features first... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Benjamin Peterson

6:35 p.m.

2010/2/12 Nick Coghlan :

...

Benjamin Peterson wrote:

...
2010/2/12 Nick Coghlan :

...
Brett Cannon wrote:

...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote: I vote on giving up the 2to3 sandbox. One other point - is there a Python 2.6 backwards compatibility restriction on 2to3 at the moment? If there isn't, should there be?

I try to keep it compatible with 2.6, since we have to backport changes.

With 2.7 just around the corner, it should probably be listed in PEP 291 on that basis.

Done.

...

Of course, PEP 291 could do with a list of 2.5 and 2.6 specific features first...

I think that section is rather pointless to keep updated, since a good list can be found in the what's new documents. What people really need to do is run the unittests on all supported versions. -- Regards, Benjamin

"Martin v. Löwis"

10:31 p.m.

...

...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote:

...
IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well. I vote on giving up the 2to3 sandbox.

Besides, if we're using hg, it should make it much easier for someone else to branch that part of the stdlib

Actually - no: hg doesn't support branching of parts of a repository. You would need to branch all of Python. Then, there wouldn't be a straight-forward place to setup.py and any other top-level files (although you could hack them into Lib, and work with a distutils manifest). Regards, Martin

"Martin v. Löwis"

10:38 p.m.

...

I personally like 2to3 in a separate repo because it fits well with my view that 2to3 is an extra application that happens to also be distributed with python.

But isn't that just a theoretical property? I know that's how 2to3 started, but who, other than the committers, actually accesses the 2to3 repo? I would be much more supportive of that view if there had been a single release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due to me creating lib2to3, you actually couldn't release it as an extra application and run it on 2.6 or 2.7, as the builtin lib2to3 would take precedence over the lib2to3 bundled with the application. Regards, Martin

Nick Coghlan

11:55 p.m.

Benjamin Peterson wrote:

...

2010/2/12 Nick Coghlan :

...
Of course, PEP 291 could do with a list of 2.5 and 2.6 specific features first...

I think that section is rather pointless to keep updated, since a good list can be found in the what's new documents. What people really need to do is run the unittests on all supported versions.

It's handy as a list of big ticket items to avoid, especially those that can significantly affect the way you structure code. Agreed that the main enforcement should be to run those tests on the relevant older versions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Barry Warsaw

13 Feb 13 Feb

8:14 a.m.

On Feb 13, 2010, at 1:31 AM, Martin v. Löwis wrote:

...

...
...
On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis" wrote:

...
IMO, it is realistic to predict that this will not actually happen. If we can agree to give up the 2to3 sandbox, we should incorporate find_pattern into the tree, and perhaps test.py as well. I vote on giving up the 2to3 sandbox.

Besides, if we're using hg, it should make it much easier for someone else to branch that part of the stdlib

Actually - no: hg doesn't support branching of parts of a repository. You would need to branch all of Python. Then, there wouldn't be a straight-forward place to setup.py and any other top-level files (although you could hack them into Lib, and work with a distutils manifest).

Does hg support an equivalent of 'bzr split'? % bzr split --help Purpose: Split a subdirectory of a tree into a separate tree. Usage: bzr split TREE Options: --usage Show usage message and options. -v, --verbose Display more information. -q, --quiet Only display errors and warnings. -h, --help Show help message. Description: This command will produce a target tree in a format that supports rich roots, like 'rich-root' or 'rich-root-pack'. These formats cannot be converted into earlier formats like 'dirstate-tags'. The TREE argument should be a subdirectory of a working tree. That subdirectory will be converted into an independent tree, with its own branch. Commits in the top-level tree will not apply to the new subtree. See also: join -Barry

Benjamin Peterson

8:48 a.m.

2010/2/13 "Martin v. Löwis" :

...

...
I personally like 2to3 in a separate repo because it fits well with my view that 2to3 is an extra application that happens to also be distributed with python.

But isn't that just a theoretical property? I know that's how 2to3 started, but who, other than the committers, actually accesses the 2to3 repo?

It's used in 3to2 for example.

...

I would be much more supportive of that view if there had been a single release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due to me creating lib2to3, you actually couldn't release it as an extra application and run it on 2.6 or 2.7, as the builtin lib2to3 would take precedence over the lib2to3 bundled with the application.

It could be distributed under another name or provide a way to override the stdlib version. -- Regards, Benjamin

"Martin v. Löwis"

9:14 a.m.

...

...
But isn't that just a theoretical property? I know that's how 2to3 started, but who, other than the committers, actually accesses the 2to3 repo?

It's used in 3to2 for example.

That doesn't really seem to be the case. AFAICT, 3to2 is a hg repository, with no inherent connection to the 2to3 svn sandbox. It does use lib2to3, but that could come either from an installed Python, from a trunk/3k checkout, or from the sandbox. Correct? So if the 2.x trunk became the official master for (lib)2to3, nothing would really change for 3to3, right? (except for the comment in the readme that you should get 2to3 from the sandbox if the trunk copy doesn't work; this comment would become obsolete as changes *would* propagate immediately into the Python trunk).

...

...
I would be much more supportive of that view if there had been a single release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due to me creating lib2to3, you actually couldn't release it as an extra application and run it on 2.6 or 2.7, as the builtin lib2to3 would take precedence over the lib2to3 bundled with the application.

It could be distributed under another name or provide a way to override the stdlib version.

Sure. However, I'm still claiming that this is theoretical. The only person who has shown a slight interest in having this as a separate project (since Collin Winter left) is you, and so far, you haven't made any efforts to produce a stand-alone release. I don't blame you at all for that, in fact, I think Python is better off with the status quo (i.e. changes to 2to3 get liberally released even with bug fix releases, basically in an exemption from the "no new features" policy - similar to -3 warnings). I still think that the best approach for projects to use 2to3 is to run 2to3 at install time from a single-source release. For that, projects will have to adjust to whatever bugs certain 2to3 releases have, rather than requiring users to download a newer version of 2to3 that fixes them. For this use case, a tightly-integrated lib2to3 (with that name and sole purpose) is the best thing. Regards, Martin

Dirkjan Ochtman

9:49 a.m.

On Sat, Feb 13, 2010 at 17:14, Barry Warsaw wrote:

...

Does hg support an equivalent of 'bzr split'?

% bzr split --help Purpose: Split a subdirectory of a tree into a separate tree. Usage: bzr split TREE

Options: --usage Show usage message and options. -v, --verbose Display more information. -q, --quiet Only display errors and warnings. -h, --help Show help message.

Description: This command will produce a target tree in a format that supports rich roots, like 'rich-root' or 'rich-root-pack'. These formats cannot be converted into earlier formats like 'dirstate-tags'.

The TREE argument should be a subdirectory of a working tree. That subdirectory will be converted into an independent tree, with its own branch. Commits in the top-level tree will not apply to the new subtree.

Is that like a clone/branch of a subdir of the original repository? We don't have what we usually call "narrow clones" yet (nor "shallow clones", the other potentially useful form of partial clones). We do have the convert extension, which allows you to create a new repository from a subtree of an old repository, but it changes all the hashes (and I don't know if we have a way to splice them back together). Cheers, Dirkjan

Benjamin Peterson

10:43 a.m.

2010/2/13 Dirkjan Ochtman :

...

On Sat, Feb 13, 2010 at 17:14, Barry Warsaw wrote:

...
Does hg support an equivalent of 'bzr split'?

% bzr split --help Purpose: Split a subdirectory of a tree into a separate tree. Usage: bzr split TREE

Options: --usage Show usage message and options. -v, --verbose Display more information. -q, --quiet Only display errors and warnings. -h, --help Show help message.

Description: This command will produce a target tree in a format that supports rich roots, like 'rich-root' or 'rich-root-pack'. These formats cannot be converted into earlier formats like 'dirstate-tags'.

The TREE argument should be a subdirectory of a working tree. That subdirectory will be converted into an independent tree, with its own branch. Commits in the top-level tree will not apply to the new subtree.

Is that like a clone/branch of a subdir of the original repository? We don't have what we usually call "narrow clones" yet (nor "shallow clones", the other potentially useful form of partial clones). We do have the convert extension, which allows you to create a new repository from a subtree of an old repository, but it changes all the hashes (and I don't know if we have a way to splice them back together).

It is not a partial clone, but rather similar to what you are referring to with the convert extension. -- Regards, Benjamin

Benjamin Peterson

11:23 a.m.

2010/2/13 "Martin v. Löwis" :

...

...
...
But isn't that just a theoretical property? I know that's how 2to3 started, but who, other than the committers, actually accesses the 2to3 repo?

It's used in 3to2 for example.

That doesn't really seem to be the case. AFAICT, 3to2 is a hg repository, with no inherent connection to the 2to3 svn sandbox. It does use lib2to3, but that could come either from an installed Python, from a trunk/3k checkout, or from the sandbox. Correct?

It has to be from the sandbox (or trunk I suppose) because it requires changes that haven't been released.

...

So if the 2.x trunk became the official master for (lib)2to3, nothing would really change for 3to3, right? (except for the comment in the readme that you should get 2to3 from the sandbox if the trunk copy doesn't work; this comment would become obsolete as changes *would* propagate immediately into the Python trunk).

Right, except you would have to clone the entire history of Python in order to get at the trunk version.

...

...
...
I would be much more supportive of that view if there had been a single release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due to me creating lib2to3, you actually couldn't release it as an extra application and run it on 2.6 or 2.7, as the builtin lib2to3 would take precedence over the lib2to3 bundled with the application.

It could be distributed under another name or provide a way to override the stdlib version.

Sure. However, I'm still claiming that this is theoretical. The only person who has shown a slight interest in having this as a separate project (since Collin Winter left) is you, and so far, you haven't made any efforts to produce a stand-alone release. I don't blame you at all for that, in fact, I think Python is better off with the status quo (i.e. changes to 2to3 get liberally released even with bug fix releases, basically in an exemption from the "no new features" policy - similar to -3 warnings).

I still think that the best approach for projects to use 2to3 is to run 2to3 at install time from a single-source release. For that, projects will have to adjust to whatever bugs certain 2to3 releases have, rather than requiring users to download a newer version of 2to3 that fixes them. For this use case, a tightly-integrated lib2to3 (with that name and sole purpose) is the best thing.

Alright. That is reasonable. The other thing is that we will loose some vcs history and some history granularity by switching development to the trunk version, since just the svnmerged revisions will be converted. -- Regards, Benjamin

"Martin v. Löwis"

11:35 a.m.

...

The other thing is that we will loose some vcs history and some history granularity by switching development to the trunk version, since just the svnmerged revisions will be converted.

I suppose it might be possible to fake the history of Lib/lib2to3 with commits that didn't actually happen, although this is probably a cure worse than the disease. We are not going to throw away the subversion repository, so it would always be possible to go back and look at the actual history. Regards, Martin

Barry Warsaw

3:22 p.m.

On Feb 13, 2010, at 1:43 PM, Benjamin Peterson wrote:

...

2010/2/13 Dirkjan Ochtman :

...
On Sat, Feb 13, 2010 at 17:14, Barry Warsaw wrote:

...
Does hg support an equivalent of 'bzr split'?

% bzr split --help Purpose: Split a subdirectory of a tree into a separate tree. Usage: bzr split TREE

Options: --usage Show usage message and options. -v, --verbose Display more information. -q, --quiet Only display errors and warnings. -h, --help Show help message.

Description: This command will produce a target tree in a format that supports rich roots, like 'rich-root' or 'rich-root-pack'. These formats cannot be converted into earlier formats like 'dirstate-tags'.

The TREE argument should be a subdirectory of a working tree. That subdirectory will be converted into an independent tree, with its own branch. Commits in the top-level tree will not apply to the new subtree.

Is that like a clone/branch of a subdir of the original repository? We don't have what we usually call "narrow clones" yet (nor "shallow clones", the other potentially useful form of partial clones). We do have the convert extension, which allows you to create a new repository from a subtree of an old repository, but it changes all the hashes (and I don't know if we have a way to splice them back together).

It is not a partial clone, but rather similar to what you are referring to with the convert extension.

Right, that's what it sounds like. I've used it on a bzr-svn converted repository to split a monolithic tree after Subversion->Bazaar conversion into separately managed subtrees. Note that 'bzr join' is the inverse. The interesting thing (both good and bad) is that after the split both subtrees have the full history of the original. -Barry

Joe Amenta

3:52 p.m.

On Sat, Feb 13, 2010 at 12:14 PM, "Martin v. Löwis" wrote:

...

...
...
But isn't that just a theoretical property? I know that's how 2to3 started, but who, other than the committers, actually accesses the 2to3 repo?

It's used in 3to2 for example.

That doesn't really seem to be the case. AFAICT, 3to2 is a hg repository, with no inherent connection to the 2to3 svn sandbox. It does use lib2to3, but that could come either from an installed Python, from a trunk/3k checkout, or from the sandbox. Correct?

So if the 2.x trunk became the official master for (lib)2to3, nothing would really change for 3to3, right? (except for the comment in the readme that you should get 2to3 from the sandbox if the trunk copy doesn't work; this comment would become obsolete as changes *would* propagate immediately into the Python trunk).

...
...
I would be much more supportive of that view if there had been a single release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due to me creating lib2to3, you actually couldn't release it as an extra application and run it on 2.6 or 2.7, as the builtin lib2to3 would take precedence over the lib2to3 bundled with the application.

It could be distributed under another name or provide a way to override the stdlib version.

Sure. However, I'm still claiming that this is theoretical. The only person who has shown a slight interest in having this as a separate project (since Collin Winter left) is you, and so far, you haven't made any efforts to produce a stand-alone release. I don't blame you at all for that, in fact, I think Python is better off with the status quo (i.e. changes to 2to3 get liberally released even with bug fix releases, basically in an exemption from the "no new features" policy - similar to -3 warnings).

I still think that the best approach for projects to use 2to3 is to run 2to3 at install time from a single-source release. For that, projects will have to adjust to whatever bugs certain 2to3 releases have, rather than requiring users to download a newer version of 2to3 that fixes them. For this use case, a tightly-integrated lib2to3 (with that name and sole purpose) is the best thing.

Regards, Martin

Yes, if the trunk were the official master for lib2to3, then 3to2 would not change at all. If fixes to lib2to3 were immediately propagated to the trunk, 3to2 would benefit from that. I support lib2to3's integration with the trunk... it's too confusing otherwise and kind of defeats the idea of "trunk": if lib2to3 is provided with Python, then shouldn't its latest version be in Python's trunk? --Joe Amenta

Collin Winter

22 Feb 22 Feb

1:09 p.m.

On Sat, Feb 13, 2010 at 2:23 PM, Benjamin Peterson wrote:

...

2010/2/13 "Martin v. Löwis" :

...
I still think that the best approach for projects to use 2to3 is to run 2to3 at install time from a single-source release. For that, projects will have to adjust to whatever bugs certain 2to3 releases have, rather than requiring users to download a newer version of 2to3 that fixes them. For this use case, a tightly-integrated lib2to3 (with that name and sole purpose) is the best thing.

Alright. That is reasonable.

The other thing is that we will loose some vcs history and some history granularity by switching development to the trunk version, since just the svnmerged revisions will be converted.

So the consensus is that 2to3 should be pulled out of the main Python tree? Should the 2to3 hg repository be deleted, then? Thanks, Collin

"Martin v. Löwis"

1:27 p.m.

...

...
The other thing is that we will loose some vcs history and some history granularity by switching development to the trunk version, since just the svnmerged revisions will be converted.

So the consensus is that 2to3 should be pulled out of the main Python tree?

Not sure what you mean by "pull out"; I had expect that the right verb should be "pull into": 2to3 should be pulled into the main Python tree.

...

Should the 2to3 hg repository be deleted, then?

Which one? To my knowledge, there is no official 2to3 repository yet. When the switchover happens, 2to3 should not be converted to its own hg repository, yes. Regards, Martin

Collin Winter

1:31 p.m.

On Mon, Feb 22, 2010 at 4:27 PM, "Martin v. Löwis" wrote:

...

...
...
The other thing is that we will loose some vcs history and some history granularity by switching development to the trunk version, since just the svnmerged revisions will be converted.

So the consensus is that 2to3 should be pulled out of the main Python tree?

Not sure what you mean by "pull out"; I had expect that the right verb should be "pull into": 2to3 should be pulled into the main Python tree.

Sorry, I meant "pulled out" as in: I want an updated version for the benchmark suite, where should I get that?

...

...
Should the 2to3 hg repository be deleted, then?

Which one? To my knowledge, there is no official 2to3 repository yet. When the switchover happens, 2to3 should not be converted to its own hg repository, yes.

This one: http://hg.python.org/2to3 Collin

Dirkjan Ochtman

1:33 p.m.

On Mon, Feb 22, 2010 at 16:09, Collin Winter wrote:

...

So the consensus is that 2to3 should be pulled out of the main Python tree? Should the 2to3 hg repository be deleted, then?

Wouldn't the former be reason to officialize the hg repository, instead of deleting it? Cheers, Dirkjan

Nick Coghlan

2:03 p.m.

Dirkjan Ochtman wrote:

...

On Mon, Feb 22, 2010 at 16:09, Collin Winter wrote:

...
So the consensus is that 2to3 should be pulled out of the main Python tree? Should the 2to3 hg repository be deleted, then?

Wouldn't the former be reason to officialize the hg repository, instead of deleting it?

I think the difference between "pull out" and "pull from" is causing confusion here (and no, I'm not sure which of those Collin actually meant either). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Collin Winter

2:05 p.m.

On Mon, Feb 22, 2010 at 5:03 PM, Nick Coghlan wrote:

...

Dirkjan Ochtman wrote:

...
On Mon, Feb 22, 2010 at 16:09, Collin Winter wrote:

...
So the consensus is that 2to3 should be pulled out of the main Python tree? Should the 2to3 hg repository be deleted, then?

Wouldn't the former be reason to officialize the hg repository, instead of deleting it?

I think the difference between "pull out" and "pull from" is causing confusion here (and no, I'm not sure which of those Collin actually meant either).

Sorry, I meant "pull from". I want an updated snapshot of 2to3 for the benchmark suite, and I'm looking for the best place to grab it from. Collin

Dirkjan Ochtman

2:40 p.m.

On Mon, Feb 22, 2010 at 17:05, Collin Winter wrote:

...

Sorry, I meant "pull from". I want an updated snapshot of 2to3 for the benchmark suite, and I'm looking for the best place to grab it from.

Well, the server that has all the stuff for doing the conversions has annoyingly been down for about 24 hours now. I've got a friend coming in who should hopefully be fixing this tomorrow in the afternoon (Atlanta time), after which I should be able to make my conversion stuff update the hg.p.o repositories with more regularity. Cheers, Dirkjan

Martin Geisler

3:38 p.m.

Dirkjan Ochtman writes: Hi everybody! I hope you have fun at PyCon :-)

...

As for the current state of The Dreaded EOL Issue, there is an extension which seems to be provide all the needed features, but it appears there are some nasty corner cases still to be fixed. Martin Geisler has been working on it over the sprint, but I think there's more work to be done here. Anyone who wants to jump in would be quite welcome (maybe Martin will clarify here what exactly the remaining issues are).

I'm sorry about the delay in my response -- but things have now finally moved forward after Benoit Boissinot (another Mercurial developer) looked at things. With the most recent fixes pushed to the eol repository[1], I can no longer break the tests by running them repeatedly in a loop. In other words, they finally appear to be stable. I feel this would be a good opportunity for people to begin testing the extension again. It seems that people has not done that so far, or at least we haven't gotten any feedback in a long time. It is now easier to test than before since changes to the .hgeol file is picked up immediatedly without it being committed. This means that you can enable eol (in .hg/hgrc, say) and play around *without* affecting others who use the repository. When you change patterns in .hgeol, you'll see the effects in the output of 'hg status' -- files that will be updated on the next commit appear modified. My dissertation is due this Friday(!), so I will not have much time to look at EOL issues this week (as usual). But please give it a spin anyway and let us hear what you think! [1]: http://bitbucket.org/mg/hg-eol/ -- Martin Geisler

Dirkjan Ochtman

3:56 p.m.

On Mon, Feb 22, 2010 at 18:38, Martin Geisler wrote:

...

My dissertation is due this Friday(!), so I will not have much time to look at EOL issues this week (as usual). But please give it a spin anyway and let us hear what you think!

I've got about 48 more hours of PyCon sprints ahead of me, so if anyone comes up with bugs (preferably concrete and reproducible) in that time frame, I can look into them more or less directly. Cheers, Dirkjan

"Martin v. Löwis"

9:55 p.m.

...

...
...
Should the 2to3 hg repository be deleted, then? Which one? To my knowledge, there is no official 2to3 repository yet. When the switchover happens, 2to3 should not be converted to its own hg repository, yes.

This one: http://hg.python.org/2to3

Ah, this shouldn't be used at all for anything (except for studying how Mercurial works). Along with the cpython repository, it is Dirkjan's test conversion. Even if it survived the ultimate migration (which it probably won't), it would get regenerated from scratch, probably changing all revision numbers. Regards, Martin

"Martin v. Löwis"

9:55 p.m.

...

Sorry, I meant "pull from". I want an updated snapshot of 2to3 for the benchmark suite, and I'm looking for the best place to grab it from.

The 2to3 code currently still lives in the subversion sandbox. Regards, Martin

Dirkjan Ochtman

10:01 p.m.

On Tue, Feb 23, 2010 at 00:55, "Martin v. Löwis" wrote:

...

Ah, this shouldn't be used at all for anything (except for studying how Mercurial works). Along with the cpython repository, it is Dirkjan's test conversion. Even if it survived the ultimate migration (which it probably won't), it would get regenerated from scratch, probably changing all revision numbers.

Actually, since this one is much simpler, it's more or less ready to be the canonical repository, if Benjamin would like that. It's very different from the cpython repository in that respect. Cheers, Dirkjan

"Martin v. Löwis"

10:06 p.m.

...

On Tue, Feb 23, 2010 at 00:55, "Martin v. Löwis" wrote:

...
Ah, this shouldn't be used at all for anything (except for studying how Mercurial works). Along with the cpython repository, it is Dirkjan's test conversion. Even if it survived the ultimate migration (which it probably won't), it would get regenerated from scratch, probably changing all revision numbers.

Actually, since this one is much simpler, it's more or less ready to be the canonical repository, if Benjamin would like that. It's very different from the cpython repository in that respect.

I thought we decided not to have a 2to3 repository at all, but let this live in the Python trunk exclusively. Regards, Martin

Dirkjan Ochtman

10:13 p.m.

On Tue, Feb 23, 2010 at 01:06, "Martin v. Löwis" wrote:

...

I thought we decided not to have a 2to3 repository at all, but let this live in the Python trunk exclusively.

That would be fine with me, I just remembered that Benjamin would like to start using hg sooner and having it as a separate repo was okay. Cheers, Dirkjan

Benjamin Peterson

23 Feb 23 Feb

5:28 p.m.

2010/2/23 Dirkjan Ochtman :

...

On Tue, Feb 23, 2010 at 01:06, "Martin v. Löwis" wrote:

...
I thought we decided not to have a 2to3 repository at all, but let this live in the Python trunk exclusively.

That would be fine with me, I just remembered that Benjamin would like to start using hg sooner and having it as a separate repo was okay.

I would :), but Martin is correct in that we agreed to start developing it as part of the trunk when the hg transition takes place. -- Regards, Benjamin

5172

Age (days ago)

5189

Last active (days ago)

List overview

Download

50 comments

12 participants

participants (12)

"Martin v. Löwis"
Barry Warsaw
Benjamin Peterson
Brett Cannon
Collin Winter
Dirkjan Ochtman
Dirkjan Ochtman
Georg Brandl
Joe Amenta
Mark Hammond
Martin Geisler
Nick Coghlan