Hello, The PyPy project recently switched from svn to mercurial. Since this day I have some difficulties to perform simple tasks, and my questions did not receive satisfying answers. I was sure the Python project would have the same issues; so I started from the repositories in http://code.python.org/hg/ and tried simple merges between Python versions. I would like several people to try the same example, and tell how they handle it. I'm new to mercurial, and I may have missed something important! Let's say you have to do the simple change shown in the diff below, and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. The diff was made against py3k. How would you do it with Mercurial? Please try to do it for real! "hg merge" is not the correct command: it merges whole branches, a change comes with all its ancestors. What we want is to "cherry-pick" a single change. "hg transplant" fails to apply the change to release31-maint because of a tiny conflict (in the diff context!) that leaves you with an ugly "hunks FAILED" and a .rej file you have to parse and apply by hand. "hg transplant" seems to succeed in release27-maint, but the test fails to run because "support" should be changed to "test_support". The change is already committed - to fix it another changeset is needed. This does not look clean to me: both changesets will be pushed to the repository, and diff review (on the python-checkins mailing list) is almost impossible. Furthermore, "hg transplant" does not keep track of integrations. There is a "transplants" file, but it stays in my local clone. Faced with a similar case in pypy, we finally manually copied the files between directories... and the fastest with our example is probably to do the change manually in all three directories. There is surely a better way to work with maintenance branches! PEP374 suggests to first modify the oldest release, but this does not seems right to me (I have three reasons in mind) At the moment, I feel that mercurial just cannot work for core Python development. At the very least before the migration we need precise use cases like this and recipes for common cases. Thanks for your help, -- Amaury Forgeot d'Arc diff -r 2777ae4d10d9 Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py Tue Dec 28 22:14:34 2010 +0100 +++ b/Lib/test/test_contextlib.py Wed Dec 29 00:08:18 2010 +0100 @@ -26,6 +26,7 @@ state.append(x) self.assertEqual(state, [1, 42, 999]) + @support.cpython_only def test_contextmanager_finally(self): state = [] @contextmanager @@ -36,10 +37,10 @@ finally: state.append(999) with self.assertRaises(ZeroDivisionError): - with woohoo() as x: + with woohoo() as y: self.assertEqual(state, [1]) - self.assertEqual(x, 42) - state.append(x) + self.assertEqual(y, 42) + state.append(y) raise ZeroDivisionError() self.assertEqual(state, [1, 42, 999])
On Wed, Dec 29, 2010 at 9:13 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Hello, The PyPy project recently switched from svn to mercurial. Since this day I have some difficulties to perform simple tasks, and my questions did not receive satisfying answers. I was sure the Python project would have the same issues; so I started from the repositories in http://code.python.org/hg/ and tried simple merges between Python versions. I would like several people to try the same example, and tell how they handle it. I'm new to mercurial, and I may have missed something important!
Let's say you have to do the simple change shown in the diff below, and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. The diff was made against py3k. How would you do it with Mercurial? Please try to do it for real!
The easiest way I found to emulate git cherry-pick (which does exactly what you want) with hg is to use import/export commands: http://mercurial.selenic.com/wiki/CommunicatingChanges It is indeed quite a pain to use in my experience, because you cannot easily refer to the original commit the cherry pick is coming from (i.e. no equivalent to git cherry-pick -x), and the conflict resolution is quite dumb. One alternative is to be careful about where you apply your patch (http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to...), but that's not very convenient either. cheers, David
2010/12/29 David Cournapeau <cournape@gmail.com>
The easiest way I found to emulate git cherry-pick (which does exactly what you want) with hg is to use import/export commands: http://mercurial.selenic.com/wiki/CommunicatingChanges
It is indeed quite a pain to use in my experience, because you cannot easily refer to the original commit the cherry pick is coming from (i.e. no equivalent to git cherry-pick -x), and the conflict resolution is quite dumb.
This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes? And finally the biased question: can you find one improvement over the current situation with svn? One alternative is to be careful about where
you apply your patch ( http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to... ), but that's not very convenient either.
I've read all this, and this method does not work, for several reasons: - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. we usually consider py3k as a child of both 2.7 and 3.1, and there is no common parent. - even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge. - in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository. (btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?) -- Amaury Forgeot d'Arc
Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc:
I've read all this, and this method does not work, for several reasons:
- py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. we usually consider py3k as a child of both 2.7 and 3.1, and there is no common parent.
Yes. Think of 2.7 as the oddball one, where revisions must be cherry-picked to, whereas the 3.x line can use the "commit in parent branch" system (and much more easily, since there are no changes in syntax and much less changes in code overall).
- even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge.
No horror at all: the next committer notices the extra changes in his merge and removes them. *Never* commit merges without reviewing the diff. (The situtation is similar, by the way, to someone typing the wrong revision number when using svnmerge, and not noticing it because he does not review the diff.)
- in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository.
Note that the repos on code.python.org are not the results of our conversion process. Those will be on hg.python.org. The former are repos contributed by Antoine (I think) that he uses with hgsubversion to work on Python using Mercurial right now; they will disappear after hg.python.org works.
(btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?)
No, you can "hg strip" away the root of the wrongly pulled revisions (which also strips all descendants), or use "hg clone -r XXX" to create a new clone with only a specified revision and all its ancestors. Georg
2010/12/29 Georg Brandl <g.brandl@gmx.net>
Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc:
- even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge.
No horror at all: the next committer notices the extra changes in his merge and removes them. *Never* commit merges without reviewing the diff. (The situtation is similar, by the way, to someone typing the wrong revision number when using svnmerge, and not noticing it because he does not review the diff.)
Except that it's easy to overlook a diff and not notice another small change mixed in your merge. Checking "hg merge -P" is probably better. But is it possible to "remove" the extra change? What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)? With subversion it's easy to work on trunk, and then have svnmerge tell you which chunk is in conflict and does not apply to the maintenance branch.
- in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository.
Note that the repos on code.python.org are not the results of our conversion process. Those will be on hg.python.org. The former are repos contributed by Antoine (I think) that he uses with hgsubversion to work on Python using Mercurial right now; they will disappear after hg.python.org works.
ok. I used them because other the repos I found were really old. And http://hg.python.org/cpython/ probably needs an initial "dummy merge" on every branch.
(btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?)
No, you can "hg strip" away the root of the wrongly pulled revisions (which also strips all descendants), or use "hg clone -r XXX" to create a new clone with only a specified revision and all its ancestors.
"hg strip" worked for me. The root revision to strip was simply the first one (=oldest) in "hg outgoing". -- Amaury Forgeot d'Arc
On Wed, Dec 29, 2010 at 10:53, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
And http://hg.python.org/cpython/ probably needs an initial "dummy merge" on every branch.
Yes, the present delays are all about getting that right. Cheers, Dirkjan
Am 29.12.2010 10:53, schrieb Amaury Forgeot d'Arc:
2010/12/29 Georg Brandl <g.brandl@gmx.net>
Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc:
- even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge.
No horror at all: the next committer notices the extra changes in his merge and removes them. *Never* commit merges without reviewing the diff. (The situtation is similar, by the way, to someone typing the wrong revision number when using svnmerge, and not noticing it because he does not review the diff.)
Except that it's easy to overlook a diff and not notice another small change mixed in your merge. Checking "hg merge -P" is probably better. But is it possible to "remove" the extra change?
No; hg merging is DAG-based, so you will always merge all ancestors. The only way to "remove" it is a null-merge.
What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)?
Finding the correct branch is not really hard. Bugfixes go to maintenance, features to trunk. You need to think about which category your change is right now too, when deciding what to backport/svnmerge. Accordingly, I would apply doc patches in release31-maint and merge them to trunk, probably all at once with one merge commit.
- in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository.
Note that the repos on code.python.org are not the results of our conversion process. Those will be on hg.python.org. The former are repos contributed by Antoine (I think) that he uses with hgsubversion to work on Python using Mercurial right now; they will disappear after hg.python.org works.
ok. I used them because other the repos I found were really old. And http://hg.python.org/cpython/ probably needs an initial "dummy merge" on every branch.
Yes, that is a very old conversion result. We should get a new one in a few days. Georg
2010/12/29 Georg Brandl <g.brandl@gmx.net>:
What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)?
Finding the correct branch is not really hard. Bugfixes go to maintenance, features to trunk.
This works well indeed, provided there is only one maintenance branch.
You need to think about which category your change is right now too, when deciding what to backport/svnmerge.
No, today this decision can take place after the development, some tickets got reopened because a backport was needed. -- Amaury Forgeot d'Arc
Am 29.12.2010 11:09, schrieb Amaury Forgeot d'Arc:
2010/12/29 Georg Brandl <g.brandl@gmx.net>:
What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)?
Finding the correct branch is not really hard. Bugfixes go to maintenance, features to trunk.
This works well indeed, provided there is only one maintenance branch.
Which there is, except for security fixes (but they are rare and require release manager intervention anyway).
You need to think about which category your change is right now too, when deciding what to backport/svnmerge.
No, today this decision can take place after the development, some tickets got reopened because a backport was needed.
A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one. Of course you may have to think a bit more about bugfix *before* plunging into the code, but is that necessarily a bad thing? We had quite a few bugfixes not backported precisely because it is very easy not to care about them right now. Georg
2010/12/29 Georg Brandl <g.brandl@gmx.net>:
You need to think about which category your change is right now too, when deciding what to backport/svnmerge.
No, today this decision can take place after the development, some tickets got reopened because a backport was needed.
A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one.
No, it's not trivial with hg: this is the very subject of the thread! -- Amaury Forgeot d'Arc
Am 29.12.2010 15:17, schrieb Amaury Forgeot d'Arc:
2010/12/29 Georg Brandl <g.brandl@gmx.net>:
You need to think about which category your change is right now too, when deciding what to backport/svnmerge.
No, today this decision can take place after the development, some tickets got reopened because a backport was needed.
A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one.
No, it's not trivial with hg: this is the very subject of the thread!
I don't understand: if you make the same change in two branches, but in different changesets, and then merge these branches, Mercurial will usually notice that and not trouble you with conflicts. Georg
2010/12/29 Georg Brandl <g.brandl@gmx.net>:
A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one.
No, it's not trivial with hg: this is the very subject of the thread!
I don't understand: if you make the same change in two branches, but in different changesets, and then merge these branches, Mercurial will usually notice that and not trouble you with conflicts.
Actually I never passed the first step: make the same change to two branches. -- Amaury Forgeot d'Arc
On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
2010/12/29 David Cournapeau <cournape@gmail.com>
The easiest way I found to emulate git cherry-pick (which does exactly what you want) with hg is to use import/export commands: http://mercurial.selenic.com/wiki/CommunicatingChanges
It is indeed quite a pain to use in my experience, because you cannot easily refer to the original commit the cherry pick is coming from (i.e. no equivalent to git cherry-pick -x), and the conflict resolution is quite dumb.
This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes?
I don't know in this specific case. As I said, when I have to use hg, that's the technique I use, and you get the issue you mention. That's a hg limitation AFAICS.
And finally the biased question: can you find one improvement over the current situation with svn?
I am not involved in the hg conversion process nor its decision (I am not even a python committer). Cherry picking is actually easier to do with svn by "accident", because its merge method, up to 1.5 at least, was really dumb and never remembered the ancestors of a previous merge. As for a few data points which may or may not be relevant: in numpy we convereted from svn -> git recently, and it has worked pretty well, with numerous new contributions happening, and better, new contributors appearing. I have been the release manager for numpy for several years, and as such had to do the kind of tasks you mention numerous times with svn, and the only words that comes to mind when remember this period would not be appropriate on a public mailing list: I always found svn to be horrible. I started using git to make my life as release manager simpler, actually. I would be surprised if python's situation did not end up being similar to numpy's one. Other projects related to numpy made the changes to a DVCS (ipython, nipy, lean scikit) before and none of them ever regretted it AFAIK, and sometimes the people who become the most vocal proponents of the new tool were the most sceptic ones before. Certainly, very few people if any asked to revert the process. cheers, David
2010/12/29 David Cournapeau <cournape@gmail.com>:
On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
2010/12/29 David Cournapeau <cournape@gmail.com>
The easiest way I found to emulate git cherry-pick (which does exactly what you want) with hg is to use import/export commands: http://mercurial.selenic.com/wiki/CommunicatingChanges
It is indeed quite a pain to use in my experience, because you cannot easily refer to the original commit the cherry pick is coming from (i.e. no equivalent to git cherry-pick -x), and the conflict resolution is quite dumb.
This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes?
I don't know in this specific case. As I said, when I have to use hg, that's the technique I use, and you get the issue you mention. That's a hg limitation AFAICS.
Yes, Georg identified three things that "hg transplant" should do better: - an option to not commit - a way to add conflict markers in the source instead of the .rej file (In this case, it may be just as easy to use the standard merge tools) - somehow share the "transplants" cache file between clones.
sometimes the people who become the most vocal proponents of the new tool were the most sceptic ones before.
I really was not sceptic before, and I certainly don't want to become one! But yesterday I was blocked the whole afternoon by something I still call an routine task with most other SCMs; and the only answer I get is "that's right, it's a pain" hg will certainly impose a change in the way we develop Python. I'm not sure everybody understands the consequences. -- Amaury Forgeot d'Arc
Amaury Forgeot d'Arc writes:
Which precise steps do you take [to cherrypick with export/import]?
hg export -r $CHERRY .tmp.patch xemacs .tmp.patch ;; Move to end of log message, type "Cherry-picked from: ", then ;; C-u M-! hg id -i -r $CHERRY RET C-x C-s C-x C-c> hg import .tmp.patch worked for me the one time I tried it, IIRC. Now I use patch queues, and avoid cherry-picking as much as possible. (Avoiding cherry-pick is not going to be possible for 3.x <-> 2.x porting, of course, but in many cases there a simple patch application is not going to work at all, anyway. Should be relatively rare, no?) CHERRY can be a tag, revno, or other alias for the revision. You may want to add other identifying information with additional arguments to "hg id", but "hg id -i" give the canonical ID. You may want to fix up committer and date using the -u and -d arguments to hg import.
How much typing or manual copy/paste is required for this very simple case?
For the simple case, the above can be scripted, and the script takes one argument, CHERRY. It would not be hard to automate adding the cherry-pick notice.
Can you do the merge in less than 10 minutes?
If no conflicts, sure, no problem. Probably about 30 seconds, including adding the cherrypicked revid to the log message. With a script, maybe 1 second. If there are conflicts, it depends on the complexity of the conflicts. I can imagine this could get hairy with the export | import method and a megapatch, but I've never had to deal with that in practice. This is one reason why it's recommended that feature branches be constructed of multiple, clearly separated patches. (This may be difficult to do for complex changes, but it's usually a reasonable goal.)
And finally the biased question: can you find one improvement over the current situation with svn?
No, I can't find just one. How many are you willing to read?<wink> svn has the advantage (and yes, it's a big one) that developers are already used to it. Otherwise, everything goes in favor of any of the dVCSes.
- even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k.
This is a problem with the committer, not with the VCS. Such changes belong in a separate commit, with an appropriate log entry, in *any* version control system. If that is done, there's no problem, you just don't cherrypick that change.
You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log.
I don't see why. Such dummy merges can help only if you are going to rely on hg merge. But you are not going to be merging 2.x and 3.x. True, some patches from the maintenance branch won't be applied to the trunk. This is expected, and it is a problem that needs to be solved, probably with a tool like svnmerge, which will do a dummy merge or a revert commit to exclude that revision from any future merge to trunk. Therefore, if we change our minds, it's perfectly safe to cherrypick. Any conflicts that occur are real conflicts, not merge artifacts.
And think of the horror if someone forgets this dummy merge.
What horror? By cherry-picking you have assumed responsibility for managing future cherry-picks from that source to that target in any case; you can no longer expect reliable help from Mercurial. It's not at all clear to me that a dummy merge could accomplish anything useful, especially in the case of 2.x <-> 3.x cherry-picks. If on the other hand you're thinking of the merge-from-maint-branch case, as I say this is a problem that needs solving anyway, just as it did with svn.
On Wed, 29 Dec 2010 23:12:28 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
worked for me the one time I tried it, IIRC. Now I use patch queues, and avoid cherry-picking as much as possible. (Avoiding cherry-pick is not going to be possible for 3.x <-> 2.x porting, of course, but in many cases there a simple patch application is not going to work at all, anyway. Should be relatively rare, no?)
No. We merge bug fixes to 2.7 on a routine basis. It is the rule rather than the exception, by a long shot. (Features of course only go into trunk...but those are irrelevant to this conversation, since there's no requirement to merge them anywhere...as are bug fixes we choose not to backport, since those don't go into the Python3 maint branch either.) Some such merges apply cleanly, many more have only a few conflicts requiring only trivial tweaks, which are made easy by the in-line merge conflict markers. A few require re-engineering of the patch, and of course it is a bit of a pain with svnmerge to deal with the ones where the target file names have changed (but again, there aren't that many of those). [*] So, since we are going to be maintaining 2.7 for a while, this is a workflow problem that we *must* solve to make the hg transition worthwhile. I have no doubt that we will, but I also have no doubt we need to *solve* it, not just wave our hands. Many thanks to Georg and Djirkan for working on the problem(s). -- R. David Murray www.bitdance.com [*] I'm speaking of stdlib changes, which is what I do, but I suspect the situation is similar for the C code, and certainly the fact that bug fixes backported to Python3 maint are routinely backported to 2.7 is true.
R. David Murray writes:
We merge bug fixes to 2.7 on a routine basis. It is the rule rather than the exception, by a long shot.
For bugfixes, of course it's routine. I understand that. My point was that the case Amaury fears, where the (new) VCS makes things harder than patching or porting by hand, is likely to be relatively uncommon, sandwiched between the "typo fixed in comment" conflicts (aka "trivial tweaks") and those that require reengineering. Also, while workflow helpers will make a big difference to the non-VCS-geeks (ie, almost all Python developers), properly speaking this isn't really an issue with Mercurial, because all of the methods for this purpose are basically "diff | patch", although the executables are called things like "svn" and "python". They all demand workflow helper scripts to regulate the flow of desired and undesired patches. The difference is that the tool for hg is a SMOP, while that for svn is a SMOEP[1].
So, since we are going to be maintaining 2.7 for a while, this is a workflow problem that we *must* solve to make the hg transition worthwhile. I have no doubt that we will, but I also have no doubt we need to *solve* it, not just wave our hands.
Certainly. I think I already said that, no? My point is simply that while Amaury's expression of his requirements is welcome, and his experimenting with hg is extremely valuable, indeed a necessary part of the transition, everything he describes so far is a known problem that we basically know how to solve. He talks about changes to the workflow, but frankly, I don't see a *need* for that.[2] IMO, changes to the workflow will be driven by kaizen, not by some brave new world revolution (Guido inter alia insisted on that) nor by thumb-in-dike disaster recovery (PEP 374 did a pretty good job on that, if I do say so myself). I wish I had more time to do real work on this (not to mention email, thank *you*, David!), but it seems like every time I start programming, I fall asleep ... and six hours later, it's back to day job or family services. :-/ Footnotes: [1] Simpler Matter Of Existing Program. [2] Aside for a need for establishing which hg commands correspond to which parts of the existing workflow, and perhaps creating helper scripts. Ie, I think the chances are pretty good that most people who have already tried hg are at least a little VCS-geeky, and probably they adjust personal workflow to the new VCS. That will not be so transparent to the "the tool should work for me, not vice-versa" majority.
On Thu, 30 Dec 2010 14:42:48 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
R. David Murray writes:
We merge bug fixes to 2.7 on a routine basis. It is the rule rather than the exception, by a long shot.
For bugfixes, of course it's routine. I understand that. My point was that the case Amaury fears, where the (new) VCS makes things harder than patching or porting by hand, is likely to be relatively uncommon, sandwiched between the "typo fixed in comment" conflicts (aka "trivial tweaks") and those that require reengineering.
I thought Amaury was saying it was harder than svnmerge, not harder than patching by hand (ie: that it *was* patching by hand, including .rej files and the lack of tracking). I also heard Georg say that this should be fixable, and that he's partly fixed the tracking issue, but not the patch/merge issue (and that doing so will require a change in the hg core).
Also, while workflow helpers will make a big difference to the non-VCS-geeks (ie, almost all Python developers), properly speaking this isn't really an issue with Mercurial, because all of the methods for this purpose are basically "diff | patch", although the executables are called things like "svn" and "python". They all demand workflow helper scripts to regulate the flow of desired and undesired patches. The difference is that the tool for hg is a SMOP, while that for svn is a SMOEP[1].
Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :)
So, since we are going to be maintaining 2.7 for a while, this is a workflow problem that we *must* solve to make the hg transition worthwhile. I have no doubt that we will, but I also have no doubt we need to *solve* it, not just wave our hands.
Certainly. I think I already said that, no? My point is simply that
Ah, I guess I did miss that, sorry.
while Amaury's expression of his requirements is welcome, and his experimenting with hg is extremely valuable, indeed a necessary part of the transition, everything he describes so far is a known problem that we basically know how to solve. He talks about changes to the workflow, but frankly, I don't see a *need* for that.[2]
Well, there will be *some* workflow change since we're talking about committing to 3.2 maint before the new trunk instead of vice versa. But you are right that that is, mostly, a detail. I'm not as convinced that changes in workflow will be that minimal, though. I don't have much experience with the dvcses yet, but what experience I have had leads me to think that understanding the DAG is a non-trivial and necessary mental leap and does affect the workflow. I don't feel as though I've made that leap yet. Hopefully Brett will be able to document this in the Python context so that the *required* leap will be much smaller.
IMO, changes to the workflow will be driven by kaizen, not by some brave new world revolution (Guido inter alia insisted on that) nor by thumb-in-dike disaster recovery (PEP 374 did a pretty good job on that, if I do say so myself).
Well, I hope you are right. I'm looking forward to the new version of the test repository and doing some real world experiments.
I wish I had more time to do real work on this (not to mention email, thank *you*, David!), but it seems like every time I start
You are welcome; thanks for the feedback. (I sometimes feel like I'm working in a bit of a vacuum, though Barry does chime in occasionally...but I do realize that people are busy; that's why I inherited this job in the first place, after all :) -- R. David Murray www.bitdance.com
R. David Murray writes:
I thought Amaury was saying it was harder than svnmerge, not harder than patching by hand (ie: that it *was* patching by hand, including .rej files and the lack of tracking). I also heard Georg say that this should be fixable, and that he's partly fixed the tracking issue,
I don't see why the tracking issue is any different than it would be for svn. If you're actually merging, either a dummy merge (what git calls an "ours merge") or reverting unwanted patches will "block" them, and hg will keep track of the ones that have been merged. If you're actually cherry-picking, then you have to keep a separate database of the patches that have been picked and those that are blocked, but this has been done successfully for years with svnmerge, right?
but not the patch/merge issue (and that doing so will require a change in the hg core).
I don't think so. For merges, there isn't a problem. For cherypicking, I haven't thought carefully about this, but ISTM that "hg export | hg import; merge post-patch /dev/null source" should give the traditional conflict markers. This will require a bit of care to get the files to merge right, since there will in general be multiple files that fail to patch, but the names can be fished out of the .rej file(s). Getting the source files will also be mildly tricky, since they live in a different branch, and aren't available in your local repository. It will also require a bit of care to get the commit log etc right. But I don't think it's that hard, conceptually. Of course it's preferable to get this feature in hg itself, but I don't think we need to wait for hg/maintain a fork.
Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :)
Sure, but in this crowd, I wouldn't waste a good worry on this particular SMOP.
Well, there will be *some* workflow change since we're talking about committing to 3.2 maint before the new trunk instead of vice versa. But you are right that that is, mostly, a detail.
AFAIK that's not hg-driven, though.
I'm not as convinced that changes in workflow will be that minimal, though. I don't have much experience with the dvcses yet, but what experience I have had leads me to think that understanding the DAG is a non-trivial and necessary mental leap and does affect the workflow.
Yes, no, and yes. That is, although understanding it is nontrivial, and once you do it will affect your workflow, it is unnecessary. The Emacs crew have proved that to my satisfaction; there are a bunch of folks there who understand DAGs quite well, but there are also a bunch who just don't want to know -- and they're doing fine. True, making the most of dVCS requires "understanding the DAG." Adapting an existing complex workflow to a new dVCS's commands also requires understanding the DAG. But using the adapted workflow simply requires learning new names for old operations. Annoying, but it will make a fair number of core devs quite happy.
Well, I hope you are right. I'm looking forward to the new version of the test repository and doing some real world experiments.
Yup. It's always risky to predict, especially the future<wink>, but I'm pretty confident that things will work out. We do need to (1) make sure we do everything for hg that we've always done for svn, (2) work out some features that hg doesn't have yet (Windows EOLs), and (3) listen carefully to those who are testing out the new repository -- there are always surprises in this kind of thing. But Python does that kind of thing very well.
Am 30.12.2010 18:54, schrieb Stephen J. Turnbull:
R. David Murray writes:
I thought Amaury was saying it was harder than svnmerge, not harder than patching by hand (ie: that it *was* patching by hand, including .rej files and the lack of tracking). I also heard Georg say that this should be fixable, and that he's partly fixed the tracking issue,
I don't see why the tracking issue is any different than it would be for svn. If you're actually merging, either a dummy merge (what git calls an "ours merge") or reverting unwanted patches will "block" them, and hg will keep track of the ones that have been merged.
Are you still talking about merges from 3.x to 2.7? Then I don't think what you say is actually true. hg will *not* be able to track the ones that get merged, and will *not* be able to block by means of dummy merges. Perhaps we aren't "actually" merging, I suppose.
If you're actually cherry-picking, then you have to keep a separate database of the patches that have been picked and those that are blocked, but this has been done successfully for years with svnmerge, right?
[Ok. So "cherry-picking" is not a special case of "merging", or "actually merging", I presume] Wrong. For subversion, merge tracking is built into the tools that we use for merging (i.e. svnmerge). For hg (IIUC), the standard procedure that people use for merging (or "cherry-picking") apparently does not track what has been merged (or "cherry-picked").
Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :)
Sure, but in this crowd, I wouldn't waste a good worry on this particular SMOP.
Given how little enthusiasm this crowd has in actually helping the migration, I doubt they show any enthusiasm in writing such tools.
Well, there will be *some* workflow change since we're talking about committing to 3.2 maint before the new trunk instead of vice versa. But you are right that that is, mostly, a detail.
AFAIK that's not hg-driven, though.
It is. So far, we have *always* merged from the development branch to the maintenance branch (i.e. "backported"), although there has been opposition from a number of committers to this workflow in recent years. I personally consider this more appropriate: with backporting, you can defer decision to backport until the original change has been confirmed as correct. With constant forward-porting, you risk breaking your maintenance branch with incorrect changes.
I'm not as convinced that changes in workflow will be that minimal, though. I don't have much experience with the dvcses yet, but what experience I have had leads me to think that understanding the DAG is a non-trivial and necessary mental leap and does affect the workflow.
But using the adapted workflow simply requires learning new names for old operations. Annoying, but it will make a fair number of core devs quite happy.
I think the new workflow will simply result in (even) less care for the maintenance branches than currently. Some people just refuse to patch multiple branches, and will continue to do so. While it was previously possible to backport, it won't be that easy anymore in the future, so it just won't be done. This actually reduces workload, but also reduces quality. Regards, Martin
Am 30.12.2010 19:31, schrieb "Martin v. Löwis":
But using the adapted workflow simply requires learning new names for old operations. Annoying, but it will make a fair number of core devs quite happy.
I think the new workflow will simply result in (even) less care for the maintenance branches than currently. Some people just refuse to patch multiple branches, and will continue to do so. While it was previously possible to backport, it won't be that easy anymore in the future, so it just won't be done.
I refuse to believe that we cannot make our committers (in particular the active ones, which is quite a small percentage) follow such simple rules, especially when they have to learn a new VCS anyway... Also, it's not really necessary for everyone to merge every change from maintenance to trunk: a) they can do multiple commits on the same subject and merge once, and b) if the change is small and no problems can be expected from merging, merging can also be done by others, just like the mass merging we had once from trunk -> py3k (just more convenient for the one doing the merge). Georg
Also, it's not really necessary for everyone to merge every change from maintenance to trunk: a) they can do multiple commits on the same subject and merge once, and b) if the change is small and no problems can be expected from merging, merging can also be done by others, just like the mass merging we had once from trunk -> py3k (just more convenient for the one doing the merge).
But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right? Regards, Martin
Am 30.12.2010 22:38, schrieb "Martin v. Löwis":
Also, it's not really necessary for everyone to merge every change from maintenance to trunk: a) they can do multiple commits on the same subject and merge once, and b) if the change is small and no problems can be expected from merging, merging can also be done by others, just like the mass merging we had once from trunk -> py3k (just more convenient for the one doing the merge).
But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right?
Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes. Georg
On 12/30/2010 4:44 PM, Georg Brandl wrote:
But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right?
Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes.
I have the impression that Benjamin plans to continue 3.1 bug maintenance for months after 3.2 final rather than issue the last bug fix the traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if 3.1.4 was released along with 3.2, so most of us would never have to bother with 3.1 and hg together. -- Terry Jan Reedy
2010/12/30 Terry Reedy <tjreedy@udel.edu>:
On 12/30/2010 4:44 PM, Georg Brandl wrote:
But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right?
Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes.
I have the impression that Benjamin plans to continue 3.1 bug maintenance for months after 3.2 final rather than issue the last bug fix the traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if 3.1.4 was released along with 3.2, so most of us would never have to bother with 3.1 and hg together.
I will try to release 3.1.4 soon after 3.2 is released, but February and the spring in general will be quite busy for me. -- Regards, Benjamin
On Fri, 31 Dec 2010 02:54:26 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
I don't see why the tracking issue is any different than it would be for svn. If you're actually merging, either a dummy merge (what git
Martin already said most of what I would have in response to your post.
For cherypicking, I haven't thought carefully about this, but ISTM that "hg export | hg import; merge post-patch /dev/null source" should give the traditional conflict markers. This will require a bit of care to get the files to merge right, since there will in general be multiple files that fail to patch, but the names can be fished out of the .rej file(s). Getting the source files will also be mildly tricky, since they live in a different branch, and aren't available in your local repository. It will also require a bit of care to get the commit log etc right. But I don't think it's that hard, conceptually.
Of course it's preferable to get this feature in hg itself, but I don't think we need to wait for hg/maintain a fork.
The fact that I really haven't a clue what you are talking about here means that I for one am not likely to be helping develop that tool, at least not any time soon. So I hope there are people who understand this stuff who will make it work for the rest of us.
Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :)
Sure, but in this crowd, I wouldn't waste a good worry on this particular SMOP.
Talent is one thing, available time, as you pointed out about yourself, is a different matter. I'm confident we can make this all work. The only question is when. -- R. David Murray www.bitdance.com
On Dec 30, 2010, at 02:50 AM, R. David Murray wrote:
You are welcome; thanks for the feedback. (I sometimes feel like I'm working in a bit of a vacuum, though Barry does chime in occasionally...but I do realize that people are busy; that's why I inherited this job in the first place, after all :)
It's you're own fault for doing such a damn good job. :) -Barry
Am 29.12.2010 01:13, schrieb Amaury Forgeot d'Arc:
Hello,
The PyPy project recently switched from svn to mercurial. Since this day I have some difficulties to perform simple tasks, and my questions did not receive satisfying answers.
I was sure the Python project would have the same issues; so I started from the repositories in http://code.python.org/hg/ and tried simple merges between Python versions. I would like several people to try the same example, and tell how they handle it. I'm new to mercurial, and I may have missed something important!
Let's say you have to do the simple change shown in the diff below, and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. The diff was made against py3k.
How would you do it with Mercurial? Please try to do it for real!
"hg merge" is not the correct command: it merges whole branches, a change comes with all its ancestors. What we want is to "cherry-pick" a single change.
"hg transplant" fails to apply the change to release31-maint because of a tiny conflict (in the diff context!) that leaves you with an ugly "hunks FAILED" and a .rej file you have to parse and apply by hand.
Yes, this has been an irritation to me as well. It's probably not so hard for a Mercurial coredev to change transplant into generating inline conflict markers though. BTW, we would apply the diff in release31-maint and then "hg merge" it to py3k. transplant still applies to 2.7 though.
"hg transplant" seems to succeed in release27-maint, but the test fails to run because "support" should be changed to "test_support". The change is already committed - to fix it another changeset is needed. This does not look clean to me: both changesets will be pushed to the repository, and diff review (on the python-checkins mailing list) is almost impossible.
Right. Seems to me transplant should get a command-line switch that always goes into break-and-continue mode, where the commit is only made after calling "hg transplant --continue".
Furthermore, "hg transplant" does not keep track of integrations. There is a "transplants" file, but it stays in my local clone.
You're wrong, it does keep track of integrations in the commit metadata. The only thing that's clone-local is the transplants cache file. Sadly, transplant only checks against this file, but it is very easy to write a hook that keeps it up to date; I've already done that work.
Faced with a similar case in pypy, we finally manually copied the files between directories... and the fastest with our example is probably to do the change manually in all three directories.
There is surely a better way to work with maintenance branches! PEP374 suggests to first modify the oldest release, but this does not seems right to me (I have three reasons in mind)
Would you care to explain those?
At the very least before the migration we need precise use cases like this and recipes for common cases.
Which is what we'll certainly get once we have the test repo and Brett is rewriting the dev guide. Georg
Hi, One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool? I for one immensely like Mercurial’s merge and utterly dislike Subversion’s (certainly because I’ve learned a good merge tool and don’t have a good editor to handle files with conflict markers).
There is surely a better way to work with maintenance branches! PEP374 suggests to first modify the oldest release, but this does not seems right to me (I have three reasons in mind) Can you give them?
Regards
On Thu, 30 Dec 2010 20:57:11 +0100 Éric Araujo <merwok@netwok.org> wrote:
Hi,
One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool? I for one immensely like Mercurial’s merge and utterly dislike Subversion’s (certainly because I’ve learned a good merge tool and don’t have a good editor to handle files with conflict markers).
IIUC this is not about merges, this is about transplants (or cherry-picking). It would be unannoying to lose svnmerge's relative convenience in that matter. Regards Antoine.
2010/12/30 Antoine Pitrou <solipsis@pitrou.net>:
On Thu, 30 Dec 2010 20:57:11 +0100 Éric Araujo <merwok@netwok.org> wrote:
Hi,
One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool? I for one immensely like Mercurial’s merge and utterly dislike Subversion’s (certainly because I’ve learned a good merge tool and don’t have a good editor to handle files with conflict markers).
IIUC this is not about merges, this is about transplants (or cherry-picking). It would be unannoying to lose svnmerge's relative convenience in that matter.
I assume you meant "annoying"? -- Regards, Benjamin
Le jeudi 30 décembre 2010 à 14:24 -0600, Benjamin Peterson a écrit :
2010/12/30 Antoine Pitrou <solipsis@pitrou.net>:
On Thu, 30 Dec 2010 20:57:11 +0100 Éric Araujo <merwok@netwok.org> wrote:
Hi,
One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool? I for one immensely like Mercurial’s merge and utterly dislike Subversion’s (certainly because I’ve learned a good merge tool and don’t have a good editor to handle files with conflict markers).
IIUC this is not about merges, this is about transplants (or cherry-picking). It would be unannoying to lose svnmerge's relative convenience in that matter.
I assume you meant "annoying"?
Well, yes. I should stop twisting my negatives (whatever it means). Antoine.
Am 30.12.2010 20:57, schrieb Éric Araujo:
Hi,
One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool?
Neither nor. They come from "hg merge" being useless when it comes to having code that is meant to live both on 2.7, 3.2 (perhaps) and 3.3. Regards, Martin
participants (12)
-
"Martin v. Löwis" -
Amaury Forgeot d'Arc -
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
David Cournapeau -
Dirkjan Ochtman -
Georg Brandl -
R. David Murray -
Stephen J. Turnbull -
Terry Reedy -
Éric Araujo