So in practice, if we implemented this plan, that would mean adding a "development" bookmark to the "main" head of the yt branch, and renaming "experimental" to "vr_refactor".

I guess I'm fine with this, but I don't think the effort of maintaining the "development" bookmark will be worth it. Even if most pull requests go in during the triage hangout, we'd still need to remember to manually update it. Just maintaining the "@" bookmark and the "experimental" bookmark as is is a struggle. Just the other day, I noticed that "@" wasn't updated after the bugfix backports were last merged in.

On Thu, Sep 10, 2015 at 3:02 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Nathan, Cameron,

Ah, yes, I did not understand the point you made about bookmarks not being updated.  That's a very valid concern.  I guess my response to this would be that since acceptance of most large PRs is happening during PR triage hangout these days, that we just make sure that someone in the hangout updates the bookmark manually as part of the merger process.

As far as confusion from updating, I think that having things update to the @ bookmark upon cloning takes care of most of the potential problems.  For the most part, users can update with "yt update" and rarely have to use hg at all.  Most of the time, I would imagine that "hg update" will only be used by people doing development and that in that case, there is an expectation of some prior mercurial knowledge.

Britton

On Thu, Sep 10, 2015 at 3:50 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:


On Thu, Sep 10, 2015 at 2:19 PM, Matthew Turk <matthewturk@gmail.com> wrote:


On Thu, Sep 10, 2015 at 2:18 PM, Cameron Hummels <chummels@gmail.com> wrote:
In general, I like Britton's idea as it fosters more group development as opposed to the problems of forks of forks that many of us experience in trying to develop unmerged features like VR-refactor and ytdata frontend.  

That said, there are some issues to be aware of.  By default, I think when you pull from the repository, hg defaults to updating to the most recent commit (ie the tip), which may not be in your active bookmark.  This means that if someone just committed to the "ytdata frontend" head/bookmark, and I pull from the main repository expecting to get the main "development" bookmark, I'm going to be put on the wrong version of the code.  This can be confusing unless we educate everyone to always make sure after they pull to update to the "development" bookmark or whatever.

As a quick note, this isn't true -- it updates to the "@" bookmark.

I have more thoughts on the other emails as well, but wanted to take this quick opportunity to be pedantic and annoying.

To be pedantically pedantic, this isn't true, and I think Cameron's was correct. The "@" bookmark is only considered when cloning a repository. Bare "hg update" will update to the tip of the named branch, even if that tip is on a different topological branch.
 
 

I'm dubious of the idea of merging more work-in-progress code into the main head of the dev branch as Nathan suggests.  As he points out, the success of this model is entirely dependent on the author/community "finishing" code that goes into the dev branch before it gets tagged as a stable release.  All this does is simply kick the can down the road as to when code gets "finished" (i.e. documented, tested, polished, the parts no one wants to do).  By delaying this process until there is a deadline (the stable release) it separates it from when the main development went on, and makes it less likely for authors to remember what was going on in the code.  In the past, this results in code kinda getting pushed through, just to not hold up the release any longer, and it ends up being unnecessarily acrimonious.  I'm in favor or the separate heads/bookmarks model that Britton suggests, but not merging those heads into the main dev bookmark until it is "finished".

Cameron


On Thu, Sep 10, 2015 at 12:01 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a first initial reaction: I don't think introducing even more heads to the yet branch is a good idea, even if they are marked by bookmarks. In fact I'd prefer to only have one head on the yt branch right now.

In addition, bitbucket's support for bookmark's in the pull request interface amounts only to labeling entries in the source and destination dropdown menu, meaning that once a pull request is merged, someone would need to manually update the corresponding bookmark to the correct head.  This is both error prone and easy to skip, so I predict that if we *did* start doing this, bookmarks would wuickly go out of sync with heads, leaving a bunch of un-bookmarked heads.

Was this prompted by your work on the reloadable datasets PR? Without something like this do you expect it to sit in limbo for a long time?

I think the solution we need is to jot be afraid to pull in more "experimental" work-in-progress stuff to the yt branch. Of course that would need to come with assurances from the author or community members that they're committed to "finishing" the code before the next major release. That way people will be able to test new code more easily (they'll get it without updating explicitly to it) and we'll catch breakages to unrelated code and wordflows more quickly.

Nathan


On Thursday, September 10, 2015, Britton Smith <brittonsmith@gmail.com> wrote:
Hi everyone,

I had some ideas for improving the yt development process that I
wanted to run by everyone.  This can be discussed further at our
upcoming team meeting and if people are in favor, I will issue a pull
request to the relevant YTEP.

STATEMENT OF PROBLEM
Currently, development proceeds roughly as follows.  The two main
active branches within the central yt repository are yt and stable.
The tip of stable is the latest release and the yt branch is the de
facto "development version" of the code.  Until recently, we have not
been very good at regularly scheduled minor releases and so the stable
branch sits for quite some time with many bugs that are fixed within
the development branch.  This effectively makes stable unusable and
pushes most users to the yt branch.

When new features are developed, pull requests are issued to the
single head of the yt branch.  Because this is the version most people
are actually using, the current policy is to not allow PR with new
functionality to be accepted until they are 100% ready (full
functionality, tests, docs, etc).  As we have already seen, this makes
collaborative development very cumbersome, as it requires people to
create forks of the fork from which the PR originates.  They then must
issue PRs to that fork after which time the original PR is updated.
The current volume render refactor is the perfect example of this.

PROPOSED SOLUTION
Before I lay out the proposed solution, I want to list a number of
recent developments that I think will make this possible:
1. Nathan's new script for backporting changes now keeps stable and yt
   synced on bugfixes.
2. We have returned to doing minor releases containing only bugfixes,
   thanks again to Nathan's hard work.  This and point 1 means that
   users are once again safe to be on stable, and now should be there
   most of the time.
3. Bitbucket now supports bookmarks, meaning that PRs can be issued to
   specific bookmarks instead of to branches or heads named only by the
   changeset hash.
4. The weekly PR triage hangouts are making it easier to process PRs
   and also providing a place to strategize getting larger PRs
   accepted.  Thanks to Hilary for keeping this going.

With the above in mind, I propose the following:
1. Create a "development" bookmark to sit at the tip of the yt
   branch.  All PRs containing relatively small new features are
   issued to this.  The requirements for acceptance remain the same:
   PRs accepted to "development" must contain all intended
   functionality and be fully documented.  This allows the
   "development" bookmark to be defined explicitly as everything that
   will be included in the next major release.
2. Large new features should have a corresponding YTEP that has been
   accepted.  After the YTEP has been accepted, a PR should be issued
   to the yt branch.  After some initial discussion, this PR is pulled
   into the main yt repo with a bookmark named after the feature.
   Once this has happened, developers can now issue new PRs
   specifically to this bookmark.  This is effectively what we have
   now with the volume render work in the "experimental" bookmark,
   only we would rename the bookmark to something like "vr-refactor".
   As with PRs issued directly to "development", only after the new
   feature is 100% ready shall it be merged into the "development"
   bookmark.
3. We continue to make use of the PR triage hangouts to establish when
   large features are ready to be merged.

I believe this will have the following benefits:
1. Large, new features can be developed collaboratively without the
   need for forks of forks of forks.
2. New, underdevelopment features are more accessible to the larger
   community by simply updating to named bookmarks from the main repo
   (no need for "just pull these changes from my fork").
3. The "development" branch is preserved as a place only for
   ready-to-be-released features (i.e., polished and documented).


All told, this is really just a small tweak on our current process.
Please comment with any thoughts, or even a +/-1 if your feelings can
be summed up thusly.

Britton

_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org




--
Cameron Hummels
NSF Postdoctoral Fellow
Department of Astronomy
California Institute of Technology

_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org