Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another. I strongly disagree with the second one. 

On Jan 18, 2016, at 12:21 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:

Just a few thoughts about this.

I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.

That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?

While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.

On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,

The last couple weeks I’ve been thinking a lot about the future of yt.
What I’d like to propose is that we shift investment in yt as a
single, monolithic codebase into yt as a project, or an ecosystem of
projects.

This came out of the discussion of the extension/affiliated packages,
analysis modules, and so on.  Britton has for a while been pitching
the idea [which I will poorly paraphrase here] that yt can be the
framework on top of which killer apps can be built.  I think this is
great.

What’s holding us back in some ways from this is that yt is currently
structured as a monolithic code base, with little to no discovery of
other packages and apps and whatnot.  We tried for a while to change
this with The Barn, but it ended up not quite taking off.  I think the
time is right to try to change the way we think about yt to be more
about yt the Project, rather than yt the Codebase; the core codebase
is an important component of this, but not the whole of it.

Encouraging an ecosystem of packages can have a few very important benefits:

 * External packages will confer greater individual credit to the
folks who develop them.
 * External packages can be versioned and developed independently; the
review process can be different.
 * yt’s core can be emphasized as a generic package, on top of which
astronomy analysis can be built.
 * Packages can be maintained wherever, including alternate locations
such as github.

On the other hand, having packages inside the main distribution makes
discoverability much, much easier.  It also enables everything to be
In The Box.  And, the continuous integration and testing system is
already set up for yt.  But, these are all possible to overcome -- we
can devise a strategy for adding packages to the CI system (and if
they are externally managed, they can also rely on yt as a dependency
and use whatever CI system they like!) and we can improve
discoverability by refocusing the website to enable this.  I've asked
Kacper about adding new packages, and it's not as easy as it might
seem, so we may need to be careful about how that process occurs; one
possibility would be to provide servers and ready-made setups, but
have individuals do the heavy lifting.  We could even have something
in the codebase that describes some packages that are available.
External packages could have much looser dependency rules, which means
they can be free to take advantage of things like OpenCL, numba, etc,
without having to add them to the primary codebase.

Synchronizing APIs and versions across extension packages may be
difficult in some particular cases, but I suspect in practice will not
be an issue, as long as we continue to have a reasonably stable
*public* API, and graduate a few things (such as .blocks) into a
public API from semi-private.

To this end, of really encouraging an ecosystem of packages, I’d like
to propose two things, in increasing order of disruptiveness.

First: Encourage extension packages.  This would mean:
 * Reorganize website to allow for extension packages to be displayed
prominently
 * Add support for name-space packages in yt
 * (possible) split out some packages from analysis_modules, including
halo finding
 * Codify process of extension package creation, including how to have
CI set up for them and build system.

The second, more disruptive proposal:
 * Split yt into subprojects.  This would include spinning out the
volume rendering and some or all of the frontends, and probably the
testing infrastructure as well.
 * Split further astro-specific routines into an astro extension, and
begin the process of doing this with other domains as well.  (As in
the long-simmering domain context YTEP.)

I’ll invite comments from everyone, but particularly from folks who
have either not contributed to an analysis module or extension package
because of concerns that would be addressed by this, as well as from
core developers this would impact.  If the thread gets too unweildy we
may also want to table this for the next yt team meeting.

Thanks,

Matt
_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org