Hi everyone, The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects. This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great. What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it. Encouraging an ecosystem of packages can have a few very important benefits: * External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github. On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase. Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private. To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness. First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system. The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.) I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting. Thanks, Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt and everyone, I support your first option of extension packages. I think the second option is far too disruptive. I’m working on two small packages that fit this description (no documentation or tests as of yet): https://github.com/jzuhone/formulas <https://github.com/jzuhone/formulas> https://github.com/jzuhone/equilibrate <https://github.com/jzuhone/equilibrate> They both depend on yt, but in my view are too domain-specific and/or specialized to be incorporated into yt itself right now. Then of course there is YT.jl, which does have documentation and testing (but is also a Julia package): https://github.com/jzuhone/YT.jl <https://github.com/jzuhone/YT.jl> These are just some examples of what external packages might look like. Best, John
On Jan 18, 2016, at 10:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Just a few thoughts about this. I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users. That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release? While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users. On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another. I strongly disagree with the second one.
On Jan 18, 2016, at 12:21 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com <mailto:matthewturk@gmail.com>> wrote: Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org <mailto:yt-dev@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org>
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan, Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into. On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another. I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> wrote:
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey All, Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so, then I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions. -d On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Matt can correct me if I am wrong, but I was under the impression that
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> wrote: the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly
disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Desika, Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase. Flask's extensions are another good example of how this type of thing is done: http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/ -Matt On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote:
Hey All,
Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so, then I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> wrote:
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would this constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
The last couple weeks I’ve been thinking a lot about the future of yt. What I’d like to propose is that we shift investment in yt as a single, monolithic codebase into yt as a project, or an ecosystem of projects.
This came out of the discussion of the extension/affiliated packages, analysis modules, and so on. Britton has for a while been pitching the idea [which I will poorly paraphrase here] that yt can be the framework on top of which killer apps can be built. I think this is great.
What’s holding us back in some ways from this is that yt is currently structured as a monolithic code base, with little to no discovery of other packages and apps and whatnot. We tried for a while to change this with The Barn, but it ended up not quite taking off. I think the time is right to try to change the way we think about yt to be more about yt the Project, rather than yt the Codebase; the core codebase is an important component of this, but not the whole of it.
Encouraging an ecosystem of packages can have a few very important benefits:
* External packages will confer greater individual credit to the folks who develop them. * External packages can be versioned and developed independently; the review process can be different. * yt’s core can be emphasized as a generic package, on top of which astronomy analysis can be built. * Packages can be maintained wherever, including alternate locations such as github.
On the other hand, having packages inside the main distribution makes discoverability much, much easier. It also enables everything to be In The Box. And, the continuous integration and testing system is already set up for yt. But, these are all possible to overcome -- we can devise a strategy for adding packages to the CI system (and if they are externally managed, they can also rely on yt as a dependency and use whatever CI system they like!) and we can improve discoverability by refocusing the website to enable this. I've asked Kacper about adding new packages, and it's not as easy as it might seem, so we may need to be careful about how that process occurs; one possibility would be to provide servers and ready-made setups, but have individuals do the heavy lifting. We could even have something in the codebase that describes some packages that are available. External packages could have much looser dependency rules, which means they can be free to take advantage of things like OpenCL, numba, etc, without having to add them to the primary codebase.
Synchronizing APIs and versions across extension packages may be difficult in some particular cases, but I suspect in practice will not be an issue, as long as we continue to have a reasonably stable *public* API, and graduate a few things (such as .blocks) into a public API from semi-private.
To this end, of really encouraging an ecosystem of packages, I’d like to propose two things, in increasing order of disruptiveness.
First: Encourage extension packages. This would mean: * Reorganize website to allow for extension packages to be displayed prominently * Add support for name-space packages in yt * (possible) split out some packages from analysis_modules, including halo finding * Codify process of extension package creation, including how to have CI set up for them and build system.
The second, more disruptive proposal: * Split yt into subprojects. This would include spinning out the volume rendering and some or all of the frontends, and probably the testing infrastructure as well. * Split further astro-specific routines into an astro extension, and begin the process of doing this with other domains as well. (As in the long-simmering domain context YTEP.)
I’ll invite comments from everyone, but particularly from folks who have either not contributed to an analysis module or extension package because of concerns that would be addressed by this, as well as from core developers this would impact. If the thread gets too unweildy we may also want to table this for the next yt team meeting.
Thanks,
Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Affiliated packages sound like a good idea to me, but I am a bit hesitant about splitting the volume renderer and frontends out of yt's core. On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Desika,
Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase.
Flask's extensions are another good example of how this type of thing is done:
http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/
-Matt
Hey All,
Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so,
I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com>
wrote:
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components
seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Just a few thoughts about this.
I think encouraging more external packages to depend on yt will be great, and will keep us more honest about the backward compatibility guarantees we've committed ourselves to. Building social capital by encouraging an "affiliated" package ecosystem like astropy has is also good, both for social capital and also by enabling cool new features for our users.
That said, I'm skeptical about some of the technical components of Matt's proposal. In particular, I'm concerned with the amount of work required to make major changes to the core yt codebase, while still maintaining backward compatibility. Who will do this work? When will it happen? Would
constitute a yt 4.0 release?
While I do think making the core yt package less astronomy focused is good, I also think it's possible to do so while maintaining strong backward compatibility guarantees, or at least going through a deprecation cycle over the course of several minor versions. I don't think making another cycle of breaking changes so soon after yt-3.0 is a good idea, as that will impose a large amount of work both on the yt developers and on our users.
On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk < matthewturk@gmail.com> wrote: > > Hi everyone, > > The last couple weeks I’ve been thinking a lot about the future of > yt. > What I’d like to propose is that we shift investment in yt as a > single, monolithic codebase into yt as a project, or an ecosystem of > projects. > > This came out of the discussion of the extension/affiliated
> analysis modules, and so on. Britton has for a while been pitching > the idea [which I will poorly paraphrase here] that yt can be the > framework on top of which killer apps can be built. I think this is > great. > > What’s holding us back in some ways from this is that yt is currently > structured as a monolithic code base, with little to no discovery of > other packages and apps and whatnot. We tried for a while to change > this with The Barn, but it ended up not quite taking off. I think > the > time is right to try to change the way we think about yt to be more > about yt the Project, rather than yt the Codebase; the core codebase > is an important component of this, but not the whole of it. > > Encouraging an ecosystem of packages can have a few very important > benefits: > > * External packages will confer greater individual credit to the > folks who develop them. > * External packages can be versioned and developed independently;
> review process can be different. > * yt’s core can be emphasized as a generic package, on top of which > astronomy analysis can be built. > * Packages can be maintained wherever, including alternate locations > such as github. > > On the other hand, having packages inside the main distribution makes > discoverability much, much easier. It also enables everything to be > In The Box. And, the continuous integration and testing system is > already set up for yt. But, these are all possible to overcome -- we > can devise a strategy for adding packages to the CI system (and if > they are externally managed, they can also rely on yt as a dependency > and use whatever CI system they like!) and we can improve > discoverability by refocusing the website to enable this. I've asked > Kacper about adding new packages, and it's not as easy as it might > seem, so we may need to be careful about how that process occurs; one > possibility would be to provide servers and ready-made setups, but > have individuals do the heavy lifting. We could even have something > in the codebase that describes some packages that are available. > External packages could have much looser dependency rules, which > means > they can be free to take advantage of things like OpenCL, numba, etc, > without having to add them to the primary codebase. > > Synchronizing APIs and versions across extension packages may be > difficult in some particular cases, but I suspect in practice will > not > be an issue, as long as we continue to have a reasonably stable > *public* API, and graduate a few things (such as .blocks) into a > public API from semi-private. > > To this end, of really encouraging an ecosystem of packages, I’d
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote: then like that this packages, the like
> to propose two things, in increasing order of disruptiveness. > > First: Encourage extension packages. This would mean: > * Reorganize website to allow for extension packages to be displayed > prominently > * Add support for name-space packages in yt > * (possible) split out some packages from analysis_modules, including > halo finding > * Codify process of extension package creation, including how to have > CI set up for them and build system. > > The second, more disruptive proposal: > * Split yt into subprojects. This would include spinning out the > volume rendering and some or all of the frontends, and probably the > testing infrastructure as well. > * Split further astro-specific routines into an astro extension, and > begin the process of doing this with other domains as well. (As in > the long-simmering domain context YTEP.) > > I’ll invite comments from everyone, but particularly from folks who > have either not contributed to an analysis module or extension > package > because of concerns that would be addressed by this, as well as from > core developers this would impact. If the thread gets too unweildy > we > may also want to table this for the next yt team meeting. > > Thanks, > > Matt > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 *phone*: 631-632-8225 *e-mail*: Michael.Zingale@stonybrook.edu *web*: http://www.astro.sunysb.edu/mzingale github: http://github.com/zingale _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I guess this is clear from the original email, but I very much like the idea of an ecosystem of a core yt package and separate extension packages. Personally, I would stop short of making frontends into external packages. I think one of the big values of yt is the library of frontends we maintain and the ability of people to install the source and immediately be able to load their data without further requirements. I think we run the risk of losing new users who have to go through too many steps just to get their data loaded. Perhaps the first thing should be to start up the domain context efforts again. By some later 3.x release, we could have an astro domain with domain specific fields and extensions all still within the main yt codebase. Then, moving to 4.0, we start to extract things into separate packages. On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale < michael.zingale@stonybrook.edu> wrote:
Affiliated packages sound like a good idea to me, but I am a bit hesitant about splitting the volume renderer and frontends out of yt's core.
On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Desika,
Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase.
Flask's extensions are another good example of how this type of thing is done:
http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/
-Matt
Hey All,
Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so,
I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com>
wrote:
Matt can correct me if I am wrong, but I was under the impression
the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components
seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
Just to quickly address one component before anybody else replies: I would volunteer to do the majority of this work (which I think from a technical perspective is not necessarily too bad), and I don't see it being a 3.X release, but maybe a 4.0 release, and I don't have answers for when it would happen since I think that is a community decision we shouldn't rush into.
On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote: > Just a few thoughts about this. > > I think encouraging more external packages to depend on yt will be > great, > and will keep us more honest about the backward compatibility > guarantees > we've committed ourselves to. Building social capital by encouraging > an > "affiliated" package ecosystem like astropy has is also good, both for > social capital and also by enabling cool new features for our users. > > That said, I'm skeptical about some of the technical components of > Matt's > proposal. In particular, I'm concerned with the amount of work > required to > make major changes to the core yt codebase, while still maintaining > backward > compatibility. Who will do this work? When will it happen? Would
> constitute a yt 4.0 release? > > While I do think making the core yt package less astronomy focused is > good, > I also think it's possible to do so while maintaining strong backward > compatibility guarantees, or at least going through a deprecation > cycle over > the course of several minor versions. I don't think making another > cycle of > breaking changes so soon after yt-3.0 is a good idea, as that will > impose a > large amount of work both on the yt developers and on our users. > > On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk < matthewturk@gmail.com> > wrote: >> >> Hi everyone, >> >> The last couple weeks I’ve been thinking a lot about the future of >> yt. >> What I’d like to propose is that we shift investment in yt as a >> single, monolithic codebase into yt as a project, or an ecosystem of >> projects. >> >> This came out of the discussion of the extension/affiliated
>> analysis modules, and so on. Britton has for a while been
>> the idea [which I will poorly paraphrase here] that yt can be the >> framework on top of which killer apps can be built. I think this is >> great. >> >> What’s holding us back in some ways from this is that yt is currently >> structured as a monolithic code base, with little to no discovery of >> other packages and apps and whatnot. We tried for a while to change >> this with The Barn, but it ended up not quite taking off. I think >> the >> time is right to try to change the way we think about yt to be more >> about yt the Project, rather than yt the Codebase; the core codebase >> is an important component of this, but not the whole of it. >> >> Encouraging an ecosystem of packages can have a few very important >> benefits: >> >> * External packages will confer greater individual credit to the >> folks who develop them. >> * External packages can be versioned and developed independently;
>> review process can be different. >> * yt’s core can be emphasized as a generic package, on top of which >> astronomy analysis can be built. >> * Packages can be maintained wherever, including alternate locations >> such as github. >> >> On the other hand, having packages inside the main distribution makes >> discoverability much, much easier. It also enables everything to be >> In The Box. And, the continuous integration and testing system is >> already set up for yt. But, these are all possible to overcome -- we >> can devise a strategy for adding packages to the CI system (and if >> they are externally managed, they can also rely on yt as a dependency >> and use whatever CI system they like!) and we can improve >> discoverability by refocusing the website to enable this. I've asked >> Kacper about adding new packages, and it's not as easy as it might >> seem, so we may need to be careful about how that process occurs; one >> possibility would be to provide servers and ready-made setups, but >> have individuals do the heavy lifting. We could even have something >> in the codebase that describes some packages that are available. >> External packages could have much looser dependency rules, which >> means >> they can be free to take advantage of things like OpenCL, numba, etc, >> without having to add them to the primary codebase. >> >> Synchronizing APIs and versions across extension packages may be >> difficult in some particular cases, but I suspect in practice will >> not >> be an issue, as long as we continue to have a reasonably stable >> *public* API, and graduate a few things (such as .blocks) into a >> public API from semi-private. >> >> To this end, of really encouraging an ecosystem of packages, I’d
>> to propose two things, in increasing order of disruptiveness. >> >> First: Encourage extension packages. This would mean: >> * Reorganize website to allow for extension packages to be displayed >> prominently >> * Add support for name-space packages in yt >> * (possible) split out some packages from analysis_modules, including >> halo finding >> * Codify process of extension package creation, including how to have >> CI set up for them and build system. >> >> The second, more disruptive proposal: >> * Split yt into subprojects. This would include spinning out the >> volume rendering and some or all of the frontends, and probably
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote: then that like that this packages, pitching the like the
>> testing infrastructure as well. >> * Split further astro-specific routines into an astro extension, and >> begin the process of doing this with other domains as well. (As in >> the long-simmering domain context YTEP.) >> >> I’ll invite comments from everyone, but particularly from folks who >> have either not contributed to an analysis module or extension >> package >> because of concerns that would be addressed by this, as well as from >> core developers this would impact. If the thread gets too unweildy >> we >> may also want to table this for the next yt team meeting. >> >> Thanks, >> >> Matt >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor
Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 *phone*: 631-632-8225 *e-mail*: Michael.Zingale@stonybrook.edu *web*: http://www.astro.sunysb.edu/mzingale github: http://github.com/zingale
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all, Thanks for all your feedback. I think it seems like the proposal #1 is a good way to go. I'm going to try to start writing up a YTEP that formalizes this, and then we can put that up for a vote -- let's not consider anything "settled" yet, but it seems pretty reasonable to move forward with a more detailed proposal. If anyone else has comments, please feel free to leave them here, or on the YTEP. -Matt On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <brittonsmith@gmail.com> wrote:
I guess this is clear from the original email, but I very much like the idea of an ecosystem of a core yt package and separate extension packages. Personally, I would stop short of making frontends into external packages. I think one of the big values of yt is the library of frontends we maintain and the ability of people to install the source and immediately be able to load their data without further requirements. I think we run the risk of losing new users who have to go through too many steps just to get their data loaded.
Perhaps the first thing should be to start up the domain context efforts again. By some later 3.x release, we could have an astro domain with domain specific fields and extensions all still within the main yt codebase. Then, moving to 4.0, we start to extract things into separate packages.
On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale <michael.zingale@stonybrook.edu> wrote:
Affiliated packages sound like a good idea to me, but I am a bit hesitant about splitting the volume renderer and frontends out of yt's core.
On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Desika,
Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase.
Flask's extensions are another good example of how this type of thing is done:
http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/
-Matt
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote:
Hey All,
Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so, then I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> wrote:
Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another.
I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
> On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk@gmail.com> > wrote: > > Hi Nathan, > > Just to quickly address one component before anybody else replies: > I > would volunteer to do the majority of this work (which I think from > a > technical perspective is not necessarily too bad), and I don't see > it > being a 3.X release, but maybe a 4.0 release, and I don't have > answers > for when it would happen since I think that is a community decision > we > shouldn't rush into. > > On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum > <nathan12343@gmail.com> wrote: >> Just a few thoughts about this. >> >> I think encouraging more external packages to depend on yt will be >> great, >> and will keep us more honest about the backward compatibility >> guarantees >> we've committed ourselves to. Building social capital by >> encouraging >> an >> "affiliated" package ecosystem like astropy has is also good, both >> for >> social capital and also by enabling cool new features for our >> users. >> >> That said, I'm skeptical about some of the technical components of >> Matt's >> proposal. In particular, I'm concerned with the amount of work >> required to >> make major changes to the core yt codebase, while still >> maintaining >> backward >> compatibility. Who will do this work? When will it happen? Would >> this >> constitute a yt 4.0 release? >> >> While I do think making the core yt package less astronomy focused >> is >> good, >> I also think it's possible to do so while maintaining strong >> backward >> compatibility guarantees, or at least going through a deprecation >> cycle over >> the course of several minor versions. I don't think making another >> cycle of >> breaking changes so soon after yt-3.0 is a good idea, as that will >> impose a >> large amount of work both on the yt developers and on our users. >> >> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk >> <matthewturk@gmail.com> >> wrote: >>> >>> Hi everyone, >>> >>> The last couple weeks I’ve been thinking a lot about the future >>> of >>> yt. >>> What I’d like to propose is that we shift investment in yt as a >>> single, monolithic codebase into yt as a project, or an ecosystem >>> of >>> projects. >>> >>> This came out of the discussion of the extension/affiliated >>> packages, >>> analysis modules, and so on. Britton has for a while been >>> pitching >>> the idea [which I will poorly paraphrase here] that yt can be the >>> framework on top of which killer apps can be built. I think this >>> is >>> great. >>> >>> What’s holding us back in some ways from this is that yt is >>> currently >>> structured as a monolithic code base, with little to no discovery >>> of >>> other packages and apps and whatnot. We tried for a while to >>> change >>> this with The Barn, but it ended up not quite taking off. I >>> think >>> the >>> time is right to try to change the way we think about yt to be >>> more >>> about yt the Project, rather than yt the Codebase; the core >>> codebase >>> is an important component of this, but not the whole of it. >>> >>> Encouraging an ecosystem of packages can have a few very >>> important >>> benefits: >>> >>> * External packages will confer greater individual credit to the >>> folks who develop them. >>> * External packages can be versioned and developed independently; >>> the >>> review process can be different. >>> * yt’s core can be emphasized as a generic package, on top of >>> which >>> astronomy analysis can be built. >>> * Packages can be maintained wherever, including alternate >>> locations >>> such as github. >>> >>> On the other hand, having packages inside the main distribution >>> makes >>> discoverability much, much easier. It also enables everything to >>> be >>> In The Box. And, the continuous integration and testing system >>> is >>> already set up for yt. But, these are all possible to overcome >>> -- we >>> can devise a strategy for adding packages to the CI system (and >>> if >>> they are externally managed, they can also rely on yt as a >>> dependency >>> and use whatever CI system they like!) and we can improve >>> discoverability by refocusing the website to enable this. I've >>> asked >>> Kacper about adding new packages, and it's not as easy as it >>> might >>> seem, so we may need to be careful about how that process occurs; >>> one >>> possibility would be to provide servers and ready-made setups, >>> but >>> have individuals do the heavy lifting. We could even have >>> something >>> in the codebase that describes some packages that are available. >>> External packages could have much looser dependency rules, which >>> means >>> they can be free to take advantage of things like OpenCL, numba, >>> etc, >>> without having to add them to the primary codebase. >>> >>> Synchronizing APIs and versions across extension packages may be >>> difficult in some particular cases, but I suspect in practice >>> will >>> not >>> be an issue, as long as we continue to have a reasonably stable >>> *public* API, and graduate a few things (such as .blocks) into a >>> public API from semi-private. >>> >>> To this end, of really encouraging an ecosystem of packages, I’d >>> like >>> to propose two things, in increasing order of disruptiveness. >>> >>> First: Encourage extension packages. This would mean: >>> * Reorganize website to allow for extension packages to be >>> displayed >>> prominently >>> * Add support for name-space packages in yt >>> * (possible) split out some packages from analysis_modules, >>> including >>> halo finding >>> * Codify process of extension package creation, including how to >>> have >>> CI set up for them and build system. >>> >>> The second, more disruptive proposal: >>> * Split yt into subprojects. This would include spinning out the >>> volume rendering and some or all of the frontends, and probably >>> the >>> testing infrastructure as well. >>> * Split further astro-specific routines into an astro extension, >>> and >>> begin the process of doing this with other domains as well. (As >>> in >>> the long-simmering domain context YTEP.) >>> >>> I’ll invite comments from everyone, but particularly from folks >>> who >>> have either not contributed to an analysis module or extension >>> package >>> because of concerns that would be addressed by this, as well as >>> from >>> core developers this would impact. If the thread gets too >>> unweildy >>> we >>> may also want to table this for the next yt team meeting. >>> >>> Thanks, >>> >>> Matt >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor
Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 phone: 631-632-8225 e-mail: Michael.Zingale@stonybrook.edu web: http://www.astro.sunysb.edu/mzingale github: http://github.com/zingale
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi everyone, I have submitted a draft YTEP here: https://bitbucket.org/yt_analysis/ytep/pull-requests/60/ytep-0029-extension-... Please feel free to leave short comments there, but if they are larger comments, I think they should perhaps come here. I didn't discuss everything in it, but I think the biggest change is non-technical, and instead that of a mindset. Also, the website. -Matt On Tue, Jan 19, 2016 at 9:20 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
Thanks for all your feedback. I think it seems like the proposal #1 is a good way to go. I'm going to try to start writing up a YTEP that formalizes this, and then we can put that up for a vote -- let's not consider anything "settled" yet, but it seems pretty reasonable to move forward with a more detailed proposal. If anyone else has comments, please feel free to leave them here, or on the YTEP.
-Matt
I guess this is clear from the original email, but I very much like the idea of an ecosystem of a core yt package and separate extension packages. Personally, I would stop short of making frontends into external
I think one of the big values of yt is the library of frontends we
and the ability of people to install the source and immediately be able to load their data without further requirements. I think we run the risk of losing new users who have to go through too many steps just to get their data loaded.
Perhaps the first thing should be to start up the domain context efforts again. By some later 3.x release, we could have an astro domain with domain specific fields and extensions all still within the main yt codebase. Then, moving to 4.0, we start to extract things into separate packages.
On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale <michael.zingale@stonybrook.edu> wrote:
Affiliated packages sound like a good idea to me, but I am a bit
hesitant
about splitting the volume renderer and frontends out of yt's core.
On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Desika,
Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase.
Flask's extensions are another good example of how this type of thing
is
done:
http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/
-Matt
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote:
Hey All,
Matt - just to maybe clarify by examples...are you thinking things
Trident or powderday are sorts of yt extensions that you mean? (If so, then I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk < matthewturk@gmail.com> wrote:
On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> wrote: > Matt can correct me if I am wrong, but I was under the impression > that > the two components of the proposal were separate and not
inevitably
> tied to > one another. > > I am in strong support of the idea of extensions, but I strongly > disagree with most of the second component of this proposal. I > believe it is > far too disruptive, and I honestly don’t see the argument for it > (except > maybe splitting some astro-specific stuff out into an extension). > But > spinning off stuff like volume rendering and other big components > like that > seems unnecessary. >
No need to correct; the second one is indeed an extension of the first, and a separate proposal.
>> On Jan 18, 2016, at 12:26 PM, Matthew Turk < matthewturk@gmail.com> >> wrote: >> >> Hi Nathan, >> >> Just to quickly address one component before anybody else replies: >> I >> would volunteer to do the majority of this work (which I think from >> a >> technical perspective is not necessarily too bad), and I don't see >> it >> being a 3.X release, but maybe a 4.0 release, and I don't have >> answers >> for when it would happen since I think that is a community decision >> we >> shouldn't rush into. >> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum >> <nathan12343@gmail.com> wrote: >>> Just a few thoughts about this. >>> >>> I think encouraging more external packages to depend on yt will be >>> great, >>> and will keep us more honest about the backward compatibility >>> guarantees >>> we've committed ourselves to. Building social capital by >>> encouraging >>> an >>> "affiliated" package ecosystem like astropy has is also good, both >>> for >>> social capital and also by enabling cool new features for our >>> users. >>> >>> That said, I'm skeptical about some of the technical components of >>> Matt's >>> proposal. In particular, I'm concerned with the amount of work >>> required to >>> make major changes to the core yt codebase, while still >>> maintaining >>> backward >>> compatibility. Who will do this work? When will it happen? Would >>> this >>> constitute a yt 4.0 release? >>> >>> While I do think making the core yt package less astronomy focused >>> is >>> good, >>> I also think it's possible to do so while maintaining strong >>> backward >>> compatibility guarantees, or at least going through a deprecation >>> cycle over >>> the course of several minor versions. I don't think making another >>> cycle of >>> breaking changes so soon after yt-3.0 is a good idea, as that will >>> impose a >>> large amount of work both on the yt developers and on our users. >>> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk >>> <matthewturk@gmail.com> >>> wrote: >>>> >>>> Hi everyone, >>>> >>>> The last couple weeks I’ve been thinking a lot about the future >>>> of >>>> yt. >>>> What I’d like to propose is that we shift investment in yt as a >>>> single, monolithic codebase into yt as a project, or an ecosystem >>>> of >>>> projects. >>>> >>>> This came out of the discussion of the extension/affiliated >>>> packages, >>>> analysis modules, and so on. Britton has for a while been >>>> pitching >>>> the idea [which I will poorly paraphrase here] that yt can be
>>>> framework on top of which killer apps can be built. I think
>>>> is >>>> great. >>>> >>>> What’s holding us back in some ways from this is that yt is >>>> currently >>>> structured as a monolithic code base, with little to no discovery >>>> of >>>> other packages and apps and whatnot. We tried for a while to >>>> change >>>> this with The Barn, but it ended up not quite taking off. I >>>> think >>>> the >>>> time is right to try to change the way we think about yt to be >>>> more >>>> about yt the Project, rather than yt the Codebase; the core >>>> codebase >>>> is an important component of this, but not the whole of it. >>>> >>>> Encouraging an ecosystem of packages can have a few very >>>> important >>>> benefits: >>>> >>>> * External packages will confer greater individual credit to
>>>> folks who develop them. >>>> * External packages can be versioned and developed independently; >>>> the >>>> review process can be different. >>>> * yt’s core can be emphasized as a generic package, on top of >>>> which >>>> astronomy analysis can be built. >>>> * Packages can be maintained wherever, including alternate >>>> locations >>>> such as github. >>>> >>>> On the other hand, having packages inside the main distribution >>>> makes >>>> discoverability much, much easier. It also enables everything to >>>> be >>>> In The Box. And, the continuous integration and testing system >>>> is >>>> already set up for yt. But, these are all possible to overcome >>>> -- we >>>> can devise a strategy for adding packages to the CI system (and >>>> if >>>> they are externally managed, they can also rely on yt as a >>>> dependency >>>> and use whatever CI system they like!) and we can improve >>>> discoverability by refocusing the website to enable this. I've >>>> asked >>>> Kacper about adding new packages, and it's not as easy as it >>>> might >>>> seem, so we may need to be careful about how that process occurs; >>>> one >>>> possibility would be to provide servers and ready-made setups, >>>> but >>>> have individuals do the heavy lifting. We could even have >>>> something >>>> in the codebase that describes some packages that are available. >>>> External packages could have much looser dependency rules, which >>>> means >>>> they can be free to take advantage of things like OpenCL, numba, >>>> etc, >>>> without having to add them to the primary codebase. >>>> >>>> Synchronizing APIs and versions across extension packages may be >>>> difficult in some particular cases, but I suspect in practice >>>> will >>>> not >>>> be an issue, as long as we continue to have a reasonably stable >>>> *public* API, and graduate a few things (such as .blocks) into a >>>> public API from semi-private. >>>> >>>> To this end, of really encouraging an ecosystem of packages, I’d >>>> like >>>> to propose two things, in increasing order of disruptiveness. >>>> >>>> First: Encourage extension packages. This would mean: >>>> * Reorganize website to allow for extension packages to be >>>> displayed >>>> prominently >>>> * Add support for name-space packages in yt >>>> * (possible) split out some packages from analysis_modules, >>>> including >>>> halo finding >>>> * Codify process of extension package creation, including how to >>>> have >>>> CI set up for them and build system. >>>> >>>> The second, more disruptive proposal: >>>> * Split yt into subprojects. This would include spinning out
On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <brittonsmith@gmail.com> wrote: packages. maintain like the this the the
>>>> volume rendering and some or all of the frontends, and probably >>>> the >>>> testing infrastructure as well. >>>> * Split further astro-specific routines into an astro extension, >>>> and >>>> begin the process of doing this with other domains as well. (As >>>> in >>>> the long-simmering domain context YTEP.) >>>> >>>> I’ll invite comments from everyone, but particularly from folks >>>> who >>>> have either not contributed to an analysis module or extension >>>> package >>>> because of concerns that would be addressed by this, as well as >>>> from >>>> core developers this would impact. If the thread gets too >>>> unweildy >>>> we >>>> may also want to table this for the next yt team meeting. >>>> >>>> Thanks, >>>> >>>> Matt >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor
Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 phone: 631-632-8225 e-mail: Michael.Zingale@stonybrook.edu web: http://www.astro.sunysb.edu/mzingale github: http://github.com/zingale
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all, A month and a day later, I've issued A PR that implements some of the discussion from this YTEP: https://bitbucket.org/yt_analysis/yt/pull-requests/2008/adding-ytextensions-... There's an example in there, but the long and the short is that it is possible with this PR to install something separately and have it imported into yt.extensions without any messing around with namespace packages, etc. -Matt On Wed, Jan 27, 2016 at 5:06 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
I have submitted a draft YTEP here:
https://bitbucket.org/yt_analysis/ytep/pull-requests/60/ytep-0029-extension-...
Please feel free to leave short comments there, but if they are larger comments, I think they should perhaps come here. I didn't discuss everything in it, but I think the biggest change is non-technical, and instead that of a mindset. Also, the website.
-Matt
On Tue, Jan 19, 2016 at 9:20 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
Thanks for all your feedback. I think it seems like the proposal #1 is a good way to go. I'm going to try to start writing up a YTEP that formalizes this, and then we can put that up for a vote -- let's not consider anything "settled" yet, but it seems pretty reasonable to move forward with a more detailed proposal. If anyone else has comments, please feel free to leave them here, or on the YTEP.
-Matt
On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <brittonsmith@gmail.com> wrote:
I guess this is clear from the original email, but I very much like the idea of an ecosystem of a core yt package and separate extension packages. Personally, I would stop short of making frontends into external packages. I think one of the big values of yt is the library of frontends we maintain and the ability of people to install the source and immediately be able to load their data without further requirements. I think we run the risk of losing new users who have to go through too many steps just to get their data loaded.
Perhaps the first thing should be to start up the domain context efforts again. By some later 3.x release, we could have an astro domain with domain specific fields and extensions all still within the main yt codebase. Then, moving to 4.0, we start to extract things into separate packages.
On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale <michael.zingale@stonybrook.edu> wrote:
Affiliated packages sound like a good idea to me, but I am a bit hesitant about splitting the volume renderer and frontends out of yt's core.
On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Desika,
Yup, that's what I meant by it. And there are things that would normally qualify as that type of extension, too, that are already in the codebase.
Flask's extensions are another good example of how this type of thing is done:
http://flask.pocoo.org/extensions/ http://flask.pocoo.org/docs/0.10/extensiondev/
-Matt
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote:
Hey All,
Matt - just to maybe clarify by examples...are you thinking things like Trident or powderday are sorts of yt extensions that you mean? (If so, then I do like this idea a lot too). I think one might imagine having a clear set of guidelines for developers of extensions to aspire toward (even if it's for example, having some basic set of or style of documentation) so that from the user point, when looking at extensions on the yt docs, there's some modicum of uniformity. That might make it easier for users to browse and understand the extensions.
-d
On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk@gmail.com> wrote: > > On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone@gmail.com> > wrote: > > Matt can correct me if I am wrong, but I was under the impression > > that > > the two components of the proposal were separate and not > > inevitably > > tied to > > one another. > > > > I am in strong support of the idea of extensions, but I strongly > > disagree with most of the second component of this proposal. I > > believe it is > > far too disruptive, and I honestly don’t see the argument for it > > (except > > maybe splitting some astro-specific stuff out into an extension). > > But > > spinning off stuff like volume rendering and other big components > > like that > > seems unnecessary. > > > > No need to correct; the second one is indeed an extension of the > first, and a separate proposal. > > >> On Jan 18, 2016, at 12:26 PM, Matthew Turk > >> <matthewturk@gmail.com> > >> wrote: > >> > >> Hi Nathan, > >> > >> Just to quickly address one component before anybody else > >> replies: > >> I > >> would volunteer to do the majority of this work (which I think > >> from > >> a > >> technical perspective is not necessarily too bad), and I don't > >> see > >> it > >> being a 3.X release, but maybe a 4.0 release, and I don't have > >> answers > >> for when it would happen since I think that is a community > >> decision > >> we > >> shouldn't rush into. > >> > >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum > >> <nathan12343@gmail.com> wrote: > >>> Just a few thoughts about this. > >>> > >>> I think encouraging more external packages to depend on yt will > >>> be > >>> great, > >>> and will keep us more honest about the backward compatibility > >>> guarantees > >>> we've committed ourselves to. Building social capital by > >>> encouraging > >>> an > >>> "affiliated" package ecosystem like astropy has is also good, > >>> both > >>> for > >>> social capital and also by enabling cool new features for our > >>> users. > >>> > >>> That said, I'm skeptical about some of the technical components > >>> of > >>> Matt's > >>> proposal. In particular, I'm concerned with the amount of work > >>> required to > >>> make major changes to the core yt codebase, while still > >>> maintaining > >>> backward > >>> compatibility. Who will do this work? When will it happen? > >>> Would > >>> this > >>> constitute a yt 4.0 release? > >>> > >>> While I do think making the core yt package less astronomy > >>> focused > >>> is > >>> good, > >>> I also think it's possible to do so while maintaining strong > >>> backward > >>> compatibility guarantees, or at least going through a > >>> deprecation > >>> cycle over > >>> the course of several minor versions. I don't think making > >>> another > >>> cycle of > >>> breaking changes so soon after yt-3.0 is a good idea, as that > >>> will > >>> impose a > >>> large amount of work both on the yt developers and on our > >>> users. > >>> > >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk > >>> <matthewturk@gmail.com> > >>> wrote: > >>>> > >>>> Hi everyone, > >>>> > >>>> The last couple weeks I’ve been thinking a lot about the > >>>> future > >>>> of > >>>> yt. > >>>> What I’d like to propose is that we shift investment in yt as > >>>> a > >>>> single, monolithic codebase into yt as a project, or an > >>>> ecosystem > >>>> of > >>>> projects. > >>>> > >>>> This came out of the discussion of the extension/affiliated > >>>> packages, > >>>> analysis modules, and so on. Britton has for a while been > >>>> pitching > >>>> the idea [which I will poorly paraphrase here] that yt can be > >>>> the > >>>> framework on top of which killer apps can be built. I think > >>>> this > >>>> is > >>>> great. > >>>> > >>>> What’s holding us back in some ways from this is that yt is > >>>> currently > >>>> structured as a monolithic code base, with little to no > >>>> discovery > >>>> of > >>>> other packages and apps and whatnot. We tried for a while to > >>>> change > >>>> this with The Barn, but it ended up not quite taking off. I > >>>> think > >>>> the > >>>> time is right to try to change the way we think about yt to be > >>>> more > >>>> about yt the Project, rather than yt the Codebase; the core > >>>> codebase > >>>> is an important component of this, but not the whole of it. > >>>> > >>>> Encouraging an ecosystem of packages can have a few very > >>>> important > >>>> benefits: > >>>> > >>>> * External packages will confer greater individual credit to > >>>> the > >>>> folks who develop them. > >>>> * External packages can be versioned and developed > >>>> independently; > >>>> the > >>>> review process can be different. > >>>> * yt’s core can be emphasized as a generic package, on top of > >>>> which > >>>> astronomy analysis can be built. > >>>> * Packages can be maintained wherever, including alternate > >>>> locations > >>>> such as github. > >>>> > >>>> On the other hand, having packages inside the main > >>>> distribution > >>>> makes > >>>> discoverability much, much easier. It also enables everything > >>>> to > >>>> be > >>>> In The Box. And, the continuous integration and testing > >>>> system > >>>> is > >>>> already set up for yt. But, these are all possible to > >>>> overcome > >>>> -- we > >>>> can devise a strategy for adding packages to the CI system > >>>> (and > >>>> if > >>>> they are externally managed, they can also rely on yt as a > >>>> dependency > >>>> and use whatever CI system they like!) and we can improve > >>>> discoverability by refocusing the website to enable this. > >>>> I've > >>>> asked > >>>> Kacper about adding new packages, and it's not as easy as it > >>>> might > >>>> seem, so we may need to be careful about how that process > >>>> occurs; > >>>> one > >>>> possibility would be to provide servers and ready-made setups, > >>>> but > >>>> have individuals do the heavy lifting. We could even have > >>>> something > >>>> in the codebase that describes some packages that are > >>>> available. > >>>> External packages could have much looser dependency rules, > >>>> which > >>>> means > >>>> they can be free to take advantage of things like OpenCL, > >>>> numba, > >>>> etc, > >>>> without having to add them to the primary codebase. > >>>> > >>>> Synchronizing APIs and versions across extension packages may > >>>> be > >>>> difficult in some particular cases, but I suspect in practice > >>>> will > >>>> not > >>>> be an issue, as long as we continue to have a reasonably > >>>> stable > >>>> *public* API, and graduate a few things (such as .blocks) into > >>>> a > >>>> public API from semi-private. > >>>> > >>>> To this end, of really encouraging an ecosystem of packages, > >>>> I’d > >>>> like > >>>> to propose two things, in increasing order of disruptiveness. > >>>> > >>>> First: Encourage extension packages. This would mean: > >>>> * Reorganize website to allow for extension packages to be > >>>> displayed > >>>> prominently > >>>> * Add support for name-space packages in yt > >>>> * (possible) split out some packages from analysis_modules, > >>>> including > >>>> halo finding > >>>> * Codify process of extension package creation, including how > >>>> to > >>>> have > >>>> CI set up for them and build system. > >>>> > >>>> The second, more disruptive proposal: > >>>> * Split yt into subprojects. This would include spinning out > >>>> the > >>>> volume rendering and some or all of the frontends, and > >>>> probably > >>>> the > >>>> testing infrastructure as well. > >>>> * Split further astro-specific routines into an astro > >>>> extension, > >>>> and > >>>> begin the process of doing this with other domains as well. > >>>> (As > >>>> in > >>>> the long-simmering domain context YTEP.) > >>>> > >>>> I’ll invite comments from everyone, but particularly from > >>>> folks > >>>> who > >>>> have either not contributed to an analysis module or extension > >>>> package > >>>> because of concerns that would be addressed by this, as well > >>>> as > >>>> from > >>>> core developers this would impact. If the thread gets too > >>>> unweildy > >>>> we > >>>> may also want to table this for the next yt team meeting. > >>>> > >>>> Thanks, > >>>> > >>>> Matt > >>>> _______________________________________________ > >>>> yt-dev mailing list > >>>> yt-dev@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>> > >>> > >>> > >>> _______________________________________________ > >>> yt-dev mailing list > >>> yt-dev@lists.spacepope.org > >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>> > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Michael Zingale Associate Professor
Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 phone: 631-632-8225 e-mail: Michael.Zingale@stonybrook.edu web: http://www.astro.sunysb.edu/mzingale github: http://github.com/zingale
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (6)
-
Britton Smith -
Desika Narayanan -
John ZuHone -
Matthew Turk -
Michael Zingale -
Nathan Goldbaum