Hey J, Sadly there are no primers on this particular issue because it is not a solved problem! Of course, you can pick up the cmake / pip / autoconf / brew / ports manuals and read through those. This will provide a basis for understanding how people have solves this problem when they have complete control over the environment. What it doesn't provide is the context for users running into issues trying to install shard object libraries on platforms that don't natively support a compiler ;). For that, you need experience and to join mailing lists where this comes up, such as yt and (shameless plug) NumFOCUS. Be Well Anthony On Thu, Nov 14, 2013 at 12:12 PM, j s oishi wrote:

I am very, very interested in this discussion, but I am an absolute neophyte when it comes to build systems, installing, and so on. Is there a good primer on these issues that I could look at?

j

On Thu, Nov 14, 2013 at 11:59 AM, Anthony Scopatz wrote:

...

Thanks for fwding this along Nathan. On Nov 14, 2013 3:25 AM, "Nathan Goldbaum" wrote:

...

There is a very interesting conversation going on over on the numfocus list about a distributed build service for the numfocus ecosystem: https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg

I’m forwarding this message from Anthony since he mentions yt. I’m hopeful that in the end this discussion leads to something concrete and useful for our users.

On November 14, 2013 at 12:16:44 AM, Anthony Scopatz (scopatz@gmail.com) wrote:

Hello All,

I'd like to volunteer to be the person to spearhead this effort, or at least be the manager/organizer of this effort. Normally I wouldn't do this (I have enough on my plate) but the build and distribution problems have become *the* central issue for PyNE http://pynesim.org/ in the past 10 days [1]. On Monday Nov. 4th, Katy hosted a PyNE workshop at UC Berkeley. Out of the ~15 attendees, 2 could successfully install PyNE...even using anaconda and/or canopy and ignoring our optional dependencies. *We absolutely have to solve this problem for the project to survive.* The PyNE stack fairly standard for this community. The fact that we have had such terrible install problems is really a serious issue. We are going to dedicate the next release to fixing this problem *rather than* doing any nuclear engineering.

Furthermore, I don't think that this is a wholly unique situation as I know other codes, such as yt, have experienced similar difficulties in this space.

I believe that to fix this problem sufficiently we must have empathy for our users, rather than for ourselves as developers. We need to be open to installing packages in the way the user is comfortable with even though it may make more work for us.

Many of the problems I feel stem from wanting to support a public API for many languages simultaneously (C/C++/Python/Fortran). PyNE is not unique in this way. NumPy also does this. However, the dependency pool for PyNE is much broader and deeper than for NumPy. Additionally as Aaron brings up, we aren't really just talking about supporting Python & Cython code. By virtue of our dependencies, we also have to support packages in these compiled libraries.

I have many ideas about how to go about this process, though it will require help from many people. I believe all of the pieces are there to do this right, but what we need is a coordinated strategy, someone to carve up tasks appropriately, and someone to make sure that this process doesn't take too long. Again, I am happy to be the cat-wrangler, if for no other reason than that PyNE will be going through this shortly.

Travis-ci, vms, etc, have come up quite a bit so far on this thread. As others have noted, this is the right idea but insufficiently executed. Therefore, I'd like to direct folks to the Build & Test Lab (BaTLab)https://www.batlab.org/. This is an NSF funded organization whose purpose it is to provide an integrated suite of platforms https://www.batlab.org/?q=platforms for building and testing scientific software. Even though it is at UW-Madison, it is open to everyone and free to use. The advantage here is that BaTLab jobs can take as long as they need to and it already supports the platforms we need in many flavors: Windows, Mac, Linux. For another project (cyclus), I am working on a little web apphttps://github.com/cyclus/polyphemuswhich exposes Travis-CI-like functionality but is backended by BaTLab. Hopefully this will be in a working state by this weekend. *Basically, there is no need for us to build the infrastructure!* Or at lease no need to do so in the short- to medium-term.

I think if every project were able to donate a person or two to work on this for the next month we really could get something working that truly addressed or routed around some of the fundamental flaws with the current system.

In summary, we (PyNE +/- yt) can run off and solve this for ourselves -OR- we (NumFOCUS) can help pool our resources and try to create a quality solution that applies to everyone. Sorry if this is a bit ranty, but it has become difficult for me to sell Python & PyNE because of building and distribution. This is long overdue and I am willing to put my money where my mouth is. If someone has better ideas, I'd love to hear them!

Be Well Anthony

1. https://groups.google.com/d/topic/pyne-dev/RxNCHvZAAEQ/discussion

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org