Hi all,
We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.)
As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types.
Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable.
That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple.
There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages.
So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front.
* Conda / Anaconda: This package manager is gaining traction, and I think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers. * Canopy: This is the Enthought package manager, which Sam has had some good experience with it. I do not have a feeling for how it works on supers. * Source-only: This is the way some packages are managed, but it is essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable. * Keep trying to plug holes as they come up in the install script.
What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody?
Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one.
-Matt
PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them.
Hi Matt,
I maintain yt 2.5 and yt 3.0 alpha builds on our cluster here at UChicago, and building from source is not difficult, once the short number of dependencies are satisfied. What is more difficult, however, is to keep up with the versions of those dependencies. Our builds currently use Matplotlib 1.2, and only recently upgraded to Cython 0.19. There is some risk to producing a broken build if the build system does not test for required (vs recommended) versions.
A package system would be nice, provided it can be entirely self contained (so as to not conflict with other packages or package versions installed on the system).
Douglas Rudd Scientific Computing Consultant Research Computing Center drudd@uchicago.edu
On Aug 29, 2013, at 2:26 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.)
As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types.
Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable.
That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple.
There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages.
So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front.
- Conda / Anaconda: This package manager is gaining traction, and I
think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers.
- Canopy: This is the Enthought package manager, which Sam has had
some good experience with it. I do not have a feeling for how it works on supers.
- Source-only: This is the way some packages are managed, but it is
essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable.
- Keep trying to plug holes as they come up in the install script.
What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody?
Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one.
-Matt
PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I think both canopy and anaconda are good solutions that will create a useful python environment out of the box with a minimum of fuss on a wide variety of platforms.
I have more experience with anaconda so my preference for our main installation avenue is there. A concern with canopy is the somewhat arbitrary choice of packages in the free (as in beer) and non-free distributions.
Anaconda has a very liberal redistribution license ( http://docs.continuum.io/anaconda/eula.html). We could conceivably distribute it ourselves on yt-project.org. I know Matt is also looking into a miniconda (http://repo.continuum.io/miniconda/) based deployment, which should work well. I think we stand a good chance of being included with the standard anaconda distribution or via conda.
Continuum has also set up binstar to allow projects to upload packaged versions of software which is then available via conda. One issue with binstar is that since we have C dependencies, we'll need buildbots to create packages for all supported OS/python combos (i.e. linux/2.7, OSX/2.7, linux/3.3 OSX/3.3).
Making the install script more lightweight is a good idea, although we'll still be susceptible to external changes in our hard dependencies. Numpy and matplotlib have historically been tough to keep up with.
On Thu, Aug 29, 2013 at 12:26 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.)
As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types.
Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable.
That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple.
There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages.
So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front.
- Conda / Anaconda: This package manager is gaining traction, and I
think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers.
- Canopy: This is the Enthought package manager, which Sam has had
some good experience with it. I do not have a feeling for how it works on supers.
- Source-only: This is the way some packages are managed, but it is
essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable.
- Keep trying to plug holes as they come up in the install script.
What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody?
Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one.
-Matt
PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I think investigating these options is a great idea, especially if you think about it as allowing the install script to not have to be updated every single time numpy/matplotlib updates their version. We can let the install script stay more stable on the several month timescale and rely on other packaging to work out if people want up to the minute installs of some package.
As for conda vs canopy, I think we should experiment with one first, and polish it off rather than attempting both at the same time. Since Nathan has more experience with conda than I do with canopy, I'd be +1 on going down the conda route first.
Sam
On Thu, Aug 29, 2013 at 12:50 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
I think both canopy and anaconda are good solutions that will create a useful python environment out of the box with a minimum of fuss on a wide variety of platforms.
I have more experience with anaconda so my preference for our main installation avenue is there. A concern with canopy is the somewhat arbitrary choice of packages in the free (as in beer) and non-free distributions.
Anaconda has a very liberal redistribution license ( http://docs.continuum.io/anaconda/eula.html). We could conceivably distribute it ourselves on yt-project.org. I know Matt is also looking into a miniconda (http://repo.continuum.io/miniconda/) based deployment, which should work well. I think we stand a good chance of being included with the standard anaconda distribution or via conda.
Continuum has also set up binstar to allow projects to upload packaged versions of software which is then available via conda. One issue with binstar is that since we have C dependencies, we'll need buildbots to create packages for all supported OS/python combos (i.e. linux/2.7, OSX/2.7, linux/3.3 OSX/3.3).
Making the install script more lightweight is a good idea, although we'll still be susceptible to external changes in our hard dependencies. Numpy and matplotlib have historically been tough to keep up with.
On Thu, Aug 29, 2013 at 12:26 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.)
As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types.
Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable.
That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple.
There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages.
So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front.
- Conda / Anaconda: This package manager is gaining traction, and I
think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers.
- Canopy: This is the Enthought package manager, which Sam has had
some good experience with it. I do not have a feeling for how it works on supers.
- Source-only: This is the way some packages are managed, but it is
essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable.
- Keep trying to plug holes as they come up in the install script.
What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody?
Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one.
-Matt
PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I like this plan too. Let's test out conda stuff and see how it all goes. I'm happy to assist in any way that I can.
I'm glad that we're addressing this. It's pretty amazing how well the bash script has worked in the past, and I think it is a testament to how hard everyone has worked to keep it updated and patched (and Matt writing it in the first place!). So good work everyone, but I think moving to a more generic means like conda will pay off in the long run now.
Cameron
On Thu, Aug 29, 2013 at 1:22 PM, Sam Skillman samskillman@gmail.com wrote:
I think investigating these options is a great idea, especially if you think about it as allowing the install script to not have to be updated every single time numpy/matplotlib updates their version. We can let the install script stay more stable on the several month timescale and rely on other packaging to work out if people want up to the minute installs of some package.
As for conda vs canopy, I think we should experiment with one first, and polish it off rather than attempting both at the same time. Since Nathan has more experience with conda than I do with canopy, I'd be +1 on going down the conda route first.
Sam
On Thu, Aug 29, 2013 at 12:50 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
I think both canopy and anaconda are good solutions that will create a useful python environment out of the box with a minimum of fuss on a wide variety of platforms.
I have more experience with anaconda so my preference for our main installation avenue is there. A concern with canopy is the somewhat arbitrary choice of packages in the free (as in beer) and non-free distributions.
Anaconda has a very liberal redistribution license ( http://docs.continuum.io/anaconda/eula.html). We could conceivably distribute it ourselves on yt-project.org. I know Matt is also looking into a miniconda (http://repo.continuum.io/miniconda/) based deployment, which should work well. I think we stand a good chance of being included with the standard anaconda distribution or via conda.
Continuum has also set up binstar to allow projects to upload packaged versions of software which is then available via conda. One issue with binstar is that since we have C dependencies, we'll need buildbots to create packages for all supported OS/python combos (i.e. linux/2.7, OSX/2.7, linux/3.3 OSX/3.3).
Making the install script more lightweight is a good idea, although we'll still be susceptible to external changes in our hard dependencies. Numpy and matplotlib have historically been tough to keep up with.
On Thu, Aug 29, 2013 at 12:26 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
We need to figure out yt packaging. This is becoming increasingly hard, particularly as the number of dependencies grows. (The upgrade to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which spurred this discussion.)
As it stands, we mainly provide yt through the install script. Every time a new version comes out, we check compatibility, we update the install script, and we deploy that. Unfortunately, as packages evolve externally to yt, this results in occasional breakages, new (implicit) dependencies, and complexity that goes super-exponentially. I like the install script, and it is what I use, but I think we need to re-strategize. It was built many years ago when packaging was a different landscape, and when we needed a way to get a relatively small number of dependencies onto a relatively small set of system types.
Every day, it seems, brings another problem with the install script. Not all of these are our fault. But more importantly, I don't think we should be spending our time on them, when we can only bandaid something for so long before it's not workable.
That being said, installation is the single biggest impediment to people using yt, so we need to ensure it is still easy and simple.
There are a few options for other installation procedures. I would like to retain a stripped down version of the install script for ease and simplicity, but removing many of the optional installs and focusing instead on the core packages.
So here are the options. I'd prefer we choose *one* as the primary method, and then we (potentially) demonstrate how to use the others. As a note, part of this process will also be the relicensing as BSD and shoring up our source-based installations, ensuring that they are correctly packaged, following best-practices guidelines for Python source. I believe I may have dropped the ball somewhat on that front.
- Conda / Anaconda: This package manager is gaining traction, and I
think that once relicensing is done we stand a good chance of being included in the base install. This would mean that someone could download Conda and just use it. Even without that inclusion, however, I've heard good things. Conda is based on binary distributions, but we could also manage our own packaging (potentially in an automated way) and update with some frequency. Conda is also somewhat tied to the Wakari platform, and being part of Conda would mean being available on the IPython-in-the-cloud that is Wakari. I believe this works well on supers.
- Canopy: This is the Enthought package manager, which Sam has had
some good experience with it. I do not have a feeling for how it works on supers.
- Source-only: This is the way some packages are managed, but it is
essentially giving up, and while I think it is a good way to go forward, I'm not sure we'll ever be trivially pip-installable.
- Keep trying to plug holes as they come up in the install script.
What I think would be very productive is to hear people's experiences with these package managers. Sam, Nathan, anybody?
Focusing on a platform-specific manager (brew, macports, apt, rpm) is a non-starter; they are good options, and we should develop a protocol for supporting platform-specific packaging systems, but they bottleneck quite seriously on person-time and we should think carefully before we tie ourselves to one.
-Matt
PS The period in the subject line was editorial. I'd very much like to settle on a path for all of this stuff; packaging remains one of the hardest issues in scientific python, as Software Carpentry has noted time and again. We're now pushing the install script, which is great for clusters, but it's a remnant of a time before packaging in Python was as mature as it is now, and before we had as many corner cases as we do now -- not because they didn't exist, but because we didn't have enough users to see them. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
I think to get everything working in a sustainable fashion, we would need buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3) tuple. At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X (not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything from source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
* Get miniconda: http://repo.continuum.io/miniconda/index.html * Run the installer for miniconda * Enter the conda environment and then install yt by doing "conda install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I think to get everything working in a sustainable fashion, we would need buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3) tuple. At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X (not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything from source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum <nathan12343@gmail.comjavascript:;> wrote:
I think to get everything working in a sustainable fashion, we would need buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3)
tuple.
At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X (not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything from source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <s@skory.usjavascript:;>
wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us javascript:; http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org javascript:; http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org javascript:; http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org javascript:; http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I think to get everything working in a sustainable fashion, we would
need
buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3)
tuple.
At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X (not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything from source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday ( https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.comwrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I think to get everything working in a sustainable fashion, we would
need
buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3)
tuple.
At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS
X
(not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything
from
source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Yup, my fault. I've stuck the recipes here:
https://bitbucket.org/yt_analysis/yt_conda
until they all get sucked into the main conda-recipes repo, which is where I'd like to see them stay. One nice thing is that -- I think -- the "-c" option can be supplied multiple times, so in principle if Nathan were to upload, you could supply his as well.
Once this gets going a bit more we can definitely work on this to make it more flexible and accessible. I am kind of liking conda in general though, and I particularly like that we could suggest miniconda and then have individuals interested in more packages do "conda install anaconda" to get basically everything else. We can also preserve the "source install" of yt, as well as the "yt update" functionality, if we provide both binary and source install methods, and the difference would be so small that they could be in the same script. One would do "conda install yt" and the other would clone the repo and do "setup.py develop" like we do now.
On Fri, Aug 30, 2013 at 1:42 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.com wrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I think to get everything working in a sustainable fashion, we would need buildbots for all platform combinations that we want to support, so all permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3) tuple. At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X (not totally clear if OS X version matters), and 32 and 64 bit windows.
Another option is to rely on conda build, which compiles everything from source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.comwrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday ( https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.comwrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
I think to get everything working in a sustainable fashion, we would
need
buildbots for all platform combinations that we want to support, so
all
permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3)
tuple.
At the moment anaconda seems to support 32 and 64 bit linux, 64 bit
OS X
(not totally clear if OS X version matters), and 32 and 64 bit
windows.
Another option is to rely on conda build, which compiles everything
from
source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have less of a skin in this than I used to, but I'd like to raise the issue of Windows & package managers. For example, Anaconda is available for Windows - would that mean that yt might "just work" on Windows? Or the opposite, and it would require a great deal of effort to get all the various things we expect to be .so's to work as .dll's (such as the Cython helpers or halo-finding stuff)?
I don't know the answers to these questions, but I think it's worth thinking about.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.comwrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.comwrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday ( https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.comwrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
I think to get everything working in a sustainable fashion, we would
need
buildbots for all platform combinations that we want to support, so
all
permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3)
tuple.
At the moment anaconda seems to support 32 and 64 bit linux, 64 bit
OS X
(not totally clear if OS X version matters), and 32 and 64 bit
windows.
Another option is to rely on conda build, which compiles everything
from
source.
On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: > > Hi all, > > I have less of a skin in this than I used to, but I'd like to raise > the issue of Windows & package managers. For example, Anaconda is > available for Windows - would that mean that yt might "just work" on > Windows? Or the opposite, and it would require a great deal of
effort
> to get all the various things we expect to be .so's to work as
.dll's
> (such as the Cython helpers or halo-finding stuff)? > > I don't know the answers to these questions, but I think it's worth > thinking about. > > -- > Stephen Skory > s@skory.us > http://stephenskory.com/ > 510.621.3687 (google voice) > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.comwrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.comwrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday ( https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone <chris.m.malone@gmail.com
wrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote:
Hi all,
Thank you for the feedback -- I am glad there is some agreement about possible ways forward, and so I'm happy to try to use this as an opportunity to explore simpler, more reliable methods than the install script.
This afternoon, I spent a bit of time with Conda, and I think it's quite nice. There are a few rough corners, particularly related to the binstar service, but it's so far pretty great. With Nathan's help, I was able to upload a yt-2.5.5 package for linux x86_64 and then install it.
The workflow that seems to work:
- Get miniconda: http://repo.continuum.io/miniconda/index.html
- Run the installer for miniconda
- Enter the conda environment and then install yt by doing "conda
install yt -c http://conda.binstar.org/yt_project/ ".
I think that this can likely all be stuck into a bash script. A simple, first pass at this is here:
http://paste.yt-project.org/show/3833/
This right now only works on Linux x86_64, but getting it to work for other machines won't be too hard. I suspect we will be able to do nightlies very easily as well. If anyone out there has an x86_64 machine they wouldn't mind trying it on, that would be very helpful! I did find that once I ran this script, I had to actually prepend the PATH afterwards as well. This means doing:
export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH export PATH=$HOME/yt-conda:$PATH source activate ytenv
At that point, everything was set up and working for me. The miniconda install offers to add paths to .bashrc, but I don't think we should go down that route. That being said, this is also a possible point of friction.
One nice thing is that this also completely works with the full anaconda; if someone wants everything that is in the anaconda install, they can even simply do "conda install anaconda" from the command line to get it. But the stripped down subset is the default.
If anyone has a chance to try this out and has feedback, I'd greatly appreciate it! I think Nathan has done something very similar for OSX. I've also put a couple simple conda recipes here: https://bitbucket.org/yt_analysis/yt_conda which we can use as a basis for distributing builds and setting them up on buildbots and the like. I'm pretty optimistic about this.
-Matt
On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: > I think to get everything working in a sustainable fashion, we would need > buildbots for all platform combinations that we want to support, so all > permutations of the (32/64 bit, linux / OS X / windows, py27/py3.3) tuple. > At the moment anaconda seems to support 32 and 64 bit linux, 64 bit OS X > (not totally clear if OS X version matters), and 32 and 64 bit windows. > > Another option is to rely on conda build, which compiles everything from > source. > > > On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: >> >> Hi all, >> >> I have less of a skin in this than I used to, but I'd like to raise >> the issue of Windows & package managers. For example, Anaconda is >> available for Windows - would that mean that yt might "just work" on >> Windows? Or the opposite, and it would require a great deal of effort >> to get all the various things we expect to be .so's to work as .dll's >> (such as the Cython helpers or halo-finding stuff)? >> >> I don't know the answers to these questions, but I think it's worth >> thinking about. >> >> -- >> Stephen Skory >> s@skory.us >> http://stephenskory.com/ >> 510.621.3687 (google voice) >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
* Make a single script that grabs the appropriate distribution of miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture. * Create a mechanism for installing all the packages we need. Nearly all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do. * Test this out lots of places * Clean up the edges in the (new) install script * Move the old install script to maintenance mode * Update all documentation to describe this and mothball other methods of installation
What would be nice:
* Make available nightly builds of yt on several architectures using binstar * Utilize more of the packages included in conda elsewhere in yt, now that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.com wrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Yup, please try on OSX as well. If you make sure Matt's binstar is in your .condarc, you should be able to get yt by doing 'conda install yt'.
I built the OSX binary on my laptop so I'd appreciate hearing about issues, particularly if there are issues on older OS X releases.
On Thursday, August 29, 2013, Matthew Turk wrote: > > Hi all, > > Thank you for the feedback -- I am glad there is some agreement about > possible ways forward, and so I'm happy to try to use this as an > opportunity to explore simpler, more reliable methods than the > install > script. > > This afternoon, I spent a bit of time with Conda, and I think it's > quite nice. There are a few rough corners, particularly related to > the binstar service, but it's so far pretty great. With Nathan's > help, I was able to upload a yt-2.5.5 package for linux x86_64 and > then install it. > > The workflow that seems to work: > > * Get miniconda: http://repo.continuum.io/miniconda/index.html > * Run the installer for miniconda > * Enter the conda environment and then install yt by doing "conda > install yt -c http://conda.binstar.org/yt_project/ ". > > I think that this can likely all be stuck into a bash script. A > simple, first pass at this is here: > > http://paste.yt-project.org/show/3833/ > > This right now only works on Linux x86_64, but getting it to work for > other machines won't be too hard. I suspect we will be able to do > nightlies very easily as well. If anyone out there has an x86_64 > machine they wouldn't mind trying it on, that would be very helpful! > I did find that once I ran this script, I had to actually prepend the > PATH afterwards as well. This means doing: > > export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH > export PATH=$HOME/yt-conda:$PATH > source activate ytenv > > At that point, everything was set up and working for me. The > miniconda install offers to add paths to .bashrc, but I don't think > we > should go down that route. That being said, this is also a possible > point of friction. > > One nice thing is that this also completely works with the full > anaconda; if someone wants everything that is in the anaconda > install, > they can even simply do "conda install anaconda" from the command > line > to get it. But the stripped down subset is the default. > > If anyone has a chance to try this out and has feedback, I'd greatly > appreciate it! I think Nathan has done something very similar for > OSX. I've also put a couple simple conda recipes here: > https://bitbucket.org/yt_analysis/yt_conda which we can use as a > basis > for distributing builds and setting them up on buildbots and the > like. > I'm pretty optimistic about this. > > -Matt > > On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum > nathan12343@gmail.com wrote: > > I think to get everything working in a sustainable fashion, we > > would need > > buildbots for all platform combinations that we want to support, so > > all > > permutations of the (32/64 bit, linux / OS X / windows, > > py27/py3.3) tuple. > > At the moment anaconda seems to support 32 and 64 bit linux, 64 bit > > OS X > > (not totally clear if OS X version matters), and 32 and 64 bit > > windows. > > > > Another option is to rely on conda build, which compiles everything > > from > > source. > > > > > > On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: > >> > >> Hi all, > >> > >> I have less of a skin in this than I used to, but I'd like to > >> raise > >> the issue of Windows & package managers. For example, Anaconda is > >> available for Windows - would that mean that yt might "just work" > >> on > >> Windows? Or the opposite, and it would require a great deal of > >> effort > >> to get all the various things we expect to be .so's to work as > >> .dll's > >> (such as the Cython helpers or halo-finding stuff)? > >> > >> I don't know the answers to these questions, but I think it's > >> worth > >> thinking about. > >> > >> -- > >> Stephen Skory > >> s@skory.us > >> http://stephenskory.com/ > >> 510.621.3687 (google voice) > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > > > > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.com wrote:
I just tried setting this up on OS X 10.7.5 and failed when attempting to create the conda environment due to a missing mercurial package:
$ conda create -n ytenv -c http://conda.binstar.org/yt_project yt mercurial ipython tornado pyzmq pygments jinja2 sphinx Error: No packages found matching: mercurial
On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum nathan12343@gmail.com wrote: > > Yup, please try on OSX as well. If you make sure Matt's binstar is in > your .condarc, you should be able to get yt by doing 'conda install yt'. > > I built the OSX binary on my laptop so I'd appreciate hearing about > issues, particularly if there are issues on older OS X releases. > > On Thursday, August 29, 2013, Matthew Turk wrote: >> >> Hi all, >> >> Thank you for the feedback -- I am glad there is some agreement about >> possible ways forward, and so I'm happy to try to use this as an >> opportunity to explore simpler, more reliable methods than the >> install >> script. >> >> This afternoon, I spent a bit of time with Conda, and I think it's >> quite nice. There are a few rough corners, particularly related to >> the binstar service, but it's so far pretty great. With Nathan's >> help, I was able to upload a yt-2.5.5 package for linux x86_64 and >> then install it. >> >> The workflow that seems to work: >> >> * Get miniconda: http://repo.continuum.io/miniconda/index.html >> * Run the installer for miniconda >> * Enter the conda environment and then install yt by doing "conda >> install yt -c http://conda.binstar.org/yt_project/ ". >> >> I think that this can likely all be stuck into a bash script. A >> simple, first pass at this is here: >> >> http://paste.yt-project.org/show/3833/ >> >> This right now only works on Linux x86_64, but getting it to work for >> other machines won't be too hard. I suspect we will be able to do >> nightlies very easily as well. If anyone out there has an x86_64 >> machine they wouldn't mind trying it on, that would be very helpful! >> I did find that once I ran this script, I had to actually prepend the >> PATH afterwards as well. This means doing: >> >> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >> export PATH=$HOME/yt-conda:$PATH >> source activate ytenv >> >> At that point, everything was set up and working for me. The >> miniconda install offers to add paths to .bashrc, but I don't think >> we >> should go down that route. That being said, this is also a possible >> point of friction. >> >> One nice thing is that this also completely works with the full >> anaconda; if someone wants everything that is in the anaconda >> install, >> they can even simply do "conda install anaconda" from the command >> line >> to get it. But the stripped down subset is the default. >> >> If anyone has a chance to try this out and has feedback, I'd greatly >> appreciate it! I think Nathan has done something very similar for >> OSX. I've also put a couple simple conda recipes here: >> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >> basis >> for distributing builds and setting them up on buildbots and the >> like. >> I'm pretty optimistic about this. >> >> -Matt >> >> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >> nathan12343@gmail.com wrote: >>> I think to get everything working in a sustainable fashion, we >>> would need >>> buildbots for all platform combinations that we want to support, so >>> all >>> permutations of the (32/64 bit, linux / OS X / windows, >>> py27/py3.3) tuple. >>> At the moment anaconda seems to support 32 and 64 bit linux, 64 bit >>> OS X >>> (not totally clear if OS X version matters), and 32 and 64 bit >>> windows. >>> >>> Another option is to rely on conda build, which compiles everything >>> from >>> source. >>> >>> >>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: >>>> >>>> Hi all, >>>> >>>> I have less of a skin in this than I used to, but I'd like to >>>> raise >>>> the issue of Windows & package managers. For example, Anaconda is >>>> available for Windows - would that mean that yt might "just work" >>>> on >>>> Windows? Or the opposite, and it would require a great deal of >>>> effort >>>> to get all the various things we expect to be .so's to work as >>>> .dll's >>>> (such as the Cython helpers or halo-finding stuff)? >>>> >>>> I don't know the answers to these questions, but I think it's >>>> worth >>>> thinking about. >>>> >>>> -- >>>> Stephen Skory >>>> s@skory.us >>>> http://stephenskory.com/ >>>> 510.621.3687 (google voice) >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com
wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do
not
have regular internet access.
I really like this idea of conda, especially as a package manager that
only
optionally makes its own edits to your .bashrc. I have always really
liked
that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to
people
just looking to use numpy and matploblib. It looks like conda will
continue
to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week
or
so when I get regular internet access, but I would be glad to do so
then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone <
chris.m.malone@gmail.com>
wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install
...`
added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is
python 3
based. Trying to build the environment with that yields an error
regarding
incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again
without
it? I think if Matt uploads a mercurial package for OS X this won't
be an
issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully
a
mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.com wrote: > > I just tried setting this up on OS X 10.7.5 and failed when
attempting
> to create the conda environment due to a missing mercurial package: > > $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt > mercurial ipython tornado pyzmq pygments jinja2 sphinx > Error: No packages found matching: mercurial > > > On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum > nathan12343@gmail.com wrote: >> >> Yup, please try on OSX as well. If you make sure Matt's binstar
is in
>> your .condarc, you should be able to get yt by doing 'conda
install yt'.
>> >> I built the OSX binary on my laptop so I'd appreciate hearing about >> issues, particularly if there are issues on older OS X releases. >> >> On Thursday, August 29, 2013, Matthew Turk wrote: >>> >>> Hi all, >>> >>> Thank you for the feedback -- I am glad there is some agreement
about
>>> possible ways forward, and so I'm happy to try to use this as an >>> opportunity to explore simpler, more reliable methods than the >>> install >>> script. >>> >>> This afternoon, I spent a bit of time with Conda, and I think it's >>> quite nice. There are a few rough corners, particularly related
to
>>> the binstar service, but it's so far pretty great. With Nathan's >>> help, I was able to upload a yt-2.5.5 package for linux x86_64 and >>> then install it. >>> >>> The workflow that seems to work: >>> >>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>> * Run the installer for miniconda >>> * Enter the conda environment and then install yt by doing "conda >>> install yt -c http://conda.binstar.org/yt_project/ ". >>> >>> I think that this can likely all be stuck into a bash script. A >>> simple, first pass at this is here: >>> >>> http://paste.yt-project.org/show/3833/ >>> >>> This right now only works on Linux x86_64, but getting it to work
for
>>> other machines won't be too hard. I suspect we will be able to do >>> nightlies very easily as well. If anyone out there has an x86_64 >>> machine they wouldn't mind trying it on, that would be very
helpful!
>>> I did find that once I ran this script, I had to actually prepend
the
>>> PATH afterwards as well. This means doing: >>> >>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>> export PATH=$HOME/yt-conda:$PATH >>> source activate ytenv >>> >>> At that point, everything was set up and working for me. The >>> miniconda install offers to add paths to .bashrc, but I don't
think
>>> we >>> should go down that route. That being said, this is also a
possible
>>> point of friction. >>> >>> One nice thing is that this also completely works with the full >>> anaconda; if someone wants everything that is in the anaconda >>> install, >>> they can even simply do "conda install anaconda" from the command >>> line >>> to get it. But the stripped down subset is the default. >>> >>> If anyone has a chance to try this out and has feedback, I'd
greatly
>>> appreciate it! I think Nathan has done something very similar for >>> OSX. I've also put a couple simple conda recipes here: >>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>> basis >>> for distributing builds and setting them up on buildbots and the >>> like. >>> I'm pretty optimistic about this. >>> >>> -Matt >>> >>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>> nathan12343@gmail.com wrote: >>>> I think to get everything working in a sustainable fashion, we >>>> would need >>>> buildbots for all platform combinations that we want to support,
so
>>>> all >>>> permutations of the (32/64 bit, linux / OS X / windows, >>>> py27/py3.3) tuple. >>>> At the moment anaconda seems to support 32 and 64 bit linux, 64
bit
>>>> OS X >>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>> windows. >>>> >>>> Another option is to rely on conda build, which compiles
everything
>>>> from >>>> source. >>>> >>>> >>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us
wrote:
>>>>> >>>>> Hi all, >>>>> >>>>> I have less of a skin in this than I used to, but I'd like to >>>>> raise >>>>> the issue of Windows & package managers. For example, Anaconda
is
>>>>> available for Windows - would that mean that yt might "just
work"
>>>>> on >>>>> Windows? Or the opposite, and it would require a great deal of >>>>> effort >>>>> to get all the various things we expect to be .so's to work as >>>>> .dll's >>>>> (such as the Cython helpers or halo-finding stuff)? >>>>> >>>>> I don't know the answers to these questions, but I think it's >>>>> worth >>>>> thinking about. >>>>> >>>>> -- >>>>> Stephen Skory >>>>> s@skory.us >>>>> http://stephenskory.com/ >>>>> 510.621.3687 (google voice) >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
Yup -- you can do it like this:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.com wrote: > > Hey Chris, > > I don't think mercurial is strictly necessary, can you try again > without > it? I think if Matt uploads a mercurial package for OS X this won't > be an > issue. I'll send him an updated tarball. > > I submitted a mercurial recipe to conda-recipes yesterday > (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully > a > mercurial build will be included in future anaconda releases. > > > On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > chris.m.malone@gmail.com wrote: >> >> I just tried setting this up on OS X 10.7.5 and failed when >> attempting >> to create the conda environment due to a missing mercurial package: >> >> $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt >> mercurial ipython tornado pyzmq pygments jinja2 sphinx >> Error: No packages found matching: mercurial >> >> >> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >> nathan12343@gmail.com wrote: >>> >>> Yup, please try on OSX as well. If you make sure Matt's binstar >>> is in >>> your .condarc, you should be able to get yt by doing 'conda >>> install yt'. >>> >>> I built the OSX binary on my laptop so I'd appreciate hearing >>> about >>> issues, particularly if there are issues on older OS X releases. >>> >>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>> >>>> Hi all, >>>> >>>> Thank you for the feedback -- I am glad there is some agreement >>>> about >>>> possible ways forward, and so I'm happy to try to use this as an >>>> opportunity to explore simpler, more reliable methods than the >>>> install >>>> script. >>>> >>>> This afternoon, I spent a bit of time with Conda, and I think >>>> it's >>>> quite nice. There are a few rough corners, particularly related >>>> to >>>> the binstar service, but it's so far pretty great. With Nathan's >>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>> and >>>> then install it. >>>> >>>> The workflow that seems to work: >>>> >>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>> * Run the installer for miniconda >>>> * Enter the conda environment and then install yt by doing "conda >>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>> >>>> I think that this can likely all be stuck into a bash script. A >>>> simple, first pass at this is here: >>>> >>>> http://paste.yt-project.org/show/3833/ >>>> >>>> This right now only works on Linux x86_64, but getting it to work >>>> for >>>> other machines won't be too hard. I suspect we will be able to >>>> do >>>> nightlies very easily as well. If anyone out there has an x86_64 >>>> machine they wouldn't mind trying it on, that would be very >>>> helpful! >>>> I did find that once I ran this script, I had to actually prepend >>>> the >>>> PATH afterwards as well. This means doing: >>>> >>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>> export PATH=$HOME/yt-conda:$PATH >>>> source activate ytenv >>>> >>>> At that point, everything was set up and working for me. The >>>> miniconda install offers to add paths to .bashrc, but I don't >>>> think >>>> we >>>> should go down that route. That being said, this is also a >>>> possible >>>> point of friction. >>>> >>>> One nice thing is that this also completely works with the full >>>> anaconda; if someone wants everything that is in the anaconda >>>> install, >>>> they can even simply do "conda install anaconda" from the command >>>> line >>>> to get it. But the stripped down subset is the default. >>>> >>>> If anyone has a chance to try this out and has feedback, I'd >>>> greatly >>>> appreciate it! I think Nathan has done something very similar >>>> for >>>> OSX. I've also put a couple simple conda recipes here: >>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>>> basis >>>> for distributing builds and setting them up on buildbots and the >>>> like. >>>> I'm pretty optimistic about this. >>>> >>>> -Matt >>>> >>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>> nathan12343@gmail.com wrote: >>>>> I think to get everything working in a sustainable fashion, we >>>>> would need >>>>> buildbots for all platform combinations that we want to support, >>>>> so >>>>> all >>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>> py27/py3.3) tuple. >>>>> At the moment anaconda seems to support 32 and 64 bit linux, 64 >>>>> bit >>>>> OS X >>>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>>> windows. >>>>> >>>>> Another option is to rely on conda build, which compiles >>>>> everything >>>>> from >>>>> source. >>>>> >>>>> >>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I have less of a skin in this than I used to, but I'd like to >>>>>> raise >>>>>> the issue of Windows & package managers. For example, Anaconda >>>>>> is >>>>>> available for Windows - would that mean that yt might "just >>>>>> work" >>>>>> on >>>>>> Windows? Or the opposite, and it would require a great deal of >>>>>> effort >>>>>> to get all the various things we expect to be .so's to work as >>>>>> .dll's >>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>> >>>>>> I don't know the answers to these questions, but I think it's >>>>>> worth >>>>>> thinking about. >>>>>> >>>>>> -- >>>>>> Stephen Skory >>>>>> s@skory.us >>>>>> http://stephenskory.com/ >>>>>> 510.621.3687 (google voice) >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Here's an updated version of the shell script Matt sent out earlier today that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways, most of which can be readily scraped from the install script:
* Install in the folder that the script was invoked in rather than $HOME * Add an option to install anaconda with a warning about needing a couple hundred megabytes of space * Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages and compilers (i.e. still need build-essentials on ubuntu and Xcode on OSX). * Add a warning at the end about modifying .bashrc for PATH and LD_LIBRARY_PATH or supply an activate script that sets them for us. * Clone yt-doc, yt-supplemental and the main yt repo and put them in the yt-conda directory. * Check if conda is already available and (optionally?) skip the miniconda step. * Ensure the user that yt loves them.
As Matt suggested on a Conda github issue ( https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101) it would be nice to first try to conda install yt and then if that fails use conda build. I think it should be possible to set this up as part of the bash script. As asmurer notes in the issue, general support for this sort of mixed binary/source distribution in conda might be difficult but with yt it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to work correctly on windows.
So long as we provide binary yt packages of (at least) the stable releases, this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out of the box.
-Nathan
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little
concerned
that there are binary incompatibilities that conda can't resolve.
Perhaps
this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev
install
script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I
think
because we use named branches in that repository.
Yup -- you can do it like this:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was
the
workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith <brittonsmith@gmail.com
wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and
do
not have regular internet access.
I really like this idea of conda, especially as a package manager
that
only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another
week
or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote: > > Hi Nathan, > > That appears to work as it built the environment and `conda install > ...` > added packages to my environment. > > One mistake I made was that I originally downloaded the "latest"
OS X
> build of Miniconda, but that happened to be Miniconda3, which is > python 3 > based. Trying to build the environment with that yields an error > regarding > incompatibility of yt and python3.3, as it should. > > Chris > > > On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum > nathan12343@gmail.com > wrote: >> >> Hey Chris, >> >> I don't think mercurial is strictly necessary, can you try again >> without >> it? I think if Matt uploads a mercurial package for OS X this
won't
>> be an >> issue. I'll send him an updated tarball. >> >> I submitted a mercurial recipe to conda-recipes yesterday >> (https://github.com/ContinuumIO/conda-recipes/pull/14) so
hopefully
>> a >> mercurial build will be included in future anaconda releases. >> >> >> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >> chris.m.malone@gmail.com wrote: >>> >>> I just tried setting this up on OS X 10.7.5 and failed when >>> attempting >>> to create the conda environment due to a missing mercurial
package:
>>> >>> $ conda create -n ytenv -c http://conda.binstar.org/yt_projectyt >>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>> Error: No packages found matching: mercurial >>> >>> >>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>> nathan12343@gmail.com wrote: >>>> >>>> Yup, please try on OSX as well. If you make sure Matt's binstar >>>> is in >>>> your .condarc, you should be able to get yt by doing 'conda >>>> install yt'. >>>> >>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>> about >>>> issues, particularly if there are issues on older OS X releases. >>>> >>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>> >>>>> Hi all, >>>>> >>>>> Thank you for the feedback -- I am glad there is some agreement >>>>> about >>>>> possible ways forward, and so I'm happy to try to use this as
an
>>>>> opportunity to explore simpler, more reliable methods than the >>>>> install >>>>> script. >>>>> >>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>> it's >>>>> quite nice. There are a few rough corners, particularly
related
>>>>> to >>>>> the binstar service, but it's so far pretty great. With
Nathan's
>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>> and >>>>> then install it. >>>>> >>>>> The workflow that seems to work: >>>>> >>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>>> * Run the installer for miniconda >>>>> * Enter the conda environment and then install yt by doing
"conda
>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>> >>>>> I think that this can likely all be stuck into a bash script.
A
>>>>> simple, first pass at this is here: >>>>> >>>>> http://paste.yt-project.org/show/3833/ >>>>> >>>>> This right now only works on Linux x86_64, but getting it to
work
>>>>> for >>>>> other machines won't be too hard. I suspect we will be able to >>>>> do >>>>> nightlies very easily as well. If anyone out there has an
x86_64
>>>>> machine they wouldn't mind trying it on, that would be very >>>>> helpful! >>>>> I did find that once I ran this script, I had to actually
prepend
>>>>> the >>>>> PATH afterwards as well. This means doing: >>>>> >>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>> export PATH=$HOME/yt-conda:$PATH >>>>> source activate ytenv >>>>> >>>>> At that point, everything was set up and working for me. The >>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>> think >>>>> we >>>>> should go down that route. That being said, this is also a >>>>> possible >>>>> point of friction. >>>>> >>>>> One nice thing is that this also completely works with the full >>>>> anaconda; if someone wants everything that is in the anaconda >>>>> install, >>>>> they can even simply do "conda install anaconda" from the
command
>>>>> line >>>>> to get it. But the stripped down subset is the default. >>>>> >>>>> If anyone has a chance to try this out and has feedback, I'd >>>>> greatly >>>>> appreciate it! I think Nathan has done something very similar >>>>> for >>>>> OSX. I've also put a couple simple conda recipes here: >>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use
as a
>>>>> basis >>>>> for distributing builds and setting them up on buildbots and
the
>>>>> like. >>>>> I'm pretty optimistic about this. >>>>> >>>>> -Matt >>>>> >>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>> nathan12343@gmail.com wrote: >>>>>> I think to get everything working in a sustainable fashion, we >>>>>> would need >>>>>> buildbots for all platform combinations that we want to
support,
>>>>>> so >>>>>> all >>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>> py27/py3.3) tuple. >>>>>> At the moment anaconda seems to support 32 and 64 bit linux,
64
>>>>>> bit >>>>>> OS X >>>>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>>>> windows. >>>>>> >>>>>> Another option is to rely on conda build, which compiles >>>>>> everything >>>>>> from >>>>>> source. >>>>>> >>>>>> >>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>> wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have less of a skin in this than I used to, but I'd like to >>>>>>> raise >>>>>>> the issue of Windows & package managers. For example,
Anaconda
>>>>>>> is >>>>>>> available for Windows - would that mean that yt might "just >>>>>>> work" >>>>>>> on >>>>>>> Windows? Or the opposite, and it would require a great deal
of
>>>>>>> effort >>>>>>> to get all the various things we expect to be .so's to work
as
>>>>>>> .dll's >>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>> >>>>>>> I don't know the answers to these questions, but I think it's >>>>>>> worth >>>>>>> thinking about. >>>>>>> >>>>>>> -- >>>>>>> Stephen Skory >>>>>>> s@skory.us >>>>>>> http://stephenskory.com/ >>>>>>> 510.621.3687 (google voice) >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
On Wed, Sep 4, 2013 at 2:54 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Here's an updated version of the shell script Matt sent out earlier today that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways, most of which can be readily scraped from the install script:
Me too. This is much improved over mine.
* Install in the folder that the script was invoked in rather than $HOME
Good idea.
* Add an option to install anaconda with a warning about needing acouple hundred megabytes of space
Excellent! I like this a lot. And, we should point to the Anaconda page for more info about packages and how Anaconda works.
* Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages andcompilers (i.e. still need build-essentials on ubuntu and Xcode on OSX). * Add a warning at the end about modifying .bashrc for PATH and LD_LIBRARY_PATH or supply an activate script that sets them for us.
I think I'd prefer to supply an activate script that both activates Conda *and* the yt environment in it. That way individuals who want to use Conda directly can do so, but we also make it possible to keep the behavior that people have already.
* Clone yt-doc, yt-supplemental and the main yt repo and put them in theyt-conda directory.
Agreed.
* Check if conda is already available and (optionally?) skip theminiconda step.
I'm not sure about this, since I think we will then just tell people to install their own yt. I think the install script should not be *too* complex in this regard.
* Ensure the user that yt loves them.
Yes.
As Matt suggested on a Conda github issue (https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101) it would be nice to first try to conda install yt and then if that fails use conda build. I think it should be possible to set this up as part of the bash script. As asmurer notes in the issue, general support for this sort of mixed binary/source distribution in conda might be difficult but with yt it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to work correctly on windows.
So long as we provide binary yt packages of (at least) the stable releases, this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out of the box.
-Nathan
Awesome work, and I am getting more excited about this as time goes on. I am going to try my hand at updating the new script in the way you mentioned, and then I will try checking it into the repo and issue a PR so we can iterate there. I hope to get to this by tomorrow morning.
-Matt
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
Yup -- you can do it like this:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote: > > Everything should be available now for 64 bit linux and OS X. > > > On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone > chris.m.malone@gmail.com > wrote: >> >> Hi Nathan, >> >> That appears to work as it built the environment and `conda >> install >> ...` >> added packages to my environment. >> >> One mistake I made was that I originally downloaded the "latest" >> OS X >> build of Miniconda, but that happened to be Miniconda3, which is >> python 3 >> based. Trying to build the environment with that yields an error >> regarding >> incompatibility of yt and python3.3, as it should. >> >> Chris >> >> >> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >> nathan12343@gmail.com >> wrote: >>> >>> Hey Chris, >>> >>> I don't think mercurial is strictly necessary, can you try again >>> without >>> it? I think if Matt uploads a mercurial package for OS X this >>> won't >>> be an >>> issue. I'll send him an updated tarball. >>> >>> I submitted a mercurial recipe to conda-recipes yesterday >>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>> hopefully >>> a >>> mercurial build will be included in future anaconda releases. >>> >>> >>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>> chris.m.malone@gmail.com wrote: >>>> >>>> I just tried setting this up on OS X 10.7.5 and failed when >>>> attempting >>>> to create the conda environment due to a missing mercurial >>>> package: >>>> >>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project >>>> yt >>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>> Error: No packages found matching: mercurial >>>> >>>> >>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>> nathan12343@gmail.com wrote: >>>>> >>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>> binstar >>>>> is in >>>>> your .condarc, you should be able to get yt by doing 'conda >>>>> install yt'. >>>>> >>>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>>> about >>>>> issues, particularly if there are issues on older OS X >>>>> releases. >>>>> >>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> Thank you for the feedback -- I am glad there is some >>>>>> agreement >>>>>> about >>>>>> possible ways forward, and so I'm happy to try to use this as >>>>>> an >>>>>> opportunity to explore simpler, more reliable methods than the >>>>>> install >>>>>> script. >>>>>> >>>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>>> it's >>>>>> quite nice. There are a few rough corners, particularly >>>>>> related >>>>>> to >>>>>> the binstar service, but it's so far pretty great. With >>>>>> Nathan's >>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>>> and >>>>>> then install it. >>>>>> >>>>>> The workflow that seems to work: >>>>>> >>>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>>>> * Run the installer for miniconda >>>>>> * Enter the conda environment and then install yt by doing >>>>>> "conda >>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>> >>>>>> I think that this can likely all be stuck into a bash script. >>>>>> A >>>>>> simple, first pass at this is here: >>>>>> >>>>>> http://paste.yt-project.org/show/3833/ >>>>>> >>>>>> This right now only works on Linux x86_64, but getting it to >>>>>> work >>>>>> for >>>>>> other machines won't be too hard. I suspect we will be able >>>>>> to >>>>>> do >>>>>> nightlies very easily as well. If anyone out there has an >>>>>> x86_64 >>>>>> machine they wouldn't mind trying it on, that would be very >>>>>> helpful! >>>>>> I did find that once I ran this script, I had to actually >>>>>> prepend >>>>>> the >>>>>> PATH afterwards as well. This means doing: >>>>>> >>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>> source activate ytenv >>>>>> >>>>>> At that point, everything was set up and working for me. The >>>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>>> think >>>>>> we >>>>>> should go down that route. That being said, this is also a >>>>>> possible >>>>>> point of friction. >>>>>> >>>>>> One nice thing is that this also completely works with the >>>>>> full >>>>>> anaconda; if someone wants everything that is in the anaconda >>>>>> install, >>>>>> they can even simply do "conda install anaconda" from the >>>>>> command >>>>>> line >>>>>> to get it. But the stripped down subset is the default. >>>>>> >>>>>> If anyone has a chance to try this out and has feedback, I'd >>>>>> greatly >>>>>> appreciate it! I think Nathan has done something very similar >>>>>> for >>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as >>>>>> a >>>>>> basis >>>>>> for distributing builds and setting them up on buildbots and >>>>>> the >>>>>> like. >>>>>> I'm pretty optimistic about this. >>>>>> >>>>>> -Matt >>>>>> >>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>> nathan12343@gmail.com wrote: >>>>>>> I think to get everything working in a sustainable fashion, >>>>>>> we >>>>>>> would need >>>>>>> buildbots for all platform combinations that we want to >>>>>>> support, >>>>>>> so >>>>>>> all >>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>> py27/py3.3) tuple. >>>>>>> At the moment anaconda seems to support 32 and 64 bit linux, >>>>>>> 64 >>>>>>> bit >>>>>>> OS X >>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>> bit >>>>>>> windows. >>>>>>> >>>>>>> Another option is to rely on conda build, which compiles >>>>>>> everything >>>>>>> from >>>>>>> source. >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I have less of a skin in this than I used to, but I'd like >>>>>>>> to >>>>>>>> raise >>>>>>>> the issue of Windows & package managers. For example, >>>>>>>> Anaconda >>>>>>>> is >>>>>>>> available for Windows - would that mean that yt might "just >>>>>>>> work" >>>>>>>> on >>>>>>>> Windows? Or the opposite, and it would require a great deal >>>>>>>> of >>>>>>>> effort >>>>>>>> to get all the various things we expect to be .so's to work >>>>>>>> as >>>>>>>> .dll's >>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>> >>>>>>>> I don't know the answers to these questions, but I think >>>>>>>> it's >>>>>>>> worth >>>>>>>> thinking about. >>>>>>>> >>>>>>>> -- >>>>>>>> Stephen Skory >>>>>>>> s@skory.us >>>>>>>> http://stephenskory.com/ >>>>>>>> 510.621.3687 (google voice) >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
I've issued a PR with my first pass at the get_yt.sh script. I've ported over a bunch of functionality from the old install_script.sh and it works for me in a clean install on Ubuntu, although it does require the "chrpath" package which I oddly did not have.
https://bitbucket.org/yt_analysis/yt/pull-request/591/first-draft-of-a-new-g...
What would be really helpful is if a few people could try it out, make modifications, commit, push to their fork, and I will then pull in those changes until it does everything it needs to. I think what I would like to preserve is that the "get_yt.sh" script should *not* be about how to install Conda/Anaconda -- it should get yt installed, and the yt environment. The case of installing yt *into* Conda is what I'd like to cover by providing binary packages, and I suspect that our audience for that will grow with time. This script aims along a slightly different vector.
I'm still not entirely sure how to handle source distributions; even with the yt-dev recipe Nathan has, it's still building binary packages. I've done my best to include the source as-is in the script, but to also use the yt recipe that Nathan has worked on. I'm not sure how the "yt-dev" recipe fits into that. Kacper, Nathan, any ideas about that? Would it just be the basic nightly package builder? Then we could have it get rebuild and reuploaded all the time; this would help with keeping people up to date, but *not* with encouraging contributions.
-Matt
PS One last thing is that our whole "get yt" documentation should be re-done. We should be much more clear about how and where to get the package, and reduce the number of options.
On Fri, Sep 6, 2013 at 9:08 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Wed, Sep 4, 2013 at 2:54 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Here's an updated version of the shell script Matt sent out earlier today that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways, most of which can be readily scraped from the install script:
Me too. This is much improved over mine.
* Install in the folder that the script was invoked in rather than $HOMEGood idea.
* Add an option to install anaconda with a warning about needing acouple hundred megabytes of space
Excellent! I like this a lot. And, we should point to the Anaconda page for more info about packages and how Anaconda works.
* Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages andcompilers (i.e. still need build-essentials on ubuntu and Xcode on OSX). * Add a warning at the end about modifying .bashrc for PATH and LD_LIBRARY_PATH or supply an activate script that sets them for us.
I think I'd prefer to supply an activate script that both activates Conda *and* the yt environment in it. That way individuals who want to use Conda directly can do so, but we also make it possible to keep the behavior that people have already.
* Clone yt-doc, yt-supplemental and the main yt repo and put them in theyt-conda directory.
Agreed.
* Check if conda is already available and (optionally?) skip theminiconda step.
I'm not sure about this, since I think we will then just tell people to install their own yt. I think the install script should not be *too* complex in this regard.
* Ensure the user that yt loves them.Yes.
As Matt suggested on a Conda github issue (https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101) it would be nice to first try to conda install yt and then if that fails use conda build. I think it should be possible to set this up as part of the bash script. As asmurer notes in the issue, general support for this sort of mixed binary/source distribution in conda might be difficult but with yt it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to work correctly on windows.
So long as we provide binary yt packages of (at least) the stable releases, this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out of the box.
-Nathan
Awesome work, and I am getting more excited about this as time goes on. I am going to try my hand at updating the new script in the way you mentioned, and then I will try checking it into the repo and issue a PR so we can iterate there. I hope to get to this by tomorrow morning.
-Matt
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
Yup -- you can do it like this:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote: > Hi everyone, > > Sorry for chiming in late. I just moved when this thread began and > do > not > have regular internet access. > > I really like this idea of conda, especially as a package manager > that > only > optionally makes its own edits to your .bashrc. I have always > really > liked > that the install script creates a clean python stack with basically > everything a python user needs. I have on occasion suggested it to > people > just looking to use numpy and matploblib. It looks like conda will > continue > to provide this nice by-product, so I'm all for it. > > I won't be in a position to help with testing and such for another > week > or > so when I get regular internet access, but I would be glad to do so > then. > > Britton > > > On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum > nathan12343@gmail.com > wrote: >> >> Everything should be available now for 64 bit linux and OS X. >> >> >> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone >> chris.m.malone@gmail.com >> wrote: >>> >>> Hi Nathan, >>> >>> That appears to work as it built the environment and `conda >>> install >>> ...` >>> added packages to my environment. >>> >>> One mistake I made was that I originally downloaded the "latest" >>> OS X >>> build of Miniconda, but that happened to be Miniconda3, which is >>> python 3 >>> based. Trying to build the environment with that yields an error >>> regarding >>> incompatibility of yt and python3.3, as it should. >>> >>> Chris >>> >>> >>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >>> nathan12343@gmail.com >>> wrote: >>>> >>>> Hey Chris, >>>> >>>> I don't think mercurial is strictly necessary, can you try again >>>> without >>>> it? I think if Matt uploads a mercurial package for OS X this >>>> won't >>>> be an >>>> issue. I'll send him an updated tarball. >>>> >>>> I submitted a mercurial recipe to conda-recipes yesterday >>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>>> hopefully >>>> a >>>> mercurial build will be included in future anaconda releases. >>>> >>>> >>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>>> chris.m.malone@gmail.com wrote: >>>>> >>>>> I just tried setting this up on OS X 10.7.5 and failed when >>>>> attempting >>>>> to create the conda environment due to a missing mercurial >>>>> package: >>>>> >>>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project >>>>> yt >>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>>> Error: No packages found matching: mercurial >>>>> >>>>> >>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>>> nathan12343@gmail.com wrote: >>>>>> >>>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>>> binstar >>>>>> is in >>>>>> your .condarc, you should be able to get yt by doing 'conda >>>>>> install yt'. >>>>>> >>>>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>>>> about >>>>>> issues, particularly if there are issues on older OS X >>>>>> releases. >>>>>> >>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Thank you for the feedback -- I am glad there is some >>>>>>> agreement >>>>>>> about >>>>>>> possible ways forward, and so I'm happy to try to use this as >>>>>>> an >>>>>>> opportunity to explore simpler, more reliable methods than the >>>>>>> install >>>>>>> script. >>>>>>> >>>>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>>>> it's >>>>>>> quite nice. There are a few rough corners, particularly >>>>>>> related >>>>>>> to >>>>>>> the binstar service, but it's so far pretty great. With >>>>>>> Nathan's >>>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>>>> and >>>>>>> then install it. >>>>>>> >>>>>>> The workflow that seems to work: >>>>>>> >>>>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>>>>> * Run the installer for miniconda >>>>>>> * Enter the conda environment and then install yt by doing >>>>>>> "conda >>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>>> >>>>>>> I think that this can likely all be stuck into a bash script. >>>>>>> A >>>>>>> simple, first pass at this is here: >>>>>>> >>>>>>> http://paste.yt-project.org/show/3833/ >>>>>>> >>>>>>> This right now only works on Linux x86_64, but getting it to >>>>>>> work >>>>>>> for >>>>>>> other machines won't be too hard. I suspect we will be able >>>>>>> to >>>>>>> do >>>>>>> nightlies very easily as well. If anyone out there has an >>>>>>> x86_64 >>>>>>> machine they wouldn't mind trying it on, that would be very >>>>>>> helpful! >>>>>>> I did find that once I ran this script, I had to actually >>>>>>> prepend >>>>>>> the >>>>>>> PATH afterwards as well. This means doing: >>>>>>> >>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>>> source activate ytenv >>>>>>> >>>>>>> At that point, everything was set up and working for me. The >>>>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>>>> think >>>>>>> we >>>>>>> should go down that route. That being said, this is also a >>>>>>> possible >>>>>>> point of friction. >>>>>>> >>>>>>> One nice thing is that this also completely works with the >>>>>>> full >>>>>>> anaconda; if someone wants everything that is in the anaconda >>>>>>> install, >>>>>>> they can even simply do "conda install anaconda" from the >>>>>>> command >>>>>>> line >>>>>>> to get it. But the stripped down subset is the default. >>>>>>> >>>>>>> If anyone has a chance to try this out and has feedback, I'd >>>>>>> greatly >>>>>>> appreciate it! I think Nathan has done something very similar >>>>>>> for >>>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as >>>>>>> a >>>>>>> basis >>>>>>> for distributing builds and setting them up on buildbots and >>>>>>> the >>>>>>> like. >>>>>>> I'm pretty optimistic about this. >>>>>>> >>>>>>> -Matt >>>>>>> >>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>>> nathan12343@gmail.com wrote: >>>>>>>> I think to get everything working in a sustainable fashion, >>>>>>>> we >>>>>>>> would need >>>>>>>> buildbots for all platform combinations that we want to >>>>>>>> support, >>>>>>>> so >>>>>>>> all >>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>>> py27/py3.3) tuple. >>>>>>>> At the moment anaconda seems to support 32 and 64 bit linux, >>>>>>>> 64 >>>>>>>> bit >>>>>>>> OS X >>>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>>> bit >>>>>>>> windows. >>>>>>>> >>>>>>>> Another option is to rely on conda build, which compiles >>>>>>>> everything >>>>>>>> from >>>>>>>> source. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I have less of a skin in this than I used to, but I'd like >>>>>>>>> to >>>>>>>>> raise >>>>>>>>> the issue of Windows & package managers. For example, >>>>>>>>> Anaconda >>>>>>>>> is >>>>>>>>> available for Windows - would that mean that yt might "just >>>>>>>>> work" >>>>>>>>> on >>>>>>>>> Windows? Or the opposite, and it would require a great deal >>>>>>>>> of >>>>>>>>> effort >>>>>>>>> to get all the various things we expect to be .so's to work >>>>>>>>> as >>>>>>>>> .dll's >>>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>>> >>>>>>>>> I don't know the answers to these questions, but I think >>>>>>>>> it's >>>>>>>>> worth >>>>>>>>> thinking about. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Stephen Skory >>>>>>>>> s@skory.us >>>>>>>>> http://stephenskory.com/ >>>>>>>>> 510.621.3687 (google voice) >>>>>>>>> _______________________________________________ >>>>>>>>> yt-dev mailing list >>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
The yt-dev recipe I created would only be good for nightly binary builds. I think that would only be useful for people who want the latest and greatest but don't want to set up a build environment, which I suspect is a relatively small group of people. We'll have to decide at some other point if we want to have buildbots and nightly builds.
Agreed about updating our documentation and website. I think we should supply both the script and instructions for setting up yt using `pip` and already-installed conda packages. We should also link to a page that lists the packages needed to manually set up yt.
On Sat, Sep 7, 2013 at 8:22 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
I've issued a PR with my first pass at the get_yt.sh script. I've ported over a bunch of functionality from the old install_script.sh and it works for me in a clean install on Ubuntu, although it does require the "chrpath" package which I oddly did not have.
https://bitbucket.org/yt_analysis/yt/pull-request/591/first-draft-of-a-new-g...
What would be really helpful is if a few people could try it out, make modifications, commit, push to their fork, and I will then pull in those changes until it does everything it needs to. I think what I would like to preserve is that the "get_yt.sh" script should *not* be about how to install Conda/Anaconda -- it should get yt installed, and the yt environment. The case of installing yt *into* Conda is what I'd like to cover by providing binary packages, and I suspect that our audience for that will grow with time. This script aims along a slightly different vector.
I'm still not entirely sure how to handle source distributions; even with the yt-dev recipe Nathan has, it's still building binary packages. I've done my best to include the source as-is in the script, but to also use the yt recipe that Nathan has worked on. I'm not sure how the "yt-dev" recipe fits into that. Kacper, Nathan, any ideas about that? Would it just be the basic nightly package builder? Then we could have it get rebuild and reuploaded all the time; this would help with keeping people up to date, but *not* with encouraging contributions.
-Matt
PS One last thing is that our whole "get yt" documentation should be re-done. We should be much more clear about how and where to get the package, and reduce the number of options.
On Fri, Sep 6, 2013 at 9:08 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Wed, Sep 4, 2013 at 2:54 AM, Nathan Goldbaum nathan12343@gmail.com
wrote:
Here's an updated version of the shell script Matt sent out earlier
today
that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways,
most
of which can be readily scraped from the install script:
Me too. This is much improved over mine.
* Install in the folder that the script was invoked in rather than$HOME
Good idea.
* Add an option to install anaconda with a warning about needing acouple hundred megabytes of space
Excellent! I like this a lot. And, we should point to the Anaconda page for more info about packages and how Anaconda works.
* Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages andcompilers (i.e. still need build-essentials on ubuntu and Xcode on OSX). * Add a warning at the end about modifying .bashrc for PATH and LD_LIBRARY_PATH or supply an activate script that sets them for us.
I think I'd prefer to supply an activate script that both activates Conda *and* the yt environment in it. That way individuals who want to use Conda directly can do so, but we also make it possible to keep the behavior that people have already.
* Clone yt-doc, yt-supplemental and the main yt repo and put themin the
yt-conda directory.
Agreed.
* Check if conda is already available and (optionally?) skip theminiconda step.
I'm not sure about this, since I think we will then just tell people to install their own yt. I think the install script should not be *too* complex in this regard.
* Ensure the user that yt loves them.Yes.
As Matt suggested on a Conda github issue (https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101)
it
would be nice to first try to conda install yt and then if that fails
use
conda build. I think it should be possible to set this up as part of
the
bash script. As asmurer notes in the issue, general support for this
sort
of mixed binary/source distribution in conda might be difficult but
with yt
it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to
work
correctly on windows.
So long as we provide binary yt packages of (at least) the stable
releases,
this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out of
the
box.
-Nathan
Awesome work, and I am getting more excited about this as time goes on. I am going to try my hand at updating the new script in the way you mentioned, and then I will try checking it into the repo and issue a PR so we can iterate there. I hope to get to this by tomorrow morning.
-Matt
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
Yup -- you can do it like this:
wget
https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com
wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but
the
binary had an incompatibility which the glibc that was installed on
the
machine. Using pip to install Mercurial (which did it from source)
was
the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com
wrote:
> Hi all, > > Thanks to everybody who has reported back on testing. After some > talking both offline and on IRC, as well as here, I think we would > need to do the following things: > > * Make a single script that grabs the appropriate distribution of > miniconda and installs it. Right now I have a mechanism for doing > this, but it's currently tied to an architecture. > * Create a mechanism for installing all the packages we need.
Nearly
> all are available inside the Continuum repos. What we're hung up
on
> is that source installs require a "recipe", and transmitting the > recipe is where I don't have an idea of what to do. > * Test this out lots of places > * Clean up the edges in the (new) install script > * Move the old install script to maintenance mode > * Update all documentation to describe this and mothball other > methods of installation > > What would be nice: > > * Make available nightly builds of yt on several architectures
using
> binstar > * Utilize more of the packages included in conda elsewhere in yt,
now
> that we can! > > Here's my current recipe for get_yt.sh: > > http://paste.yt-project.org/show/3843/ > > (The config thing may get switched to include the --system
argument,
> to modify the "yt-conda" condarc.) The step that I'm most stuck
on
> is > getting the yt recipe to people. If we want to make it possible
and
> easy to build from source, we need to get the contents of a "conda > recipe" to people. They can then run "conda build ." in the > directory. Here are the recipes that we've been playing with: > > https://bitbucket.org/yt_analysis/yt_conda/src > > Basically, in get_yt.sh, to do from source instead of from binary
we
> need to insert a step at the end that downloads the recipes
somehow
> and then cd's into the right directory and builds them. The
reason
> this is tricky is that we often need to bootstrap ourselves; we
can't
> assume anything exists. We can download the .tar.bz2 of the
current
> tip of the repo, but it includes the hash in the directory name
that
> it extracts to. > > So I think what we need is a mechanism for getting the current
state
> of the repo, figuring out the name of the repo's directory, moves > into > it, and then builds. I believe that all/most of this becomes
much,
> much easier if hg gets included in Anaconda, which Nathan has > submitted a PR for. So hopefully that will be taken care of, but > until that time we can possibly figure something out. I'm not
sure
> that we have the resources to continually support binary nightly > builds in perpetuity for all the architectures that people run
on, so
> having source would be awesome. Plus, one of the biggest appeals
of
> how we distribute yt is that the source is included; I would very > much > not like to give this up. > > Thoughts? Has anyone else tested any of this out? > > -Matt > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > brittonsmith@gmail.com > wrote: >> Hi everyone, >> >> Sorry for chiming in late. I just moved when this thread began
and
>> do >> not >> have regular internet access. >> >> I really like this idea of conda, especially as a package manager >> that >> only >> optionally makes its own edits to your .bashrc. I have always >> really >> liked >> that the install script creates a clean python stack with
basically
>> everything a python user needs. I have on occasion suggested it
to
>> people >> just looking to use numpy and matploblib. It looks like conda
will
>> continue >> to provide this nice by-product, so I'm all for it. >> >> I won't be in a position to help with testing and such for
another
>> week >> or >> so when I get regular internet access, but I would be glad to do
so
>> then. >> >> Britton >> >> >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum >> nathan12343@gmail.com >> wrote: >>> >>> Everything should be available now for 64 bit linux and OS X. >>> >>> >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone >>> chris.m.malone@gmail.com >>> wrote: >>>> >>>> Hi Nathan, >>>> >>>> That appears to work as it built the environment and `conda >>>> install >>>> ...` >>>> added packages to my environment. >>>> >>>> One mistake I made was that I originally downloaded the
"latest"
>>>> OS X >>>> build of Miniconda, but that happened to be Miniconda3, which
is
>>>> python 3 >>>> based. Trying to build the environment with that yields an
error
>>>> regarding >>>> incompatibility of yt and python3.3, as it should. >>>> >>>> Chris >>>> >>>> >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >>>> nathan12343@gmail.com >>>> wrote: >>>>> >>>>> Hey Chris, >>>>> >>>>> I don't think mercurial is strictly necessary, can you try
again
>>>>> without >>>>> it? I think if Matt uploads a mercurial package for OS X this >>>>> won't >>>>> be an >>>>> issue. I'll send him an updated tarball. >>>>> >>>>> I submitted a mercurial recipe to conda-recipes yesterday >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>>>> hopefully >>>>> a >>>>> mercurial build will be included in future anaconda releases. >>>>> >>>>> >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>>>> chris.m.malone@gmail.com wrote: >>>>>> >>>>>> I just tried setting this up on OS X 10.7.5 and failed when >>>>>> attempting >>>>>> to create the conda environment due to a missing mercurial >>>>>> package: >>>>>> >>>>>> $ conda create -n ytenv -c
http://conda.binstar.org/yt_project
>>>>>> yt >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>>>> Error: No packages found matching: mercurial >>>>>> >>>>>> >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>>>> nathan12343@gmail.com wrote: >>>>>>> >>>>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>>>> binstar >>>>>>> is in >>>>>>> your .condarc, you should be able to get yt by doing 'conda >>>>>>> install yt'. >>>>>>> >>>>>>> I built the OSX binary on my laptop so I'd appreciate
hearing
>>>>>>> about >>>>>>> issues, particularly if there are issues on older OS X >>>>>>> releases. >>>>>>> >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Thank you for the feedback -- I am glad there is some >>>>>>>> agreement >>>>>>>> about >>>>>>>> possible ways forward, and so I'm happy to try to use this
as
>>>>>>>> an >>>>>>>> opportunity to explore simpler, more reliable methods than
the
>>>>>>>> install >>>>>>>> script. >>>>>>>> >>>>>>>> This afternoon, I spent a bit of time with Conda, and I
think
>>>>>>>> it's >>>>>>>> quite nice. There are a few rough corners, particularly >>>>>>>> related >>>>>>>> to >>>>>>>> the binstar service, but it's so far pretty great. With >>>>>>>> Nathan's >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux
x86_64
>>>>>>>> and >>>>>>>> then install it. >>>>>>>> >>>>>>>> The workflow that seems to work: >>>>>>>> >>>>>>>> * Get miniconda:
http://repo.continuum.io/miniconda/index.html
>>>>>>>> * Run the installer for miniconda >>>>>>>> * Enter the conda environment and then install yt by doing >>>>>>>> "conda >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>>>> >>>>>>>> I think that this can likely all be stuck into a bash
script.
>>>>>>>> A >>>>>>>> simple, first pass at this is here: >>>>>>>> >>>>>>>> http://paste.yt-project.org/show/3833/ >>>>>>>> >>>>>>>> This right now only works on Linux x86_64, but getting it
to
>>>>>>>> work >>>>>>>> for >>>>>>>> other machines won't be too hard. I suspect we will be
able
>>>>>>>> to >>>>>>>> do >>>>>>>> nightlies very easily as well. If anyone out there has an >>>>>>>> x86_64 >>>>>>>> machine they wouldn't mind trying it on, that would be very >>>>>>>> helpful! >>>>>>>> I did find that once I ran this script, I had to actually >>>>>>>> prepend >>>>>>>> the >>>>>>>> PATH afterwards as well. This means doing: >>>>>>>> >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>>>> source activate ytenv >>>>>>>> >>>>>>>> At that point, everything was set up and working for me.
The
>>>>>>>> miniconda install offers to add paths to .bashrc, but I
don't
>>>>>>>> think >>>>>>>> we >>>>>>>> should go down that route. That being said, this is also a >>>>>>>> possible >>>>>>>> point of friction. >>>>>>>> >>>>>>>> One nice thing is that this also completely works with the >>>>>>>> full >>>>>>>> anaconda; if someone wants everything that is in the
anaconda
>>>>>>>> install, >>>>>>>> they can even simply do "conda install anaconda" from the >>>>>>>> command >>>>>>>> line >>>>>>>> to get it. But the stripped down subset is the default. >>>>>>>> >>>>>>>> If anyone has a chance to try this out and has feedback,
I'd
>>>>>>>> greatly >>>>>>>> appreciate it! I think Nathan has done something very
similar
>>>>>>>> for >>>>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can
use as
>>>>>>>> a >>>>>>>> basis >>>>>>>> for distributing builds and setting them up on buildbots
and
>>>>>>>> the >>>>>>>> like. >>>>>>>> I'm pretty optimistic about this. >>>>>>>> >>>>>>>> -Matt >>>>>>>> >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>>>> nathan12343@gmail.com wrote: >>>>>>>>> I think to get everything working in a sustainable
fashion,
>>>>>>>>> we >>>>>>>>> would need >>>>>>>>> buildbots for all platform combinations that we want to >>>>>>>>> support, >>>>>>>>> so >>>>>>>>> all >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>>>> py27/py3.3) tuple. >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit
linux,
>>>>>>>>> 64 >>>>>>>>> bit >>>>>>>>> OS X >>>>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>>>> bit >>>>>>>>> windows. >>>>>>>>> >>>>>>>>> Another option is to rely on conda build, which compiles >>>>>>>>> everything >>>>>>>>> from >>>>>>>>> source. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <
s@skory.us>
>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have less of a skin in this than I used to, but I'd
like
>>>>>>>>>> to >>>>>>>>>> raise >>>>>>>>>> the issue of Windows & package managers. For example, >>>>>>>>>> Anaconda >>>>>>>>>> is >>>>>>>>>> available for Windows - would that mean that yt might
"just
>>>>>>>>>> work" >>>>>>>>>> on >>>>>>>>>> Windows? Or the opposite, and it would require a great
deal
>>>>>>>>>> of >>>>>>>>>> effort >>>>>>>>>> to get all the various things we expect to be .so's to
work
>>>>>>>>>> as >>>>>>>>>> .dll's >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>>>> >>>>>>>>>> I don't know the answers to these questions, but I think >>>>>>>>>> it's >>>>>>>>>> worth >>>>>>>>>> thinking about. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Stephen Skory >>>>>>>>>> s@skory.us >>>>>>>>>> http://stephenskory.com/ >>>>>>>>>> 510.621.3687 (google voice) >>>>>>>>>> _______________________________________________ >>>>>>>>>> yt-dev mailing list >>>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> yt-dev mailing list >>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I tried installing this script on Pleiades but had the same problem as Matt, with the requirement of the chrpath package. Of course, on Pleiades we don't have the option of installing it ourselves. Thoughts, besides asking them to install it? I guess that might be the only option.
On Sep 7, 2013, at 1:27 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
The yt-dev recipe I created would only be good for nightly binary builds. I think that would only be useful for people who want the latest and greatest but don't want to set up a build environment, which I suspect is a relatively small group of people. We'll have to decide at some other point if we want to have buildbots and nightly builds.
Agreed about updating our documentation and website. I think we should supply both the script and instructions for setting up yt using `pip` and already-installed conda packages. We should also link to a page that lists the packages needed to manually set up yt.
On Sat, Sep 7, 2013 at 8:22 AM, Matthew Turk matthewturk@gmail.com wrote: Hi all,
I've issued a PR with my first pass at the get_yt.sh script. I've ported over a bunch of functionality from the old install_script.sh and it works for me in a clean install on Ubuntu, although it does require the "chrpath" package which I oddly did not have.
https://bitbucket.org/yt_analysis/yt/pull-request/591/first-draft-of-a-new-g...
What would be really helpful is if a few people could try it out, make modifications, commit, push to their fork, and I will then pull in those changes until it does everything it needs to. I think what I would like to preserve is that the "get_yt.sh" script should *not* be about how to install Conda/Anaconda -- it should get yt installed, and the yt environment. The case of installing yt *into* Conda is what I'd like to cover by providing binary packages, and I suspect that our audience for that will grow with time. This script aims along a slightly different vector.
I'm still not entirely sure how to handle source distributions; even with the yt-dev recipe Nathan has, it's still building binary packages. I've done my best to include the source as-is in the script, but to also use the yt recipe that Nathan has worked on. I'm not sure how the "yt-dev" recipe fits into that. Kacper, Nathan, any ideas about that? Would it just be the basic nightly package builder? Then we could have it get rebuild and reuploaded all the time; this would help with keeping people up to date, but *not* with encouraging contributions.
-Matt
PS One last thing is that our whole "get yt" documentation should be re-done. We should be much more clear about how and where to get the package, and reduce the number of options.
On Fri, Sep 6, 2013 at 9:08 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Wed, Sep 4, 2013 at 2:54 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Here's an updated version of the shell script Matt sent out earlier today that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways, most of which can be readily scraped from the install script:
Me too. This is much improved over mine.
* Install in the folder that the script was invoked in rather than $HOMEGood idea.
* Add an option to install anaconda with a warning about needing acouple hundred megabytes of space
Excellent! I like this a lot. And, we should point to the Anaconda page for more info about packages and how Anaconda works.
* Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages andcompilers (i.e. still need build-essentials on ubuntu and Xcode on OSX). * Add a warning at the end about modifying .bashrc for PATH and LD_LIBRARY_PATH or supply an activate script that sets them for us.
I think I'd prefer to supply an activate script that both activates Conda *and* the yt environment in it. That way individuals who want to use Conda directly can do so, but we also make it possible to keep the behavior that people have already.
* Clone yt-doc, yt-supplemental and the main yt repo and put them in theyt-conda directory.
Agreed.
* Check if conda is already available and (optionally?) skip theminiconda step.
I'm not sure about this, since I think we will then just tell people to install their own yt. I think the install script should not be *too* complex in this regard.
* Ensure the user that yt loves them.Yes.
As Matt suggested on a Conda github issue (https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101) it would be nice to first try to conda install yt and then if that fails use conda build. I think it should be possible to set this up as part of the bash script. As asmurer notes in the issue, general support for this sort of mixed binary/source distribution in conda might be difficult but with yt it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to work correctly on windows.
So long as we provide binary yt packages of (at least) the stable releases, this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out of the box.
-Nathan
Awesome work, and I am getting more excited about this as time goes on. I am going to try my hand at updating the new script in the way you mentioned, and then I will try checking it into the repo and issue a PR so we can iterate there. I hope to get to this by tomorrow morning.
-Matt
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
Yup -- you can do it like this:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
> Hi all, > > Thanks to everybody who has reported back on testing. After some > talking both offline and on IRC, as well as here, I think we would > need to do the following things: > > * Make a single script that grabs the appropriate distribution of > miniconda and installs it. Right now I have a mechanism for doing > this, but it's currently tied to an architecture. > * Create a mechanism for installing all the packages we need. Nearly > all are available inside the Continuum repos. What we're hung up on > is that source installs require a "recipe", and transmitting the > recipe is where I don't have an idea of what to do. > * Test this out lots of places > * Clean up the edges in the (new) install script > * Move the old install script to maintenance mode > * Update all documentation to describe this and mothball other > methods of installation > > What would be nice: > > * Make available nightly builds of yt on several architectures using > binstar > * Utilize more of the packages included in conda elsewhere in yt, now > that we can! > > Here's my current recipe for get_yt.sh: > > http://paste.yt-project.org/show/3843/ > > (The config thing may get switched to include the --system argument, > to modify the "yt-conda" condarc.) The step that I'm most stuck on > is > getting the yt recipe to people. If we want to make it possible and > easy to build from source, we need to get the contents of a "conda > recipe" to people. They can then run "conda build ." in the > directory. Here are the recipes that we've been playing with: > > https://bitbucket.org/yt_analysis/yt_conda/src > > Basically, in get_yt.sh, to do from source instead of from binary we > need to insert a step at the end that downloads the recipes somehow > and then cd's into the right directory and builds them. The reason > this is tricky is that we often need to bootstrap ourselves; we can't > assume anything exists. We can download the .tar.bz2 of the current > tip of the repo, but it includes the hash in the directory name that > it extracts to. > > So I think what we need is a mechanism for getting the current state > of the repo, figuring out the name of the repo's directory, moves > into > it, and then builds. I believe that all/most of this becomes much, > much easier if hg gets included in Anaconda, which Nathan has > submitted a PR for. So hopefully that will be taken care of, but > until that time we can possibly figure something out. I'm not sure > that we have the resources to continually support binary nightly > builds in perpetuity for all the architectures that people run on, so > having source would be awesome. Plus, one of the biggest appeals of > how we distribute yt is that the source is included; I would very > much > not like to give this up. > > Thoughts? Has anyone else tested any of this out? > > -Matt > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > brittonsmith@gmail.com > wrote: >> Hi everyone, >> >> Sorry for chiming in late. I just moved when this thread began and >> do >> not >> have regular internet access. >> >> I really like this idea of conda, especially as a package manager >> that >> only >> optionally makes its own edits to your .bashrc. I have always >> really >> liked >> that the install script creates a clean python stack with basically >> everything a python user needs. I have on occasion suggested it to >> people >> just looking to use numpy and matploblib. It looks like conda will >> continue >> to provide this nice by-product, so I'm all for it. >> >> I won't be in a position to help with testing and such for another >> week >> or >> so when I get regular internet access, but I would be glad to do so >> then. >> >> Britton >> >> >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum >> nathan12343@gmail.com >> wrote: >>> >>> Everything should be available now for 64 bit linux and OS X. >>> >>> >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone >>> chris.m.malone@gmail.com >>> wrote: >>>> >>>> Hi Nathan, >>>> >>>> That appears to work as it built the environment and `conda >>>> install >>>> ...` >>>> added packages to my environment. >>>> >>>> One mistake I made was that I originally downloaded the "latest" >>>> OS X >>>> build of Miniconda, but that happened to be Miniconda3, which is >>>> python 3 >>>> based. Trying to build the environment with that yields an error >>>> regarding >>>> incompatibility of yt and python3.3, as it should. >>>> >>>> Chris >>>> >>>> >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >>>> nathan12343@gmail.com >>>> wrote: >>>>> >>>>> Hey Chris, >>>>> >>>>> I don't think mercurial is strictly necessary, can you try again >>>>> without >>>>> it? I think if Matt uploads a mercurial package for OS X this >>>>> won't >>>>> be an >>>>> issue. I'll send him an updated tarball. >>>>> >>>>> I submitted a mercurial recipe to conda-recipes yesterday >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>>>> hopefully >>>>> a >>>>> mercurial build will be included in future anaconda releases. >>>>> >>>>> >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>>>> chris.m.malone@gmail.com wrote: >>>>>> >>>>>> I just tried setting this up on OS X 10.7.5 and failed when >>>>>> attempting >>>>>> to create the conda environment due to a missing mercurial >>>>>> package: >>>>>> >>>>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project >>>>>> yt >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>>>> Error: No packages found matching: mercurial >>>>>> >>>>>> >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>>>> nathan12343@gmail.com wrote: >>>>>>> >>>>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>>>> binstar >>>>>>> is in >>>>>>> your .condarc, you should be able to get yt by doing 'conda >>>>>>> install yt'. >>>>>>> >>>>>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>>>>> about >>>>>>> issues, particularly if there are issues on older OS X >>>>>>> releases. >>>>>>> >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Thank you for the feedback -- I am glad there is some >>>>>>>> agreement >>>>>>>> about >>>>>>>> possible ways forward, and so I'm happy to try to use this as >>>>>>>> an >>>>>>>> opportunity to explore simpler, more reliable methods than the >>>>>>>> install >>>>>>>> script. >>>>>>>> >>>>>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>>>>> it's >>>>>>>> quite nice. There are a few rough corners, particularly >>>>>>>> related >>>>>>>> to >>>>>>>> the binstar service, but it's so far pretty great. With >>>>>>>> Nathan's >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>>>>> and >>>>>>>> then install it. >>>>>>>> >>>>>>>> The workflow that seems to work: >>>>>>>> >>>>>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>>>>>> * Run the installer for miniconda >>>>>>>> * Enter the conda environment and then install yt by doing >>>>>>>> "conda >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>>>> >>>>>>>> I think that this can likely all be stuck into a bash script. >>>>>>>> A >>>>>>>> simple, first pass at this is here: >>>>>>>> >>>>>>>> http://paste.yt-project.org/show/3833/ >>>>>>>> >>>>>>>> This right now only works on Linux x86_64, but getting it to >>>>>>>> work >>>>>>>> for >>>>>>>> other machines won't be too hard. I suspect we will be able >>>>>>>> to >>>>>>>> do >>>>>>>> nightlies very easily as well. If anyone out there has an >>>>>>>> x86_64 >>>>>>>> machine they wouldn't mind trying it on, that would be very >>>>>>>> helpful! >>>>>>>> I did find that once I ran this script, I had to actually >>>>>>>> prepend >>>>>>>> the >>>>>>>> PATH afterwards as well. This means doing: >>>>>>>> >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>>>> source activate ytenv >>>>>>>> >>>>>>>> At that point, everything was set up and working for me. The >>>>>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>>>>> think >>>>>>>> we >>>>>>>> should go down that route. That being said, this is also a >>>>>>>> possible >>>>>>>> point of friction. >>>>>>>> >>>>>>>> One nice thing is that this also completely works with the >>>>>>>> full >>>>>>>> anaconda; if someone wants everything that is in the anaconda >>>>>>>> install, >>>>>>>> they can even simply do "conda install anaconda" from the >>>>>>>> command >>>>>>>> line >>>>>>>> to get it. But the stripped down subset is the default. >>>>>>>> >>>>>>>> If anyone has a chance to try this out and has feedback, I'd >>>>>>>> greatly >>>>>>>> appreciate it! I think Nathan has done something very similar >>>>>>>> for >>>>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as >>>>>>>> a >>>>>>>> basis >>>>>>>> for distributing builds and setting them up on buildbots and >>>>>>>> the >>>>>>>> like. >>>>>>>> I'm pretty optimistic about this. >>>>>>>> >>>>>>>> -Matt >>>>>>>> >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>>>> nathan12343@gmail.com wrote: >>>>>>>>> I think to get everything working in a sustainable fashion, >>>>>>>>> we >>>>>>>>> would need >>>>>>>>> buildbots for all platform combinations that we want to >>>>>>>>> support, >>>>>>>>> so >>>>>>>>> all >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>>>> py27/py3.3) tuple. >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit linux, >>>>>>>>> 64 >>>>>>>>> bit >>>>>>>>> OS X >>>>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>>>> bit >>>>>>>>> windows. >>>>>>>>> >>>>>>>>> Another option is to rely on conda build, which compiles >>>>>>>>> everything >>>>>>>>> from >>>>>>>>> source. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have less of a skin in this than I used to, but I'd like >>>>>>>>>> to >>>>>>>>>> raise >>>>>>>>>> the issue of Windows & package managers. For example, >>>>>>>>>> Anaconda >>>>>>>>>> is >>>>>>>>>> available for Windows - would that mean that yt might "just >>>>>>>>>> work" >>>>>>>>>> on >>>>>>>>>> Windows? Or the opposite, and it would require a great deal >>>>>>>>>> of >>>>>>>>>> effort >>>>>>>>>> to get all the various things we expect to be .so's to work >>>>>>>>>> as >>>>>>>>>> .dll's >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>>>> >>>>>>>>>> I don't know the answers to these questions, but I think >>>>>>>>>> it's >>>>>>>>>> worth >>>>>>>>>> thinking about. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Stephen Skory >>>>>>>>>> s@skory.us >>>>>>>>>> http://stephenskory.com/ >>>>>>>>>> 510.621.3687 (google voice) >>>>>>>>>> _______________________________________________ >>>>>>>>>> yt-dev mailing list >>>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> yt-dev mailing list >>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hmm, that's a good point. chrpath is pretty minimal - we could always build it ourselves and put it in $YT_CONDA/bin if it isn't already available.
On Tue, Sep 10, 2013 at 9:33 AM, John ZuHone jzuhone@gmail.com wrote:
I tried installing this script on Pleiades but had the same problem as Matt, with the requirement of the chrpath package. Of course, on Pleiades we don't have the option of installing it ourselves. Thoughts, besides asking them to install it? I guess that might be the only option.
On Sep 7, 2013, at 1:27 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
The yt-dev recipe I created would only be good for nightly binary builds. I think that would only be useful for people who want the latest and greatest but don't want to set up a build environment, which I suspect is a relatively small group of people. We'll have to decide at some other point if we want to have buildbots and nightly builds.
Agreed about updating our documentation and website. I think we should supply both the script and instructions for setting up yt using `pip` and already-installed conda packages. We should also link to a page that lists the packages needed to manually set up yt.
On Sat, Sep 7, 2013 at 8:22 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
I've issued a PR with my first pass at the get_yt.sh script. I've ported over a bunch of functionality from the old install_script.sh and it works for me in a clean install on Ubuntu, although it does require the "chrpath" package which I oddly did not have.
https://bitbucket.org/yt_analysis/yt/pull-request/591/first-draft-of-a-new-g...
What would be really helpful is if a few people could try it out, make modifications, commit, push to their fork, and I will then pull in those changes until it does everything it needs to. I think what I would like to preserve is that the "get_yt.sh" script should *not* be about how to install Conda/Anaconda -- it should get yt installed, and the yt environment. The case of installing yt *into* Conda is what I'd like to cover by providing binary packages, and I suspect that our audience for that will grow with time. This script aims along a slightly different vector.
I'm still not entirely sure how to handle source distributions; even with the yt-dev recipe Nathan has, it's still building binary packages. I've done my best to include the source as-is in the script, but to also use the yt recipe that Nathan has worked on. I'm not sure how the "yt-dev" recipe fits into that. Kacper, Nathan, any ideas about that? Would it just be the basic nightly package builder? Then we could have it get rebuild and reuploaded all the time; this would help with keeping people up to date, but *not* with encouraging contributions.
-Matt
PS One last thing is that our whole "get yt" documentation should be re-done. We should be much more clear about how and where to get the package, and reduce the number of options.
On Fri, Sep 6, 2013 at 9:08 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Wed, Sep 4, 2013 at 2:54 AM, Nathan Goldbaum nathan12343@gmail.com
wrote:
Here's an updated version of the shell script Matt sent out earlier
today
that builds yt and then installs it based on the yt-conda recipe: http://paste.yt-project.org/show/3848/
I like this a lot and think it can be improved in several simple ways,
most
of which can be readily scraped from the install script:
Me too. This is much improved over mine.
* Install in the folder that the script was invoked in rather than$HOME
Good idea.
* Add an option to install anaconda with a warning about needing acouple hundred megabytes of space
Excellent! I like this a lot. And, we should point to the Anaconda page for more info about packages and how Anaconda works.
* Install the miniconda package appropriate for the platform * Add warnings on various platforms regarding needed packages andcompilers (i.e. still need build-essentials on ubuntu and Xcode on
OSX).
* Add a warning at the end about modifying .bashrc for PATH andLD_LIBRARY_PATH or supply an activate script that sets them for us.
I think I'd prefer to supply an activate script that both activates Conda *and* the yt environment in it. That way individuals who want to use Conda directly can do so, but we also make it possible to keep the behavior that people have already.
* Clone yt-doc, yt-supplemental and the main yt repo and put themin the
yt-conda directory.
Agreed.
* Check if conda is already available and (optionally?) skip theminiconda step.
I'm not sure about this, since I think we will then just tell people to install their own yt. I think the install script should not be *too* complex in this regard.
* Ensure the user that yt loves them.Yes.
As Matt suggested on a Conda github issue (https://github.com/ContinuumIO/conda/issues/176#issuecomment-23753101)
it
would be nice to first try to conda install yt and then if that fails
use
conda build. I think it should be possible to set this up as part of
the
bash script. As asmurer notes in the issue, general support for this
sort
of mixed binary/source distribution in conda might be difficult but
with yt
it works since our setup script is platform agnostic so long as the compilation environment is set up correctly and we don't even try to
work
correctly on windows.
So long as we provide binary yt packages of (at least) the stable
releases,
this should obviate the need for proper compiler setup, which avoid a significant source of pain for OS X users as well as linux users on platforms like ubuntu that don't provide sane build environments out
of the
box.
-Nathan
Awesome work, and I am getting more excited about this as time goes on. I am going to try my hand at updating the new script in the way you mentioned, and then I will try checking it into the repo and issue a PR so we can iterate there. I hope to get to this by tomorrow morning.
-Matt
On Tue, Sep 3, 2013 at 12:28 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hi Nathan,
On Tue, Sep 3, 2013 at 3:10 PM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo,
I
think because we use named branches in that repository.
Yup -- you can do it like this:
wget
https://bitbucket.org/yt_analysis/yt_conda/raw/default/yt/meta.yaml
...although for me right now it's failing from a bad BB certificate.
The only problem with "pip install" is that it no longer lists pip installations inside the "conda list" or "conda update" outputs. This has been discussed recently on the mailing list for Anaconda:
https://groups.google.com/a/continuum.io/forum/?fromgroups#%21topic/anaconda...
...but there has not yet been a resolution.
-Matt
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com
wrote:
> > Works fine for me on OS X x86_64. > > On my Goddard-controlled Linux x86_64 server, everything worked
fine
> except Mercurial, which I "conda install"-ed using the yt link,
but the
> binary had an incompatibility which the glibc that was installed
on the
> machine. Using pip to install Mercurial (which did it from source)
was
> the > workaround. > > On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com
wrote:
> > > Hi all, > > > > Thanks to everybody who has reported back on testing. After some > > talking both offline and on IRC, as well as here, I think we
would
> > need to do the following things: > > > > * Make a single script that grabs the appropriate distribution of > > miniconda and installs it. Right now I have a mechanism for
doing
> > this, but it's currently tied to an architecture. > > * Create a mechanism for installing all the packages we need.
Nearly
> > all are available inside the Continuum repos. What we're hung
up on
> > is that source installs require a "recipe", and transmitting the > > recipe is where I don't have an idea of what to do. > > * Test this out lots of places > > * Clean up the edges in the (new) install script > > * Move the old install script to maintenance mode > > * Update all documentation to describe this and mothball other > > methods of installation > > > > What would be nice: > > > > * Make available nightly builds of yt on several architectures
using
> > binstar > > * Utilize more of the packages included in conda elsewhere in
yt, now
> > that we can! > > > > Here's my current recipe for get_yt.sh: > > > > http://paste.yt-project.org/show/3843/ > > > > (The config thing may get switched to include the --system
argument,
> > to modify the "yt-conda" condarc.) The step that I'm most stuck
on
> > is > > getting the yt recipe to people. If we want to make it possible
and
> > easy to build from source, we need to get the contents of a
"conda
> > recipe" to people. They can then run "conda build ." in the > > directory. Here are the recipes that we've been playing with: > > > > https://bitbucket.org/yt_analysis/yt_conda/src > > > > Basically, in get_yt.sh, to do from source instead of from
binary we
> > need to insert a step at the end that downloads the recipes
somehow
> > and then cd's into the right directory and builds them. The
reason
> > this is tricky is that we often need to bootstrap ourselves; we
can't
> > assume anything exists. We can download the .tar.bz2 of the
current
> > tip of the repo, but it includes the hash in the directory name
that
> > it extracts to. > > > > So I think what we need is a mechanism for getting the current
state
> > of the repo, figuring out the name of the repo's directory, moves > > into > > it, and then builds. I believe that all/most of this becomes
much,
> > much easier if hg gets included in Anaconda, which Nathan has > > submitted a PR for. So hopefully that will be taken care of, but > > until that time we can possibly figure something out. I'm not
sure
> > that we have the resources to continually support binary nightly > > builds in perpetuity for all the architectures that people run
on, so
> > having source would be awesome. Plus, one of the biggest
appeals of
> > how we distribute yt is that the source is included; I would very > > much > > not like to give this up. > > > > Thoughts? Has anyone else tested any of this out? > > > > -Matt > > > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > > brittonsmith@gmail.com > > wrote: > >> Hi everyone, > >> > >> Sorry for chiming in late. I just moved when this thread began
and
> >> do > >> not > >> have regular internet access. > >> > >> I really like this idea of conda, especially as a package
manager
> >> that > >> only > >> optionally makes its own edits to your .bashrc. I have always > >> really > >> liked > >> that the install script creates a clean python stack with
basically
> >> everything a python user needs. I have on occasion suggested
it to
> >> people > >> just looking to use numpy and matploblib. It looks like conda
will
> >> continue > >> to provide this nice by-product, so I'm all for it. > >> > >> I won't be in a position to help with testing and such for
another
> >> week > >> or > >> so when I get regular internet access, but I would be glad to
do so
> >> then. > >> > >> Britton > >> > >> > >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum > >> nathan12343@gmail.com > >> wrote: > >>> > >>> Everything should be available now for 64 bit linux and OS X. > >>> > >>> > >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone > >>> chris.m.malone@gmail.com > >>> wrote: > >>>> > >>>> Hi Nathan, > >>>> > >>>> That appears to work as it built the environment and `conda > >>>> install > >>>> ...` > >>>> added packages to my environment. > >>>> > >>>> One mistake I made was that I originally downloaded the
"latest"
> >>>> OS X > >>>> build of Miniconda, but that happened to be Miniconda3, which
is
> >>>> python 3 > >>>> based. Trying to build the environment with that yields an
error
> >>>> regarding > >>>> incompatibility of yt and python3.3, as it should. > >>>> > >>>> Chris > >>>> > >>>> > >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum > >>>> nathan12343@gmail.com > >>>> wrote: > >>>>> > >>>>> Hey Chris, > >>>>> > >>>>> I don't think mercurial is strictly necessary, can you try
again
> >>>>> without > >>>>> it? I think if Matt uploads a mercurial package for OS X
this
> >>>>> won't > >>>>> be an > >>>>> issue. I'll send him an updated tarball. > >>>>> > >>>>> I submitted a mercurial recipe to conda-recipes yesterday > >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so > >>>>> hopefully > >>>>> a > >>>>> mercurial build will be included in future anaconda releases. > >>>>> > >>>>> > >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > >>>>> chris.m.malone@gmail.com wrote: > >>>>>> > >>>>>> I just tried setting this up on OS X 10.7.5 and failed when > >>>>>> attempting > >>>>>> to create the conda environment due to a missing mercurial > >>>>>> package: > >>>>>> > >>>>>> $ conda create -n ytenv -c
http://conda.binstar.org/yt_project
> >>>>>> yt > >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx > >>>>>> Error: No packages found matching: mercurial > >>>>>> > >>>>>> > >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum > >>>>>> nathan12343@gmail.com wrote: > >>>>>>> > >>>>>>> Yup, please try on OSX as well. If you make sure Matt's > >>>>>>> binstar > >>>>>>> is in > >>>>>>> your .condarc, you should be able to get yt by doing 'conda > >>>>>>> install yt'. > >>>>>>> > >>>>>>> I built the OSX binary on my laptop so I'd appreciate
hearing
> >>>>>>> about > >>>>>>> issues, particularly if there are issues on older OS X > >>>>>>> releases. > >>>>>>> > >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: > >>>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> Thank you for the feedback -- I am glad there is some > >>>>>>>> agreement > >>>>>>>> about > >>>>>>>> possible ways forward, and so I'm happy to try to use
this as
> >>>>>>>> an > >>>>>>>> opportunity to explore simpler, more reliable methods
than the
> >>>>>>>> install > >>>>>>>> script. > >>>>>>>> > >>>>>>>> This afternoon, I spent a bit of time with Conda, and I
think
> >>>>>>>> it's > >>>>>>>> quite nice. There are a few rough corners, particularly > >>>>>>>> related > >>>>>>>> to > >>>>>>>> the binstar service, but it's so far pretty great. With > >>>>>>>> Nathan's > >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux
x86_64
> >>>>>>>> and > >>>>>>>> then install it. > >>>>>>>> > >>>>>>>> The workflow that seems to work: > >>>>>>>> > >>>>>>>> * Get miniconda:
http://repo.continuum.io/miniconda/index.html
> >>>>>>>> * Run the installer for miniconda > >>>>>>>> * Enter the conda environment and then install yt by doing > >>>>>>>> "conda > >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". > >>>>>>>> > >>>>>>>> I think that this can likely all be stuck into a bash
script.
> >>>>>>>> A > >>>>>>>> simple, first pass at this is here: > >>>>>>>> > >>>>>>>> http://paste.yt-project.org/show/3833/ > >>>>>>>> > >>>>>>>> This right now only works on Linux x86_64, but getting it
to
> >>>>>>>> work > >>>>>>>> for > >>>>>>>> other machines won't be too hard. I suspect we will be
able
> >>>>>>>> to > >>>>>>>> do > >>>>>>>> nightlies very easily as well. If anyone out there has an > >>>>>>>> x86_64 > >>>>>>>> machine they wouldn't mind trying it on, that would be
very
> >>>>>>>> helpful! > >>>>>>>> I did find that once I ran this script, I had to actually > >>>>>>>> prepend > >>>>>>>> the > >>>>>>>> PATH afterwards as well. This means doing: > >>>>>>>> > >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH > >>>>>>>> export PATH=$HOME/yt-conda:$PATH > >>>>>>>> source activate ytenv > >>>>>>>> > >>>>>>>> At that point, everything was set up and working for me.
The
> >>>>>>>> miniconda install offers to add paths to .bashrc, but I
don't
> >>>>>>>> think > >>>>>>>> we > >>>>>>>> should go down that route. That being said, this is also
a
> >>>>>>>> possible > >>>>>>>> point of friction. > >>>>>>>> > >>>>>>>> One nice thing is that this also completely works with the > >>>>>>>> full > >>>>>>>> anaconda; if someone wants everything that is in the
anaconda
> >>>>>>>> install, > >>>>>>>> they can even simply do "conda install anaconda" from the > >>>>>>>> command > >>>>>>>> line > >>>>>>>> to get it. But the stripped down subset is the default. > >>>>>>>> > >>>>>>>> If anyone has a chance to try this out and has feedback,
I'd
> >>>>>>>> greatly > >>>>>>>> appreciate it! I think Nathan has done something very
similar
> >>>>>>>> for > >>>>>>>> OSX. I've also put a couple simple conda recipes here: > >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can
use as
> >>>>>>>> a > >>>>>>>> basis > >>>>>>>> for distributing builds and setting them up on buildbots
and
> >>>>>>>> the > >>>>>>>> like. > >>>>>>>> I'm pretty optimistic about this. > >>>>>>>> > >>>>>>>> -Matt > >>>>>>>> > >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum > >>>>>>>> nathan12343@gmail.com wrote: > >>>>>>>>> I think to get everything working in a sustainable
fashion,
> >>>>>>>>> we > >>>>>>>>> would need > >>>>>>>>> buildbots for all platform combinations that we want to > >>>>>>>>> support, > >>>>>>>>> so > >>>>>>>>> all > >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, > >>>>>>>>> py27/py3.3) tuple. > >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit
linux,
> >>>>>>>>> 64 > >>>>>>>>> bit > >>>>>>>>> OS X > >>>>>>>>> (not totally clear if OS X version matters), and 32 and
64
> >>>>>>>>> bit > >>>>>>>>> windows. > >>>>>>>>> > >>>>>>>>> Another option is to rely on conda build, which compiles > >>>>>>>>> everything > >>>>>>>>> from > >>>>>>>>> source. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <
s@skory.us>
> >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I have less of a skin in this than I used to, but I'd
like
> >>>>>>>>>> to > >>>>>>>>>> raise > >>>>>>>>>> the issue of Windows & package managers. For example, > >>>>>>>>>> Anaconda > >>>>>>>>>> is > >>>>>>>>>> available for Windows - would that mean that yt might
"just
> >>>>>>>>>> work" > >>>>>>>>>> on > >>>>>>>>>> Windows? Or the opposite, and it would require a great
deal
> >>>>>>>>>> of > >>>>>>>>>> effort > >>>>>>>>>> to get all the various things we expect to be .so's to
work
> >>>>>>>>>> as > >>>>>>>>>> .dll's > >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? > >>>>>>>>>> > >>>>>>>>>> I don't know the answers to these questions, but I think > >>>>>>>>>> it's > >>>>>>>>>> worth > >>>>>>>>>> thinking about. > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Stephen Skory > >>>>>>>>>> s@skory.us > >>>>>>>>>> http://stephenskory.com/ > >>>>>>>>>> 510.621.3687 (google voice) > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> yt-dev mailing list > >>>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> yt-dev mailing list > >>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> yt-dev mailing list > >>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> yt-dev mailing list > >>>>>>> yt-dev@lists.spacepope.org > >>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> yt-dev mailing list > >>>>>> yt-dev@lists.spacepope.org > >>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> yt-dev mailing list > >>>>> yt-dev@lists.spacepope.org > >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-dev mailing list > >>>> yt-dev@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>> > >>> > >>> > >>> _______________________________________________ > >>> yt-dev mailing list > >>> yt-dev@lists.spacepope.org > >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>> > >> > >> > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >> > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py", line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py", line 31, in <module> from grid_patch import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py", line 35, in <module> from yt.data_objects.data_containers import YTFieldData File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py", line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py", line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 39, in <module> from yt.utilities.lib import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py", line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote: Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey Chris,
I don't think mercurial is strictly necessary, can you try again without it? I think if Matt uploads a mercurial package for OS X this won't be an issue. I'll send him an updated tarball.
I submitted a mercurial recipe to conda-recipes yesterday (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a mercurial build will be included in future anaconda releases.
On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone chris.m.malone@gmail.com wrote: > > I just tried setting this up on OS X 10.7.5 and failed when attempting > to create the conda environment due to a missing mercurial package: > > $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt > mercurial ipython tornado pyzmq pygments jinja2 sphinx > Error: No packages found matching: mercurial > > > On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum > nathan12343@gmail.com wrote: >> >> Yup, please try on OSX as well. If you make sure Matt's binstar is in >> your .condarc, you should be able to get yt by doing 'conda install yt'. >> >> I built the OSX binary on my laptop so I'd appreciate hearing about >> issues, particularly if there are issues on older OS X releases. >> >> On Thursday, August 29, 2013, Matthew Turk wrote: >>> >>> Hi all, >>> >>> Thank you for the feedback -- I am glad there is some agreement about >>> possible ways forward, and so I'm happy to try to use this as an >>> opportunity to explore simpler, more reliable methods than the >>> install >>> script. >>> >>> This afternoon, I spent a bit of time with Conda, and I think it's >>> quite nice. There are a few rough corners, particularly related to >>> the binstar service, but it's so far pretty great. With Nathan's >>> help, I was able to upload a yt-2.5.5 package for linux x86_64 and >>> then install it. >>> >>> The workflow that seems to work: >>> >>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>> * Run the installer for miniconda >>> * Enter the conda environment and then install yt by doing "conda >>> install yt -c http://conda.binstar.org/yt_project/ ". >>> >>> I think that this can likely all be stuck into a bash script. A >>> simple, first pass at this is here: >>> >>> http://paste.yt-project.org/show/3833/ >>> >>> This right now only works on Linux x86_64, but getting it to work for >>> other machines won't be too hard. I suspect we will be able to do >>> nightlies very easily as well. If anyone out there has an x86_64 >>> machine they wouldn't mind trying it on, that would be very helpful! >>> I did find that once I ran this script, I had to actually prepend the >>> PATH afterwards as well. This means doing: >>> >>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>> export PATH=$HOME/yt-conda:$PATH >>> source activate ytenv >>> >>> At that point, everything was set up and working for me. The >>> miniconda install offers to add paths to .bashrc, but I don't think >>> we >>> should go down that route. That being said, this is also a possible >>> point of friction. >>> >>> One nice thing is that this also completely works with the full >>> anaconda; if someone wants everything that is in the anaconda >>> install, >>> they can even simply do "conda install anaconda" from the command >>> line >>> to get it. But the stripped down subset is the default. >>> >>> If anyone has a chance to try this out and has feedback, I'd greatly >>> appreciate it! I think Nathan has done something very similar for >>> OSX. I've also put a couple simple conda recipes here: >>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>> basis >>> for distributing builds and setting them up on buildbots and the >>> like. >>> I'm pretty optimistic about this. >>> >>> -Matt >>> >>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>> nathan12343@gmail.com wrote: >>>> I think to get everything working in a sustainable fashion, we >>>> would need >>>> buildbots for all platform combinations that we want to support, so >>>> all >>>> permutations of the (32/64 bit, linux / OS X / windows, >>>> py27/py3.3) tuple. >>>> At the moment anaconda seems to support 32 and 64 bit linux, 64 bit >>>> OS X >>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>> windows. >>>> >>>> Another option is to rely on conda build, which compiles everything >>>> from >>>> source. >>>> >>>> >>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I have less of a skin in this than I used to, but I'd like to >>>>> raise >>>>> the issue of Windows & package managers. For example, Anaconda is >>>>> available for Windows - would that mean that yt might "just work" >>>>> on >>>>> Windows? Or the opposite, and it would require a great deal of >>>>> effort >>>>> to get all the various things we expect to be .so's to work as >>>>> .dll's >>>>> (such as the Cython helpers or halo-finding stuff)? >>>>> >>>>> I don't know the answers to these questions, but I think it's >>>>> worth >>>>> thinking about. >>>>> >>>>> -- >>>>> Stephen Skory >>>>> s@skory.us >>>>> http://stephenskory.com/ >>>>> 510.621.3687 (google voice) >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
On 10.09.2013 18:54, John ZuHone wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py", line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py", line 31, in <module> from grid_patch import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py", line 35, in <module> from yt.data_objects.data_containers import YTFieldData File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py", line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py", line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 39, in <module> from yt.utilities.lib import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py", line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
Hi John, to allow for binary compatibility on generic linux boxes we would need to provide our own packages build around conda on something old. SL6 (or SL5 even) seems to be a good choice.
For now I think manually building chrpath if necessary and going with script from PR 591 should be the preferred way of installing yt in user restricted environment.
BTW, We could add building chrpath to the script as it has no other dependencies. Cheers, Kacper
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote: Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install ...` added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is python 3 based. Trying to build the environment with that yields an error regarding incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum nathan12343@gmail.com wrote: > > Hey Chris, > > I don't think mercurial is strictly necessary, can you try again without > it? I think if Matt uploads a mercurial package for OS X this won't be an > issue. I'll send him an updated tarball. > > I submitted a mercurial recipe to conda-recipes yesterday > (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully a > mercurial build will be included in future anaconda releases. > > > On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > chris.m.malone@gmail.com wrote: >> >> I just tried setting this up on OS X 10.7.5 and failed when attempting >> to create the conda environment due to a missing mercurial package: >> >> $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt >> mercurial ipython tornado pyzmq pygments jinja2 sphinx >> Error: No packages found matching: mercurial >> >> >> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >> nathan12343@gmail.com wrote: >>> >>> Yup, please try on OSX as well. If you make sure Matt's binstar is in >>> your .condarc, you should be able to get yt by doing 'conda install yt'. >>> >>> I built the OSX binary on my laptop so I'd appreciate hearing about >>> issues, particularly if there are issues on older OS X releases. >>> >>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>> >>>> Hi all, >>>> >>>> Thank you for the feedback -- I am glad there is some agreement about >>>> possible ways forward, and so I'm happy to try to use this as an >>>> opportunity to explore simpler, more reliable methods than the >>>> install >>>> script. >>>> >>>> This afternoon, I spent a bit of time with Conda, and I think it's >>>> quite nice. There are a few rough corners, particularly related to >>>> the binstar service, but it's so far pretty great. With Nathan's >>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 and >>>> then install it. >>>> >>>> The workflow that seems to work: >>>> >>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>> * Run the installer for miniconda >>>> * Enter the conda environment and then install yt by doing "conda >>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>> >>>> I think that this can likely all be stuck into a bash script. A >>>> simple, first pass at this is here: >>>> >>>> http://paste.yt-project.org/show/3833/ >>>> >>>> This right now only works on Linux x86_64, but getting it to work for >>>> other machines won't be too hard. I suspect we will be able to do >>>> nightlies very easily as well. If anyone out there has an x86_64 >>>> machine they wouldn't mind trying it on, that would be very helpful! >>>> I did find that once I ran this script, I had to actually prepend the >>>> PATH afterwards as well. This means doing: >>>> >>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>> export PATH=$HOME/yt-conda:$PATH >>>> source activate ytenv >>>> >>>> At that point, everything was set up and working for me. The >>>> miniconda install offers to add paths to .bashrc, but I don't think >>>> we >>>> should go down that route. That being said, this is also a possible >>>> point of friction. >>>> >>>> One nice thing is that this also completely works with the full >>>> anaconda; if someone wants everything that is in the anaconda >>>> install, >>>> they can even simply do "conda install anaconda" from the command >>>> line >>>> to get it. But the stripped down subset is the default. >>>> >>>> If anyone has a chance to try this out and has feedback, I'd greatly >>>> appreciate it! I think Nathan has done something very similar for >>>> OSX. I've also put a couple simple conda recipes here: >>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>>> basis >>>> for distributing builds and setting them up on buildbots and the >>>> like. >>>> I'm pretty optimistic about this. >>>> >>>> -Matt >>>> >>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>> nathan12343@gmail.com wrote: >>>>> I think to get everything working in a sustainable fashion, we >>>>> would need >>>>> buildbots for all platform combinations that we want to support, so >>>>> all >>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>> py27/py3.3) tuple. >>>>> At the moment anaconda seems to support 32 and 64 bit linux, 64 bit >>>>> OS X >>>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>>> windows. >>>>> >>>>> Another option is to rely on conda build, which compiles everything >>>>> from >>>>> source. >>>>> >>>>> >>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I have less of a skin in this than I used to, but I'd like to >>>>>> raise >>>>>> the issue of Windows & package managers. For example, Anaconda is >>>>>> available for Windows - would that mean that yt might "just work" >>>>>> on >>>>>> Windows? Or the opposite, and it would require a great deal of >>>>>> effort >>>>>> to get all the various things we expect to be .so's to work as >>>>>> .dll's >>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>> >>>>>> I don't know the answers to these questions, but I think it's >>>>>> worth >>>>>> thinking about. >>>>>> >>>>>> -- >>>>>> Stephen Skory >>>>>> s@skory.us >>>>>> http://stephenskory.com/ >>>>>> 510.621.3687 (google voice) >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey John,
A few questions:
1. What did you choose for INST_YT_SOURCE in get_yt.sh? If INST_YT_SOURCE=1, what happens when you try INST_YT_SOURCE=0?
2. Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries it tries to link against. This should be the very first thing printed out by the setup script. Is it linking against anaconda's libraries or system libraries?
3. What's the output of `ldd /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort of gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against anaconda's hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I run into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py", line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py", line 31, in <module> from grid_patch import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py", line 35, in <module> from yt.data_objects.data_containers import YTFieldData File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py", line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py", line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 39, in <module> from yt.utilities.lib import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py", line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com
wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do
not
have regular internet access.
I really like this idea of conda, especially as a package manager that
only
optionally makes its own edits to your .bashrc. I have always really
liked
that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to
people
just looking to use numpy and matploblib. It looks like conda will
continue
to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another
week or
so when I get regular internet access, but I would be glad to do so
then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone <
chris.m.malone@gmail.com>
wrote:
Hi Nathan,
That appears to work as it built the environment and `conda install
...`
added packages to my environment.
One mistake I made was that I originally downloaded the "latest" OS X build of Miniconda, but that happened to be Miniconda3, which is
python 3
based. Trying to build the environment with that yields an error
regarding
incompatibility of yt and python3.3, as it should.
Chris
On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote: > > Hey Chris, > > I don't think mercurial is strictly necessary, can you try again
without
> it? I think if Matt uploads a mercurial package for OS X this
won't be an
> issue. I'll send him an updated tarball. > > I submitted a mercurial recipe to conda-recipes yesterday > (https://github.com/ContinuumIO/conda-recipes/pull/14) so
hopefully a
> mercurial build will be included in future anaconda releases. > > > On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > chris.m.malone@gmail.com wrote: >> >> I just tried setting this up on OS X 10.7.5 and failed when
attempting
>> to create the conda environment due to a missing mercurial package: >> >> $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt >> mercurial ipython tornado pyzmq pygments jinja2 sphinx >> Error: No packages found matching: mercurial >> >> >> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >> nathan12343@gmail.com wrote: >>> >>> Yup, please try on OSX as well. If you make sure Matt's binstar
is in
>>> your .condarc, you should be able to get yt by doing 'conda
install yt'.
>>> >>> I built the OSX binary on my laptop so I'd appreciate hearing
about
>>> issues, particularly if there are issues on older OS X releases. >>> >>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>> >>>> Hi all, >>>> >>>> Thank you for the feedback -- I am glad there is some agreement
about
>>>> possible ways forward, and so I'm happy to try to use this as an >>>> opportunity to explore simpler, more reliable methods than the >>>> install >>>> script. >>>> >>>> This afternoon, I spent a bit of time with Conda, and I think
it's
>>>> quite nice. There are a few rough corners, particularly related
to
>>>> the binstar service, but it's so far pretty great. With Nathan's >>>> help, I was able to upload a yt-2.5.5 package for linux x86_64
and
>>>> then install it. >>>> >>>> The workflow that seems to work: >>>> >>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>> * Run the installer for miniconda >>>> * Enter the conda environment and then install yt by doing "conda >>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>> >>>> I think that this can likely all be stuck into a bash script. A >>>> simple, first pass at this is here: >>>> >>>> http://paste.yt-project.org/show/3833/ >>>> >>>> This right now only works on Linux x86_64, but getting it to
work for
>>>> other machines won't be too hard. I suspect we will be able to
do
>>>> nightlies very easily as well. If anyone out there has an x86_64 >>>> machine they wouldn't mind trying it on, that would be very
helpful!
>>>> I did find that once I ran this script, I had to actually
prepend the
>>>> PATH afterwards as well. This means doing: >>>> >>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>> export PATH=$HOME/yt-conda:$PATH >>>> source activate ytenv >>>> >>>> At that point, everything was set up and working for me. The >>>> miniconda install offers to add paths to .bashrc, but I don't
think
>>>> we >>>> should go down that route. That being said, this is also a
possible
>>>> point of friction. >>>> >>>> One nice thing is that this also completely works with the full >>>> anaconda; if someone wants everything that is in the anaconda >>>> install, >>>> they can even simply do "conda install anaconda" from the command >>>> line >>>> to get it. But the stripped down subset is the default. >>>> >>>> If anyone has a chance to try this out and has feedback, I'd
greatly
>>>> appreciate it! I think Nathan has done something very similar
for
>>>> OSX. I've also put a couple simple conda recipes here: >>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>>> basis >>>> for distributing builds and setting them up on buildbots and the >>>> like. >>>> I'm pretty optimistic about this. >>>> >>>> -Matt >>>> >>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>> nathan12343@gmail.com wrote: >>>>> I think to get everything working in a sustainable fashion, we >>>>> would need >>>>> buildbots for all platform combinations that we want to
support, so
>>>>> all >>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>> py27/py3.3) tuple. >>>>> At the moment anaconda seems to support 32 and 64 bit linux, 64
bit
>>>>> OS X >>>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>>> windows. >>>>> >>>>> Another option is to rely on conda build, which compiles
everything
>>>>> from >>>>> source. >>>>> >>>>> >>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us
wrote:
>>>>>> >>>>>> Hi all, >>>>>> >>>>>> I have less of a skin in this than I used to, but I'd like to >>>>>> raise >>>>>> the issue of Windows & package managers. For example, Anaconda
is
>>>>>> available for Windows - would that mean that yt might "just
work"
>>>>>> on >>>>>> Windows? Or the opposite, and it would require a great deal of >>>>>> effort >>>>>> to get all the various things we expect to be .so's to work as >>>>>> .dll's >>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>> >>>>>> I don't know the answers to these questions, but I think it's >>>>>> worth >>>>>> thinking about. >>>>>> >>>>>> -- >>>>>> Stephen Skory >>>>>> s@skory.us >>>>>> http://stephenskory.com/ >>>>>> 510.621.3687 (google voice) >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey John,
A few questions:
- What did you choose for INST_YT_SOURCE in get_yt.sh? If INST_YT_SOURCE=1,
what happens when you try INST_YT_SOURCE=0?
- Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py
install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries it tries to link against. This should be the very first thing printed out by the setup script. Is it linking against anaconda's libraries or system libraries?
- What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort of gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against anaconda's hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I run into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py", line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py", line 31, in <module> from grid_patch import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py", line 35, in <module> from yt.data_objects.data_containers import YTFieldData File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py", line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py", line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 39, in <module> from yt.utilities.lib import \ File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py", line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and do not have regular internet access.
I really like this idea of conda, especially as a package manager that only optionally makes its own edits to your .bashrc. I have always really liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Everything should be available now for 64 bit linux and OS X.
On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone chris.m.malone@gmail.com wrote: > > Hi Nathan, > > That appears to work as it built the environment and `conda install > ...` > added packages to my environment. > > One mistake I made was that I originally downloaded the "latest" OS > X > build of Miniconda, but that happened to be Miniconda3, which is > python 3 > based. Trying to build the environment with that yields an error > regarding > incompatibility of yt and python3.3, as it should. > > Chris > > > On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum > nathan12343@gmail.com > wrote: >> >> Hey Chris, >> >> I don't think mercurial is strictly necessary, can you try again >> without >> it? I think if Matt uploads a mercurial package for OS X this >> won't be an >> issue. I'll send him an updated tarball. >> >> I submitted a mercurial recipe to conda-recipes yesterday >> (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully >> a >> mercurial build will be included in future anaconda releases. >> >> >> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >> chris.m.malone@gmail.com wrote: >>> >>> I just tried setting this up on OS X 10.7.5 and failed when >>> attempting >>> to create the conda environment due to a missing mercurial >>> package: >>> >>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt >>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>> Error: No packages found matching: mercurial >>> >>> >>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>> nathan12343@gmail.com wrote: >>>> >>>> Yup, please try on OSX as well. If you make sure Matt's binstar >>>> is in >>>> your .condarc, you should be able to get yt by doing 'conda >>>> install yt'. >>>> >>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>> about >>>> issues, particularly if there are issues on older OS X releases. >>>> >>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>> >>>>> Hi all, >>>>> >>>>> Thank you for the feedback -- I am glad there is some agreement >>>>> about >>>>> possible ways forward, and so I'm happy to try to use this as an >>>>> opportunity to explore simpler, more reliable methods than the >>>>> install >>>>> script. >>>>> >>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>> it's >>>>> quite nice. There are a few rough corners, particularly related >>>>> to >>>>> the binstar service, but it's so far pretty great. With >>>>> Nathan's >>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>> and >>>>> then install it. >>>>> >>>>> The workflow that seems to work: >>>>> >>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html >>>>> * Run the installer for miniconda >>>>> * Enter the conda environment and then install yt by doing >>>>> "conda >>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>> >>>>> I think that this can likely all be stuck into a bash script. A >>>>> simple, first pass at this is here: >>>>> >>>>> http://paste.yt-project.org/show/3833/ >>>>> >>>>> This right now only works on Linux x86_64, but getting it to >>>>> work for >>>>> other machines won't be too hard. I suspect we will be able to >>>>> do >>>>> nightlies very easily as well. If anyone out there has an >>>>> x86_64 >>>>> machine they wouldn't mind trying it on, that would be very >>>>> helpful! >>>>> I did find that once I ran this script, I had to actually >>>>> prepend the >>>>> PATH afterwards as well. This means doing: >>>>> >>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>> export PATH=$HOME/yt-conda:$PATH >>>>> source activate ytenv >>>>> >>>>> At that point, everything was set up and working for me. The >>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>> think >>>>> we >>>>> should go down that route. That being said, this is also a >>>>> possible >>>>> point of friction. >>>>> >>>>> One nice thing is that this also completely works with the full >>>>> anaconda; if someone wants everything that is in the anaconda >>>>> install, >>>>> they can even simply do "conda install anaconda" from the >>>>> command >>>>> line >>>>> to get it. But the stripped down subset is the default. >>>>> >>>>> If anyone has a chance to try this out and has feedback, I'd >>>>> greatly >>>>> appreciate it! I think Nathan has done something very similar >>>>> for >>>>> OSX. I've also put a couple simple conda recipes here: >>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a >>>>> basis >>>>> for distributing builds and setting them up on buildbots and the >>>>> like. >>>>> I'm pretty optimistic about this. >>>>> >>>>> -Matt >>>>> >>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>> nathan12343@gmail.com wrote: >>>>>> I think to get everything working in a sustainable fashion, we >>>>>> would need >>>>>> buildbots for all platform combinations that we want to >>>>>> support, so >>>>>> all >>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>> py27/py3.3) tuple. >>>>>> At the moment anaconda seems to support 32 and 64 bit linux, 64 >>>>>> bit >>>>>> OS X >>>>>> (not totally clear if OS X version matters), and 32 and 64 bit >>>>>> windows. >>>>>> >>>>>> Another option is to rely on conda build, which compiles >>>>>> everything >>>>>> from >>>>>> source. >>>>>> >>>>>> >>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>> wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have less of a skin in this than I used to, but I'd like to >>>>>>> raise >>>>>>> the issue of Windows & package managers. For example, Anaconda >>>>>>> is >>>>>>> available for Windows - would that mean that yt might "just >>>>>>> work" >>>>>>> on >>>>>>> Windows? Or the opposite, and it would require a great deal of >>>>>>> effort >>>>>>> to get all the various things we expect to be .so's to work as >>>>>>> .dll's >>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>> >>>>>>> I don't know the answers to these questions, but I think it's >>>>>>> worth >>>>>>> thinking about. >>>>>>> >>>>>>> -- >>>>>>> Stephen Skory >>>>>>> s@skory.us >>>>>>> http://stephenskory.com/ >>>>>>> 510.621.3687 (google voice) >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Not 100% sure. It certainly won't work on OS X at the moment.
-Nathan
On Tue, Sep 10, 2013 at 10:59 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey John,
A few questions:
- What did you choose for INST_YT_SOURCE in get_yt.sh? If
INST_YT_SOURCE=1,
what happens when you try INST_YT_SOURCE=0?
- Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py
install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries it tries to link against. This should be the very first thing printed out
by
the setup script. Is it linking against anaconda's libraries or system libraries?
- What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries
provided
by anaconda unless we tailor our build environment to match the sort of gymnastics that `conda build` does to sanitize the build environment.
Just
yesterday I was having no end of grief trying to link against anaconda's hdf5 library. yt is able to link against this library inside a conda
build
environment, but when I try to link in my normal shell environment I run into issues - mostly because the OS X hdf5 library on anaconda was
compiled
on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py",
line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py",
line
60, in <module> from yt.data_objects.api import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py",
line 31, in <module> from grid_patch import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py",
line 35, in <module> from yt.data_objects.data_containers import YTFieldData File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py",
line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py",
line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import
\
File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py",
line 39, in <module> from yt.utilities.lib import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py",
line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required
by
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com
wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't
resolve.
Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was
the
workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need. Nearly
all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt, now
that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on
is
getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves
into
it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very
much
not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith <
brittonsmith@gmail.com>
wrote:
Hi everyone,
Sorry for chiming in late. I just moved when this thread began and
do
not have regular internet access.
I really like this idea of conda, especially as a package manager
that
only optionally makes its own edits to your .bashrc. I have always
really
liked that the install script creates a clean python stack with basically everything a python user needs. I have on occasion suggested it to people just looking to use numpy and matploblib. It looks like conda will continue to provide this nice by-product, so I'm all for it.
I won't be in a position to help with testing and such for another week or so when I get regular internet access, but I would be glad to do so then.
Britton
On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum nathan12343@gmail.com wrote: > > Everything should be available now for 64 bit linux and OS X. > > > On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone > chris.m.malone@gmail.com > wrote: >> >> Hi Nathan, >> >> That appears to work as it built the environment and `conda
install
>> ...` >> added packages to my environment. >> >> One mistake I made was that I originally downloaded the "latest"
OS
>> X >> build of Miniconda, but that happened to be Miniconda3, which is >> python 3 >> based. Trying to build the environment with that yields an error >> regarding >> incompatibility of yt and python3.3, as it should. >> >> Chris >> >> >> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >> nathan12343@gmail.com >> wrote: >>> >>> Hey Chris, >>> >>> I don't think mercurial is strictly necessary, can you try again >>> without >>> it? I think if Matt uploads a mercurial package for OS X this >>> won't be an >>> issue. I'll send him an updated tarball. >>> >>> I submitted a mercurial recipe to conda-recipes yesterday >>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so
hopefully
>>> a >>> mercurial build will be included in future anaconda releases. >>> >>> >>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>> chris.m.malone@gmail.com wrote: >>>> >>>> I just tried setting this up on OS X 10.7.5 and failed when >>>> attempting >>>> to create the conda environment due to a missing mercurial >>>> package: >>>> >>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_projectyt >>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>> Error: No packages found matching: mercurial >>>> >>>> >>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>> nathan12343@gmail.com wrote: >>>>> >>>>> Yup, please try on OSX as well. If you make sure Matt's
binstar
>>>>> is in >>>>> your .condarc, you should be able to get yt by doing 'conda >>>>> install yt'. >>>>> >>>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>>> about >>>>> issues, particularly if there are issues on older OS X
releases.
>>>>> >>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> Thank you for the feedback -- I am glad there is some
agreement
>>>>>> about >>>>>> possible ways forward, and so I'm happy to try to use this as
an
>>>>>> opportunity to explore simpler, more reliable methods than the >>>>>> install >>>>>> script. >>>>>> >>>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>>> it's >>>>>> quite nice. There are a few rough corners, particularly
related
>>>>>> to >>>>>> the binstar service, but it's so far pretty great. With >>>>>> Nathan's >>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64 >>>>>> and >>>>>> then install it. >>>>>> >>>>>> The workflow that seems to work: >>>>>> >>>>>> * Get miniconda:
http://repo.continuum.io/miniconda/index.html
>>>>>> * Run the installer for miniconda >>>>>> * Enter the conda environment and then install yt by doing >>>>>> "conda >>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>> >>>>>> I think that this can likely all be stuck into a bash script.
A
>>>>>> simple, first pass at this is here: >>>>>> >>>>>> http://paste.yt-project.org/show/3833/ >>>>>> >>>>>> This right now only works on Linux x86_64, but getting it to >>>>>> work for >>>>>> other machines won't be too hard. I suspect we will be able
to
>>>>>> do >>>>>> nightlies very easily as well. If anyone out there has an >>>>>> x86_64 >>>>>> machine they wouldn't mind trying it on, that would be very >>>>>> helpful! >>>>>> I did find that once I ran this script, I had to actually >>>>>> prepend the >>>>>> PATH afterwards as well. This means doing: >>>>>> >>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>> source activate ytenv >>>>>> >>>>>> At that point, everything was set up and working for me. The >>>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>>> think >>>>>> we >>>>>> should go down that route. That being said, this is also a >>>>>> possible >>>>>> point of friction. >>>>>> >>>>>> One nice thing is that this also completely works with the
full
>>>>>> anaconda; if someone wants everything that is in the anaconda >>>>>> install, >>>>>> they can even simply do "conda install anaconda" from the >>>>>> command >>>>>> line >>>>>> to get it. But the stripped down subset is the default. >>>>>> >>>>>> If anyone has a chance to try this out and has feedback, I'd >>>>>> greatly >>>>>> appreciate it! I think Nathan has done something very similar >>>>>> for >>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use
as a
>>>>>> basis >>>>>> for distributing builds and setting them up on buildbots and
the
>>>>>> like. >>>>>> I'm pretty optimistic about this. >>>>>> >>>>>> -Matt >>>>>> >>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>> nathan12343@gmail.com wrote: >>>>>>> I think to get everything working in a sustainable fashion,
we
>>>>>>> would need >>>>>>> buildbots for all platform combinations that we want to >>>>>>> support, so >>>>>>> all >>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>> py27/py3.3) tuple. >>>>>>> At the moment anaconda seems to support 32 and 64 bit linux,
64
>>>>>>> bit >>>>>>> OS X >>>>>>> (not totally clear if OS X version matters), and 32 and 64
bit
>>>>>>> windows. >>>>>>> >>>>>>> Another option is to rely on conda build, which compiles >>>>>>> everything >>>>>>> from >>>>>>> source. >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I have less of a skin in this than I used to, but I'd like
to
>>>>>>>> raise >>>>>>>> the issue of Windows & package managers. For example,
Anaconda
>>>>>>>> is >>>>>>>> available for Windows - would that mean that yt might "just >>>>>>>> work" >>>>>>>> on >>>>>>>> Windows? Or the opposite, and it would require a great deal
of
>>>>>>>> effort >>>>>>>> to get all the various things we expect to be .so's to work
as
>>>>>>>> .dll's >>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>> >>>>>>>> I don't know the answers to these questions, but I think
it's
>>>>>>>> worth >>>>>>>> thinking about. >>>>>>>> >>>>>>>> -- >>>>>>>> Stephen Skory >>>>>>>> s@skory.us >>>>>>>> http://stephenskory.com/ >>>>>>>> 510.621.3687 (google voice) >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Nathan,
Okay -- but if that's the price we pay, I think it may still be worth it.
-Matt
On Tue, Sep 10, 2013 at 2:09 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Not 100% sure. It certainly won't work on OS X at the moment.
-Nathan
On Tue, Sep 10, 2013 at 10:59 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Hey John,
A few questions:
- What did you choose for INST_YT_SOURCE in get_yt.sh? If
INST_YT_SOURCE=1, what happens when you try INST_YT_SOURCE=0?
- Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py
install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries it tries to link against. This should be the very first thing printed out by the setup script. Is it linking against anaconda's libraries or system libraries?
- What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort of gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against anaconda's hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I run into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py", line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py", line 31, in <module> from grid_patch import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py", line 35, in <module> from yt.data_objects.data_containers import YTFieldData File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py", line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py", line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 39, in <module> from yt.utilities.lib import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py", line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source) was the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
Thanks to everybody who has reported back on testing. After some talking both offline and on IRC, as well as here, I think we would need to do the following things:
- Make a single script that grabs the appropriate distribution of
miniconda and installs it. Right now I have a mechanism for doing this, but it's currently tied to an architecture.
- Create a mechanism for installing all the packages we need.
Nearly all are available inside the Continuum repos. What we're hung up on is that source installs require a "recipe", and transmitting the recipe is where I don't have an idea of what to do.
- Test this out lots of places
- Clean up the edges in the (new) install script
- Move the old install script to maintenance mode
- Update all documentation to describe this and mothball other
methods of installation
What would be nice:
- Make available nightly builds of yt on several architectures using
binstar
- Utilize more of the packages included in conda elsewhere in yt,
now that we can!
Here's my current recipe for get_yt.sh:
http://paste.yt-project.org/show/3843/
(The config thing may get switched to include the --system argument, to modify the "yt-conda" condarc.) The step that I'm most stuck on is getting the yt recipe to people. If we want to make it possible and easy to build from source, we need to get the contents of a "conda recipe" to people. They can then run "conda build ." in the directory. Here are the recipes that we've been playing with:
https://bitbucket.org/yt_analysis/yt_conda/src
Basically, in get_yt.sh, to do from source instead of from binary we need to insert a step at the end that downloads the recipes somehow and then cd's into the right directory and builds them. The reason this is tricky is that we often need to bootstrap ourselves; we can't assume anything exists. We can download the .tar.bz2 of the current tip of the repo, but it includes the hash in the directory name that it extracts to.
So I think what we need is a mechanism for getting the current state of the repo, figuring out the name of the repo's directory, moves into it, and then builds. I believe that all/most of this becomes much, much easier if hg gets included in Anaconda, which Nathan has submitted a PR for. So hopefully that will be taken care of, but until that time we can possibly figure something out. I'm not sure that we have the resources to continually support binary nightly builds in perpetuity for all the architectures that people run on, so having source would be awesome. Plus, one of the biggest appeals of how we distribute yt is that the source is included; I would very much not like to give this up.
Thoughts? Has anyone else tested any of this out?
-Matt
On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith brittonsmith@gmail.com wrote: > Hi everyone, > > Sorry for chiming in late. I just moved when this thread began and > do > not > have regular internet access. > > I really like this idea of conda, especially as a package manager > that > only > optionally makes its own edits to your .bashrc. I have always > really > liked > that the install script creates a clean python stack with basically > everything a python user needs. I have on occasion suggested it to > people > just looking to use numpy and matploblib. It looks like conda will > continue > to provide this nice by-product, so I'm all for it. > > I won't be in a position to help with testing and such for another > week or > so when I get regular internet access, but I would be glad to do so > then. > > Britton > > > On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum > nathan12343@gmail.com > wrote: >> >> Everything should be available now for 64 bit linux and OS X. >> >> >> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone >> chris.m.malone@gmail.com >> wrote: >>> >>> Hi Nathan, >>> >>> That appears to work as it built the environment and `conda >>> install >>> ...` >>> added packages to my environment. >>> >>> One mistake I made was that I originally downloaded the "latest" >>> OS >>> X >>> build of Miniconda, but that happened to be Miniconda3, which is >>> python 3 >>> based. Trying to build the environment with that yields an error >>> regarding >>> incompatibility of yt and python3.3, as it should. >>> >>> Chris >>> >>> >>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >>> nathan12343@gmail.com >>> wrote: >>>> >>>> Hey Chris, >>>> >>>> I don't think mercurial is strictly necessary, can you try again >>>> without >>>> it? I think if Matt uploads a mercurial package for OS X this >>>> won't be an >>>> issue. I'll send him an updated tarball. >>>> >>>> I submitted a mercurial recipe to conda-recipes yesterday >>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>>> hopefully >>>> a >>>> mercurial build will be included in future anaconda releases. >>>> >>>> >>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>>> chris.m.malone@gmail.com wrote: >>>>> >>>>> I just tried setting this up on OS X 10.7.5 and failed when >>>>> attempting >>>>> to create the conda environment due to a missing mercurial >>>>> package: >>>>> >>>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project >>>>> yt >>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>>> Error: No packages found matching: mercurial >>>>> >>>>> >>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>>> nathan12343@gmail.com wrote: >>>>>> >>>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>>> binstar >>>>>> is in >>>>>> your .condarc, you should be able to get yt by doing 'conda >>>>>> install yt'. >>>>>> >>>>>> I built the OSX binary on my laptop so I'd appreciate hearing >>>>>> about >>>>>> issues, particularly if there are issues on older OS X >>>>>> releases. >>>>>> >>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Thank you for the feedback -- I am glad there is some >>>>>>> agreement >>>>>>> about >>>>>>> possible ways forward, and so I'm happy to try to use this as >>>>>>> an >>>>>>> opportunity to explore simpler, more reliable methods than >>>>>>> the >>>>>>> install >>>>>>> script. >>>>>>> >>>>>>> This afternoon, I spent a bit of time with Conda, and I think >>>>>>> it's >>>>>>> quite nice. There are a few rough corners, particularly >>>>>>> related >>>>>>> to >>>>>>> the binstar service, but it's so far pretty great. With >>>>>>> Nathan's >>>>>>> help, I was able to upload a yt-2.5.5 package for linux >>>>>>> x86_64 >>>>>>> and >>>>>>> then install it. >>>>>>> >>>>>>> The workflow that seems to work: >>>>>>> >>>>>>> * Get miniconda: >>>>>>> http://repo.continuum.io/miniconda/index.html >>>>>>> * Run the installer for miniconda >>>>>>> * Enter the conda environment and then install yt by doing >>>>>>> "conda >>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>>> >>>>>>> I think that this can likely all be stuck into a bash script. >>>>>>> A >>>>>>> simple, first pass at this is here: >>>>>>> >>>>>>> http://paste.yt-project.org/show/3833/ >>>>>>> >>>>>>> This right now only works on Linux x86_64, but getting it to >>>>>>> work for >>>>>>> other machines won't be too hard. I suspect we will be able >>>>>>> to >>>>>>> do >>>>>>> nightlies very easily as well. If anyone out there has an >>>>>>> x86_64 >>>>>>> machine they wouldn't mind trying it on, that would be very >>>>>>> helpful! >>>>>>> I did find that once I ran this script, I had to actually >>>>>>> prepend the >>>>>>> PATH afterwards as well. This means doing: >>>>>>> >>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>>> source activate ytenv >>>>>>> >>>>>>> At that point, everything was set up and working for me. The >>>>>>> miniconda install offers to add paths to .bashrc, but I don't >>>>>>> think >>>>>>> we >>>>>>> should go down that route. That being said, this is also a >>>>>>> possible >>>>>>> point of friction. >>>>>>> >>>>>>> One nice thing is that this also completely works with the >>>>>>> full >>>>>>> anaconda; if someone wants everything that is in the anaconda >>>>>>> install, >>>>>>> they can even simply do "conda install anaconda" from the >>>>>>> command >>>>>>> line >>>>>>> to get it. But the stripped down subset is the default. >>>>>>> >>>>>>> If anyone has a chance to try this out and has feedback, I'd >>>>>>> greatly >>>>>>> appreciate it! I think Nathan has done something very >>>>>>> similar >>>>>>> for >>>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use >>>>>>> as a >>>>>>> basis >>>>>>> for distributing builds and setting them up on buildbots and >>>>>>> the >>>>>>> like. >>>>>>> I'm pretty optimistic about this. >>>>>>> >>>>>>> -Matt >>>>>>> >>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>>> nathan12343@gmail.com wrote: >>>>>>>> I think to get everything working in a sustainable fashion, >>>>>>>> we >>>>>>>> would need >>>>>>>> buildbots for all platform combinations that we want to >>>>>>>> support, so >>>>>>>> all >>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>>> py27/py3.3) tuple. >>>>>>>> At the moment anaconda seems to support 32 and 64 bit linux, >>>>>>>> 64 >>>>>>>> bit >>>>>>>> OS X >>>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>>> bit >>>>>>>> windows. >>>>>>>> >>>>>>>> Another option is to rely on conda build, which compiles >>>>>>>> everything >>>>>>>> from >>>>>>>> source. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory s@skory.us >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I have less of a skin in this than I used to, but I'd like >>>>>>>>> to >>>>>>>>> raise >>>>>>>>> the issue of Windows & package managers. For example, >>>>>>>>> Anaconda >>>>>>>>> is >>>>>>>>> available for Windows - would that mean that yt might "just >>>>>>>>> work" >>>>>>>>> on >>>>>>>>> Windows? Or the opposite, and it would require a great deal >>>>>>>>> of >>>>>>>>> effort >>>>>>>>> to get all the various things we expect to be .so's to work >>>>>>>>> as >>>>>>>>> .dll's >>>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>>> >>>>>>>>> I don't know the answers to these questions, but I think >>>>>>>>> it's >>>>>>>>> worth >>>>>>>>> thinking about. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Stephen Skory >>>>>>>>> s@skory.us >>>>>>>>> http://stephenskory.com/ >>>>>>>>> 510.621.3687 (google voice) >>>>>>>>> _______________________________________________ >>>>>>>>> yt-dev mailing list >>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Why does conda come with libraries built on such an old system? That seems a bit strange, especially for Mac, which has evolved quite a bit since 10.5. Is this sort of thing really not a problem for us?
Britton
On Tue, Sep 10, 2013 at 7:10 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
Okay -- but if that's the price we pay, I think it may still be worth it.
-Matt
On Tue, Sep 10, 2013 at 2:09 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Not 100% sure. It certainly won't work on OS X at the moment.
-Nathan
On Tue, Sep 10, 2013 at 10:59 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Hey John,
A few questions:
- What did you choose for INST_YT_SOURCE in get_yt.sh? If
INST_YT_SOURCE=1, what happens when you try INST_YT_SOURCE=0?
- Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py
install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries
it
tries to link against. This should be the very first thing printed
out
by the setup script. Is it linking against anaconda's libraries or
system
libraries?
- What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort
of
gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against
anaconda's
hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I
run
into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com
wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py",
line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py",
line 31, in <module> from grid_patch import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py",
line 35, in <module> from yt.data_objects.data_containers import YTFieldData File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py",
line 45, in <module> from yt.data_objects.derived_quantities import
GridChildMaskWrapper
File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py",
line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface
import
\ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py",
line 39, in <module> from yt.utilities.lib import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py",
line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found
(required
by
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue
tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo, I think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com
wrote:
Works fine for me on OS X x86_64.
On my Goddard-controlled Linux x86_64 server, everything worked fine except Mercurial, which I "conda install"-ed using the yt link, but the binary had an incompatibility which the glibc that was installed on the machine. Using pip to install Mercurial (which did it from source)
was
the workaround.
On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com wrote:
> Hi all, > > Thanks to everybody who has reported back on testing. After some > talking both offline and on IRC, as well as here, I think we would > need to do the following things: > > * Make a single script that grabs the appropriate distribution of > miniconda and installs it. Right now I have a mechanism for doing > this, but it's currently tied to an architecture. > * Create a mechanism for installing all the packages we need. > Nearly > all are available inside the Continuum repos. What we're hung up
on
> is that source installs require a "recipe", and transmitting the > recipe is where I don't have an idea of what to do. > * Test this out lots of places > * Clean up the edges in the (new) install script > * Move the old install script to maintenance mode > * Update all documentation to describe this and mothball other > methods of installation > > What would be nice: > > * Make available nightly builds of yt on several architectures
using
> binstar > * Utilize more of the packages included in conda elsewhere in yt, > now > that we can! > > Here's my current recipe for get_yt.sh: > > http://paste.yt-project.org/show/3843/ > > (The config thing may get switched to include the --system
argument,
> to modify the "yt-conda" condarc.) The step that I'm most stuck
on
> is > getting the yt recipe to people. If we want to make it possible
and
> easy to build from source, we need to get the contents of a "conda > recipe" to people. They can then run "conda build ." in the > directory. Here are the recipes that we've been playing with: > > https://bitbucket.org/yt_analysis/yt_conda/src > > Basically, in get_yt.sh, to do from source instead of from binary
we
> need to insert a step at the end that downloads the recipes
somehow
> and then cd's into the right directory and builds them. The
reason
> this is tricky is that we often need to bootstrap ourselves; we > can't > assume anything exists. We can download the .tar.bz2 of the
current
> tip of the repo, but it includes the hash in the directory name
that
> it extracts to. > > So I think what we need is a mechanism for getting the current
state
> of the repo, figuring out the name of the repo's directory, moves > into > it, and then builds. I believe that all/most of this becomes
much,
> much easier if hg gets included in Anaconda, which Nathan has > submitted a PR for. So hopefully that will be taken care of, but > until that time we can possibly figure something out. I'm not
sure
> that we have the resources to continually support binary nightly > builds in perpetuity for all the architectures that people run on, > so > having source would be awesome. Plus, one of the biggest appeals
of
> how we distribute yt is that the source is included; I would very > much > not like to give this up. > > Thoughts? Has anyone else tested any of this out? > > -Matt > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > brittonsmith@gmail.com > wrote: >> Hi everyone, >> >> Sorry for chiming in late. I just moved when this thread began
and
>> do >> not >> have regular internet access. >> >> I really like this idea of conda, especially as a package manager >> that >> only >> optionally makes its own edits to your .bashrc. I have always >> really >> liked >> that the install script creates a clean python stack with
basically
>> everything a python user needs. I have on occasion suggested it
to
>> people >> just looking to use numpy and matploblib. It looks like conda
will
>> continue >> to provide this nice by-product, so I'm all for it. >> >> I won't be in a position to help with testing and such for
another
>> week or >> so when I get regular internet access, but I would be glad to do
so
>> then. >> >> Britton >> >> >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum >> nathan12343@gmail.com >> wrote: >>> >>> Everything should be available now for 64 bit linux and OS X. >>> >>> >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone >>> chris.m.malone@gmail.com >>> wrote: >>>> >>>> Hi Nathan, >>>> >>>> That appears to work as it built the environment and `conda >>>> install >>>> ...` >>>> added packages to my environment. >>>> >>>> One mistake I made was that I originally downloaded the
"latest"
>>>> OS >>>> X >>>> build of Miniconda, but that happened to be Miniconda3, which
is
>>>> python 3 >>>> based. Trying to build the environment with that yields an
error
>>>> regarding >>>> incompatibility of yt and python3.3, as it should. >>>> >>>> Chris >>>> >>>> >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum >>>> nathan12343@gmail.com >>>> wrote: >>>>> >>>>> Hey Chris, >>>>> >>>>> I don't think mercurial is strictly necessary, can you try
again
>>>>> without >>>>> it? I think if Matt uploads a mercurial package for OS X this >>>>> won't be an >>>>> issue. I'll send him an updated tarball. >>>>> >>>>> I submitted a mercurial recipe to conda-recipes yesterday >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so >>>>> hopefully >>>>> a >>>>> mercurial build will be included in future anaconda releases. >>>>> >>>>> >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone >>>>> chris.m.malone@gmail.com wrote: >>>>>> >>>>>> I just tried setting this up on OS X 10.7.5 and failed when >>>>>> attempting >>>>>> to create the conda environment due to a missing mercurial >>>>>> package: >>>>>> >>>>>> $ conda create -n ytenv -c
http://conda.binstar.org/yt_project
>>>>>> yt >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx >>>>>> Error: No packages found matching: mercurial >>>>>> >>>>>> >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum >>>>>> nathan12343@gmail.com wrote: >>>>>>> >>>>>>> Yup, please try on OSX as well. If you make sure Matt's >>>>>>> binstar >>>>>>> is in >>>>>>> your .condarc, you should be able to get yt by doing 'conda >>>>>>> install yt'. >>>>>>> >>>>>>> I built the OSX binary on my laptop so I'd appreciate
hearing
>>>>>>> about >>>>>>> issues, particularly if there are issues on older OS X >>>>>>> releases. >>>>>>> >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Thank you for the feedback -- I am glad there is some >>>>>>>> agreement >>>>>>>> about >>>>>>>> possible ways forward, and so I'm happy to try to use this
as
>>>>>>>> an >>>>>>>> opportunity to explore simpler, more reliable methods than >>>>>>>> the >>>>>>>> install >>>>>>>> script. >>>>>>>> >>>>>>>> This afternoon, I spent a bit of time with Conda, and I
think
>>>>>>>> it's >>>>>>>> quite nice. There are a few rough corners, particularly >>>>>>>> related >>>>>>>> to >>>>>>>> the binstar service, but it's so far pretty great. With >>>>>>>> Nathan's >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux >>>>>>>> x86_64 >>>>>>>> and >>>>>>>> then install it. >>>>>>>> >>>>>>>> The workflow that seems to work: >>>>>>>> >>>>>>>> * Get miniconda: >>>>>>>> http://repo.continuum.io/miniconda/index.html >>>>>>>> * Run the installer for miniconda >>>>>>>> * Enter the conda environment and then install yt by doing >>>>>>>> "conda >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". >>>>>>>> >>>>>>>> I think that this can likely all be stuck into a bash
script.
>>>>>>>> A >>>>>>>> simple, first pass at this is here: >>>>>>>> >>>>>>>> http://paste.yt-project.org/show/3833/ >>>>>>>> >>>>>>>> This right now only works on Linux x86_64, but getting it
to
>>>>>>>> work for >>>>>>>> other machines won't be too hard. I suspect we will be
able
>>>>>>>> to >>>>>>>> do >>>>>>>> nightlies very easily as well. If anyone out there has an >>>>>>>> x86_64 >>>>>>>> machine they wouldn't mind trying it on, that would be very >>>>>>>> helpful! >>>>>>>> I did find that once I ran this script, I had to actually >>>>>>>> prepend the >>>>>>>> PATH afterwards as well. This means doing: >>>>>>>> >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH >>>>>>>> export PATH=$HOME/yt-conda:$PATH >>>>>>>> source activate ytenv >>>>>>>> >>>>>>>> At that point, everything was set up and working for me.
The
>>>>>>>> miniconda install offers to add paths to .bashrc, but I
don't
>>>>>>>> think >>>>>>>> we >>>>>>>> should go down that route. That being said, this is also a >>>>>>>> possible >>>>>>>> point of friction. >>>>>>>> >>>>>>>> One nice thing is that this also completely works with the >>>>>>>> full >>>>>>>> anaconda; if someone wants everything that is in the
anaconda
>>>>>>>> install, >>>>>>>> they can even simply do "conda install anaconda" from the >>>>>>>> command >>>>>>>> line >>>>>>>> to get it. But the stripped down subset is the default. >>>>>>>> >>>>>>>> If anyone has a chance to try this out and has feedback,
I'd
>>>>>>>> greatly >>>>>>>> appreciate it! I think Nathan has done something very >>>>>>>> similar >>>>>>>> for >>>>>>>> OSX. I've also put a couple simple conda recipes here: >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can
use
>>>>>>>> as a >>>>>>>> basis >>>>>>>> for distributing builds and setting them up on buildbots
and
>>>>>>>> the >>>>>>>> like. >>>>>>>> I'm pretty optimistic about this. >>>>>>>> >>>>>>>> -Matt >>>>>>>> >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum >>>>>>>> nathan12343@gmail.com wrote: >>>>>>>>> I think to get everything working in a sustainable
fashion,
>>>>>>>>> we >>>>>>>>> would need >>>>>>>>> buildbots for all platform combinations that we want to >>>>>>>>> support, so >>>>>>>>> all >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, >>>>>>>>> py27/py3.3) tuple. >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit
linux,
>>>>>>>>> 64 >>>>>>>>> bit >>>>>>>>> OS X >>>>>>>>> (not totally clear if OS X version matters), and 32 and 64 >>>>>>>>> bit >>>>>>>>> windows. >>>>>>>>> >>>>>>>>> Another option is to rely on conda build, which compiles >>>>>>>>> everything >>>>>>>>> from >>>>>>>>> source. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <
s@skory.us>
>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have less of a skin in this than I used to, but I'd
like
>>>>>>>>>> to >>>>>>>>>> raise >>>>>>>>>> the issue of Windows & package managers. For example, >>>>>>>>>> Anaconda >>>>>>>>>> is >>>>>>>>>> available for Windows - would that mean that yt might
"just
>>>>>>>>>> work" >>>>>>>>>> on >>>>>>>>>> Windows? Or the opposite, and it would require a great
deal
>>>>>>>>>> of >>>>>>>>>> effort >>>>>>>>>> to get all the various things we expect to be .so's to
work
>>>>>>>>>> as >>>>>>>>>> .dll's >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? >>>>>>>>>> >>>>>>>>>> I don't know the answers to these questions, but I think >>>>>>>>>> it's >>>>>>>>>> worth >>>>>>>>>> thinking about. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Stephen Skory >>>>>>>>>> s@skory.us >>>>>>>>>> http://stephenskory.com/ >>>>>>>>>> 510.621.3687 (google voice) >>>>>>>>>> _______________________________________________ >>>>>>>>>> yt-dev mailing list >>>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>>> >>>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> yt-dev mailing list >>>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-dev mailing list >>>>>>>> yt-dev@lists.spacepope.org >>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-dev mailing list >>>>>>> yt-dev@lists.spacepope.org >>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-dev mailing list >>>>>> yt-dev@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
It is a problem. See the discussion I had with Ilan Schnell (who works at Continuum Analytics on conda and anaconda) yesterday:
https://github.com/ContinuumIO/anaconda-issues/issues/18#issuecomment-241064...
There's some magic in the `conda build` command that allows us to link against these libraries, even if they were built on a 10.5 machine. This fails when we try to link against the library normally, for example when you do `setup.py install` for a package that links directly against hdf5.
Nathan
On Tue, Sep 10, 2013 at 12:24 PM, Britton Smith brittonsmith@gmail.comwrote:
Why does conda come with libraries built on such an old system? That seems a bit strange, especially for Mac, which has evolved quite a bit since 10.5. Is this sort of thing really not a problem for us?
Britton
On Tue, Sep 10, 2013 at 7:10 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi Nathan,
Okay -- but if that's the price we pay, I think it may still be worth it.
-Matt
On Tue, Sep 10, 2013 at 2:09 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
Not 100% sure. It certainly won't work on OS X at the moment.
-Nathan
On Tue, Sep 10, 2013 at 10:59 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Hey John,
A few questions:
- What did you choose for INST_YT_SOURCE in get_yt.sh? If
INST_YT_SOURCE=1, what happens when you try INST_YT_SOURCE=0?
- Again, if you had INST_YT_SOURCE=1, when you manually run
`setup.py
install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries
it
tries to link against. This should be the very first thing printed
out
by the setup script. Is it linking against anaconda's libraries or
system
libraries?
- What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort
of
gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against
anaconda's
hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I
run
into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone jzuhone@gmail.com
wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py",
line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py",
line 31, in <module> from grid_patch import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py",
line 35, in <module> from yt.data_objects.data_containers import YTFieldData File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py",
line 45, in <module> from yt.data_objects.derived_quantities import
GridChildMaskWrapper
File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py",
line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface
import
\ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py",
line 39, in <module> from yt.utilities.lib import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py",
line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found
(required
by
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum nathan12343@gmail.com wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue
tracker.
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo,
I
think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone jzuhone@gmail.com
wrote:
> > Works fine for me on OS X x86_64. > > On my Goddard-controlled Linux x86_64 server, everything worked
fine
> except Mercurial, which I "conda install"-ed using the yt link, but > the > binary had an incompatibility which the glibc that was installed on > the > machine. Using pip to install Mercurial (which did it from source)
was
> the > workaround. > > On Sep 3, 2013, at 2:54 PM, Matthew Turk matthewturk@gmail.com > wrote: > > > Hi all, > > > > Thanks to everybody who has reported back on testing. After some > > talking both offline and on IRC, as well as here, I think we
would
> > need to do the following things: > > > > * Make a single script that grabs the appropriate distribution of > > miniconda and installs it. Right now I have a mechanism for
doing
> > this, but it's currently tied to an architecture. > > * Create a mechanism for installing all the packages we need. > > Nearly > > all are available inside the Continuum repos. What we're hung
up on
> > is that source installs require a "recipe", and transmitting the > > recipe is where I don't have an idea of what to do. > > * Test this out lots of places > > * Clean up the edges in the (new) install script > > * Move the old install script to maintenance mode > > * Update all documentation to describe this and mothball other > > methods of installation > > > > What would be nice: > > > > * Make available nightly builds of yt on several architectures
using
> > binstar > > * Utilize more of the packages included in conda elsewhere in yt, > > now > > that we can! > > > > Here's my current recipe for get_yt.sh: > > > > http://paste.yt-project.org/show/3843/ > > > > (The config thing may get switched to include the --system
argument,
> > to modify the "yt-conda" condarc.) The step that I'm most stuck
on
> > is > > getting the yt recipe to people. If we want to make it possible
and
> > easy to build from source, we need to get the contents of a
"conda
> > recipe" to people. They can then run "conda build ." in the > > directory. Here are the recipes that we've been playing with: > > > > https://bitbucket.org/yt_analysis/yt_conda/src > > > > Basically, in get_yt.sh, to do from source instead of from
binary we
> > need to insert a step at the end that downloads the recipes
somehow
> > and then cd's into the right directory and builds them. The
reason
> > this is tricky is that we often need to bootstrap ourselves; we > > can't > > assume anything exists. We can download the .tar.bz2 of the
current
> > tip of the repo, but it includes the hash in the directory name
that
> > it extracts to. > > > > So I think what we need is a mechanism for getting the current
state
> > of the repo, figuring out the name of the repo's directory, moves > > into > > it, and then builds. I believe that all/most of this becomes
much,
> > much easier if hg gets included in Anaconda, which Nathan has > > submitted a PR for. So hopefully that will be taken care of, but > > until that time we can possibly figure something out. I'm not
sure
> > that we have the resources to continually support binary nightly > > builds in perpetuity for all the architectures that people run
on,
> > so > > having source would be awesome. Plus, one of the biggest
appeals of
> > how we distribute yt is that the source is included; I would very > > much > > not like to give this up. > > > > Thoughts? Has anyone else tested any of this out? > > > > -Matt > > > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > > brittonsmith@gmail.com > > wrote: > >> Hi everyone, > >> > >> Sorry for chiming in late. I just moved when this thread began
and
> >> do > >> not > >> have regular internet access. > >> > >> I really like this idea of conda, especially as a package
manager
> >> that > >> only > >> optionally makes its own edits to your .bashrc. I have always > >> really > >> liked > >> that the install script creates a clean python stack with
basically
> >> everything a python user needs. I have on occasion suggested
it to
> >> people > >> just looking to use numpy and matploblib. It looks like conda
will
> >> continue > >> to provide this nice by-product, so I'm all for it. > >> > >> I won't be in a position to help with testing and such for
another
> >> week or > >> so when I get regular internet access, but I would be glad to
do so
> >> then. > >> > >> Britton > >> > >> > >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum > >> nathan12343@gmail.com > >> wrote: > >>> > >>> Everything should be available now for 64 bit linux and OS X. > >>> > >>> > >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone > >>> chris.m.malone@gmail.com > >>> wrote: > >>>> > >>>> Hi Nathan, > >>>> > >>>> That appears to work as it built the environment and `conda > >>>> install > >>>> ...` > >>>> added packages to my environment. > >>>> > >>>> One mistake I made was that I originally downloaded the
"latest"
> >>>> OS > >>>> X > >>>> build of Miniconda, but that happened to be Miniconda3, which
is
> >>>> python 3 > >>>> based. Trying to build the environment with that yields an
error
> >>>> regarding > >>>> incompatibility of yt and python3.3, as it should. > >>>> > >>>> Chris > >>>> > >>>> > >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum > >>>> nathan12343@gmail.com > >>>> wrote: > >>>>> > >>>>> Hey Chris, > >>>>> > >>>>> I don't think mercurial is strictly necessary, can you try
again
> >>>>> without > >>>>> it? I think if Matt uploads a mercurial package for OS X
this
> >>>>> won't be an > >>>>> issue. I'll send him an updated tarball. > >>>>> > >>>>> I submitted a mercurial recipe to conda-recipes yesterday > >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so > >>>>> hopefully > >>>>> a > >>>>> mercurial build will be included in future anaconda releases. > >>>>> > >>>>> > >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > >>>>> chris.m.malone@gmail.com wrote: > >>>>>> > >>>>>> I just tried setting this up on OS X 10.7.5 and failed when > >>>>>> attempting > >>>>>> to create the conda environment due to a missing mercurial > >>>>>> package: > >>>>>> > >>>>>> $ conda create -n ytenv -c
http://conda.binstar.org/yt_project
> >>>>>> yt > >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx > >>>>>> Error: No packages found matching: mercurial > >>>>>> > >>>>>> > >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum > >>>>>> nathan12343@gmail.com wrote: > >>>>>>> > >>>>>>> Yup, please try on OSX as well. If you make sure Matt's > >>>>>>> binstar > >>>>>>> is in > >>>>>>> your .condarc, you should be able to get yt by doing 'conda > >>>>>>> install yt'. > >>>>>>> > >>>>>>> I built the OSX binary on my laptop so I'd appreciate
hearing
> >>>>>>> about > >>>>>>> issues, particularly if there are issues on older OS X > >>>>>>> releases. > >>>>>>> > >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: > >>>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> Thank you for the feedback -- I am glad there is some > >>>>>>>> agreement > >>>>>>>> about > >>>>>>>> possible ways forward, and so I'm happy to try to use
this as
> >>>>>>>> an > >>>>>>>> opportunity to explore simpler, more reliable methods than > >>>>>>>> the > >>>>>>>> install > >>>>>>>> script. > >>>>>>>> > >>>>>>>> This afternoon, I spent a bit of time with Conda, and I
think
> >>>>>>>> it's > >>>>>>>> quite nice. There are a few rough corners, particularly > >>>>>>>> related > >>>>>>>> to > >>>>>>>> the binstar service, but it's so far pretty great. With > >>>>>>>> Nathan's > >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux > >>>>>>>> x86_64 > >>>>>>>> and > >>>>>>>> then install it. > >>>>>>>> > >>>>>>>> The workflow that seems to work: > >>>>>>>> > >>>>>>>> * Get miniconda: > >>>>>>>> http://repo.continuum.io/miniconda/index.html > >>>>>>>> * Run the installer for miniconda > >>>>>>>> * Enter the conda environment and then install yt by doing > >>>>>>>> "conda > >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". > >>>>>>>> > >>>>>>>> I think that this can likely all be stuck into a bash
script.
> >>>>>>>> A > >>>>>>>> simple, first pass at this is here: > >>>>>>>> > >>>>>>>> http://paste.yt-project.org/show/3833/ > >>>>>>>> > >>>>>>>> This right now only works on Linux x86_64, but getting it
to
> >>>>>>>> work for > >>>>>>>> other machines won't be too hard. I suspect we will be
able
> >>>>>>>> to > >>>>>>>> do > >>>>>>>> nightlies very easily as well. If anyone out there has an > >>>>>>>> x86_64 > >>>>>>>> machine they wouldn't mind trying it on, that would be
very
> >>>>>>>> helpful! > >>>>>>>> I did find that once I ran this script, I had to actually > >>>>>>>> prepend the > >>>>>>>> PATH afterwards as well. This means doing: > >>>>>>>> > >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH > >>>>>>>> export PATH=$HOME/yt-conda:$PATH > >>>>>>>> source activate ytenv > >>>>>>>> > >>>>>>>> At that point, everything was set up and working for me.
The
> >>>>>>>> miniconda install offers to add paths to .bashrc, but I
don't
> >>>>>>>> think > >>>>>>>> we > >>>>>>>> should go down that route. That being said, this is also
a
> >>>>>>>> possible > >>>>>>>> point of friction. > >>>>>>>> > >>>>>>>> One nice thing is that this also completely works with the > >>>>>>>> full > >>>>>>>> anaconda; if someone wants everything that is in the
anaconda
> >>>>>>>> install, > >>>>>>>> they can even simply do "conda install anaconda" from the > >>>>>>>> command > >>>>>>>> line > >>>>>>>> to get it. But the stripped down subset is the default. > >>>>>>>> > >>>>>>>> If anyone has a chance to try this out and has feedback,
I'd
> >>>>>>>> greatly > >>>>>>>> appreciate it! I think Nathan has done something very > >>>>>>>> similar > >>>>>>>> for > >>>>>>>> OSX. I've also put a couple simple conda recipes here: > >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can
use
> >>>>>>>> as a > >>>>>>>> basis > >>>>>>>> for distributing builds and setting them up on buildbots
and
> >>>>>>>> the > >>>>>>>> like. > >>>>>>>> I'm pretty optimistic about this. > >>>>>>>> > >>>>>>>> -Matt > >>>>>>>> > >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum > >>>>>>>> nathan12343@gmail.com wrote: > >>>>>>>>> I think to get everything working in a sustainable
fashion,
> >>>>>>>>> we > >>>>>>>>> would need > >>>>>>>>> buildbots for all platform combinations that we want to > >>>>>>>>> support, so > >>>>>>>>> all > >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, > >>>>>>>>> py27/py3.3) tuple. > >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit
linux,
> >>>>>>>>> 64 > >>>>>>>>> bit > >>>>>>>>> OS X > >>>>>>>>> (not totally clear if OS X version matters), and 32 and
64
> >>>>>>>>> bit > >>>>>>>>> windows. > >>>>>>>>> > >>>>>>>>> Another option is to rely on conda build, which compiles > >>>>>>>>> everything > >>>>>>>>> from > >>>>>>>>> source. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <
s@skory.us>
> >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I have less of a skin in this than I used to, but I'd
like
> >>>>>>>>>> to > >>>>>>>>>> raise > >>>>>>>>>> the issue of Windows & package managers. For example, > >>>>>>>>>> Anaconda > >>>>>>>>>> is > >>>>>>>>>> available for Windows - would that mean that yt might
"just
> >>>>>>>>>> work" > >>>>>>>>>> on > >>>>>>>>>> Windows? Or the opposite, and it would require a great
deal
> >>>>>>>>>> of > >>>>>>>>>> effort > >>>>>>>>>> to get all the various things we expect to be .so's to
work
> >>>>>>>>>> as > >>>>>>>>>> .dll's > >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? > >>>>>>>>>> > >>>>>>>>>> I don't know the answers to these questions, but I think > >>>>>>>>>> it's > >>>>>>>>>> worth > >>>>>>>>>> thinking about. > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Stephen Skory > >>>>>>>>>> s@skory.us > >>>>>>>>>> http://stephenskory.com/ > >>>>>>>>>> 510.621.3687 (google voice) > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> yt-dev mailing list > >>>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>>> > >>>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> yt-dev mailing list > >>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> yt-dev mailing list > >>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> yt-dev mailing list > >>>>>>> yt-dev@lists.spacepope.org > >>>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> yt-dev mailing list > >>>>>> yt-dev@lists.spacepope.org > >>>>>>
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> yt-dev mailing list > >>>>> yt-dev@lists.spacepope.org > >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-dev mailing list > >>>> yt-dev@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>> > >>> > >>> > >>> _______________________________________________ > >>> yt-dev mailing list > >>> yt-dev@lists.spacepope.org > >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>> > >> > >> > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >> > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-
Britton Smith -
Cameron Hummels -
Chris Malone -
Douglas Harvey Rudd -
John ZuHone -
Kacper Kowalik -
Matthew Turk -
Nathan Goldbaum -
Sam Skillman -
Stephen Skory