It is a problem. See the discussion I had with Ilan Schnell (who works at Continuum Analytics on conda and anaconda) yesterday: https://github.com/ContinuumIO/anaconda-issues/issues/18#issuecomment-241064... There's some magic in the `conda build` command that allows us to link against these libraries, even if they were built on a 10.5 machine. This fails when we try to link against the library normally, for example when you do `setup.py install` for a package that links directly against hdf5. Nathan On Tue, Sep 10, 2013 at 12:24 PM, Britton Smith <brittonsmith@gmail.com>wrote:
Why does conda come with libraries built on such an old system? That seems a bit strange, especially for Mac, which has evolved quite a bit since 10.5. Is this sort of thing really not a problem for us?
Britton
On Tue, Sep 10, 2013 at 7:10 PM, Matthew Turk <matthewturk@gmail.com>wrote:
Hi Nathan,
Okay -- but if that's the price we pay, I think it may still be worth it.
-Matt
Not 100% sure. It certainly won't work on OS X at the moment.
-Nathan
On Tue, Sep 10, 2013 at 10:59 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Nathan,
On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Hey John,
A few questions:
1. What did you choose for INST_YT_SOURCE in get_yt.sh? If INST_YT_SOURCE=1, what happens when you try INST_YT_SOURCE=0?
2. Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py install` in the yt source repository, it should print out a few lines showing the root directory for the PNG, Freetype, and HDF5 libraries it tries to link against. This should be the very first thing printed out by the setup script. Is it linking against anaconda's libraries or system libraries?
3. What's the output of `ldd
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
I'm beginning to suspect that we cannot link against the libraries provided by anaconda unless we tailor our build environment to match the sort
of
gymnastics that `conda build` does to sanitize the build environment. Just yesterday I was having no end of grief trying to link against anaconda's hdf5 library. yt is able to link against this library inside a conda build environment, but when I try to link in my normal shell environment I run into issues - mostly because the OS X hdf5 library on anaconda was compiled on an OS X 10.5 machine.
If I read you correctly, you are saying that, practically speaking, this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc outside of yt. Right?
On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone <jzuhone@gmail.com>
wrote:
Hi Nathan and all,
Here's a little more info on that error you asked about regarding glibc. It came up again for me on Pleiades, this time with yt itself.
Traceback (most recent call last): File "/u/jzuhone/anaconda/bin/yt", line 4, in <module> from yt.utilities.command_line import run_main File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py",
line 29, in <module> from yt.mods import * File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line 60, in <module> from yt.data_objects.api import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py",
line 31, in <module> from grid_patch import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py",
line 35, in <module> from yt.data_objects.data_containers import YTFieldData File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py",
line 45, in <module> from yt.data_objects.derived_quantities import GridChildMaskWrapper File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py",
line 36, in <module> from yt.utilities.parallel_tools.parallel_analysis_interface import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py",
line 39, in <module> from yt.utilities.lib import \ File
"/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py",
line 35, in <module> from .png_writer import * ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by
/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
Best,
John
On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
John, can you give a little more info on that error? I'm a little concerned that there are binary incompatibilities that conda can't resolve. Perhaps this is something we should report on the conda issue
Matt, can't we get the recipes in the same way we get the latest dev install script? Something like:
wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
This doesn't currently work, although it does work for the yt repo,
I
think because we use named branches in that repository.
On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone <jzuhone@gmail.com> wrote: > > Works fine for me on OS X x86_64. > > On my Goddard-controlled Linux x86_64 server, everything worked fine > except Mercurial, which I "conda install"-ed using the yt link, but > the > binary had an incompatibility which the glibc that was installed on > the > machine. Using pip to install Mercurial (which did it from source) was > the > workaround. > > On Sep 3, 2013, at 2:54 PM, Matthew Turk <matthewturk@gmail.com> > wrote: > > > Hi all, > > > > Thanks to everybody who has reported back on testing. After some > > talking both offline and on IRC, as well as here, I think we would > > need to do the following things: > > > > * Make a single script that grabs the appropriate distribution of > > miniconda and installs it. Right now I have a mechanism for doing > > this, but it's currently tied to an architecture. > > * Create a mechanism for installing all the packages we need. > > Nearly > > all are available inside the Continuum repos. What we're hung up on > > is that source installs require a "recipe", and transmitting the > > recipe is where I don't have an idea of what to do. > > * Test this out lots of places > > * Clean up the edges in the (new) install script > > * Move the old install script to maintenance mode > > * Update all documentation to describe this and mothball other > > methods of installation > > > > What would be nice: > > > > * Make available nightly builds of yt on several architectures using > > binstar > > * Utilize more of the packages included in conda elsewhere in yt, > > now > > that we can! > > > > Here's my current recipe for get_yt.sh: > > > > http://paste.yt-project.org/show/3843/ > > > > (The config thing may get switched to include the --system argument, > > to modify the "yt-conda" condarc.) The step that I'm most stuck on > > is > > getting the yt recipe to people. If we want to make it possible and > > easy to build from source, we need to get the contents of a "conda > > recipe" to people. They can then run "conda build ." in the > > directory. Here are the recipes that we've been playing with: > > > > https://bitbucket.org/yt_analysis/yt_conda/src > > > > Basically, in get_yt.sh, to do from source instead of from binary we > > need to insert a step at the end that downloads the recipes somehow > > and then cd's into the right directory and builds them. The reason > > this is tricky is that we often need to bootstrap ourselves; we > > can't > > assume anything exists. We can download the .tar.bz2 of the current > > tip of the repo, but it includes the hash in the directory name
> > it extracts to. > > > > So I think what we need is a mechanism for getting the current state > > of the repo, figuring out the name of the repo's directory, moves > > into > > it, and then builds. I believe that all/most of this becomes much, > > much easier if hg gets included in Anaconda, which Nathan has > > submitted a PR for. So hopefully that will be taken care of, but > > until that time we can possibly figure something out. I'm not sure > > that we have the resources to continually support binary nightly > > builds in perpetuity for all the architectures that people run on, > > so > > having source would be awesome. Plus, one of the biggest appeals of > > how we distribute yt is that the source is included; I would very > > much > > not like to give this up. > > > > Thoughts? Has anyone else tested any of this out? > > > > -Matt > > > > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith > > <brittonsmith@gmail.com> > > wrote: > >> Hi everyone, > >> > >> Sorry for chiming in late. I just moved when this thread began and > >> do > >> not > >> have regular internet access. > >> > >> I really like this idea of conda, especially as a package manager > >> that > >> only > >> optionally makes its own edits to your .bashrc. I have always > >> really > >> liked > >> that the install script creates a clean python stack with basically > >> everything a python user needs. I have on occasion suggested it to > >> people > >> just looking to use numpy and matploblib. It looks like conda will > >> continue > >> to provide this nice by-product, so I'm all for it. > >> > >> I won't be in a position to help with testing and such for another > >> week or > >> so when I get regular internet access, but I would be glad to do so > >> then. > >> > >> Britton > >> > >> > >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum > >> <nathan12343@gmail.com> > >> wrote: > >>> > >>> Everything should be available now for 64 bit linux and OS X. > >>> > >>> > >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone > >>> <chris.m.malone@gmail.com> > >>> wrote: > >>>> > >>>> Hi Nathan, > >>>> > >>>> That appears to work as it built the environment and `conda > >>>> install > >>>> ...` > >>>> added packages to my environment. > >>>> > >>>> One mistake I made was that I originally downloaded the "latest" > >>>> OS > >>>> X > >>>> build of Miniconda, but that happened to be Miniconda3, which is > >>>> python 3 > >>>> based. Trying to build the environment with that yields an error > >>>> regarding > >>>> incompatibility of yt and python3.3, as it should. > >>>> > >>>> Chris > >>>> > >>>> > >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum > >>>> <nathan12343@gmail.com> > >>>> wrote: > >>>>> > >>>>> Hey Chris, > >>>>> > >>>>> I don't think mercurial is strictly necessary, can you try again > >>>>> without > >>>>> it? I think if Matt uploads a mercurial package for OS X
> >>>>> won't be an > >>>>> issue. I'll send him an updated tarball. > >>>>> > >>>>> I submitted a mercurial recipe to conda-recipes yesterday > >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so > >>>>> hopefully > >>>>> a > >>>>> mercurial build will be included in future anaconda releases. > >>>>> > >>>>> > >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone > >>>>> <chris.m.malone@gmail.com> wrote: > >>>>>> > >>>>>> I just tried setting this up on OS X 10.7.5 and failed when > >>>>>> attempting > >>>>>> to create the conda environment due to a missing mercurial > >>>>>> package: > >>>>>> > >>>>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project > >>>>>> yt > >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx > >>>>>> Error: No packages found matching: mercurial > >>>>>> > >>>>>> > >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum > >>>>>> <nathan12343@gmail.com> wrote: > >>>>>>> > >>>>>>> Yup, please try on OSX as well. If you make sure Matt's > >>>>>>> binstar > >>>>>>> is in > >>>>>>> your .condarc, you should be able to get yt by doing 'conda > >>>>>>> install yt'. > >>>>>>> > >>>>>>> I built the OSX binary on my laptop so I'd appreciate hearing > >>>>>>> about > >>>>>>> issues, particularly if there are issues on older OS X > >>>>>>> releases. > >>>>>>> > >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote: > >>>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> Thank you for the feedback -- I am glad there is some > >>>>>>>> agreement > >>>>>>>> about > >>>>>>>> possible ways forward, and so I'm happy to try to use
> >>>>>>>> an > >>>>>>>> opportunity to explore simpler, more reliable methods than > >>>>>>>> the > >>>>>>>> install > >>>>>>>> script. > >>>>>>>> > >>>>>>>> This afternoon, I spent a bit of time with Conda, and I
> >>>>>>>> it's > >>>>>>>> quite nice. There are a few rough corners, particularly > >>>>>>>> related > >>>>>>>> to > >>>>>>>> the binstar service, but it's so far pretty great. With > >>>>>>>> Nathan's > >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux > >>>>>>>> x86_64 > >>>>>>>> and > >>>>>>>> then install it. > >>>>>>>> > >>>>>>>> The workflow that seems to work: > >>>>>>>> > >>>>>>>> * Get miniconda: > >>>>>>>> http://repo.continuum.io/miniconda/index.html > >>>>>>>> * Run the installer for miniconda > >>>>>>>> * Enter the conda environment and then install yt by doing > >>>>>>>> "conda > >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ". > >>>>>>>> > >>>>>>>> I think that this can likely all be stuck into a bash
> >>>>>>>> A > >>>>>>>> simple, first pass at this is here: > >>>>>>>> > >>>>>>>> http://paste.yt-project.org/show/3833/ > >>>>>>>> > >>>>>>>> This right now only works on Linux x86_64, but getting it to > >>>>>>>> work for > >>>>>>>> other machines won't be too hard. I suspect we will be able > >>>>>>>> to > >>>>>>>> do > >>>>>>>> nightlies very easily as well. If anyone out there has an > >>>>>>>> x86_64 > >>>>>>>> machine they wouldn't mind trying it on, that would be very > >>>>>>>> helpful! > >>>>>>>> I did find that once I ran this script, I had to actually > >>>>>>>> prepend the > >>>>>>>> PATH afterwards as well. This means doing: > >>>>>>>> > >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH > >>>>>>>> export PATH=$HOME/yt-conda:$PATH > >>>>>>>> source activate ytenv > >>>>>>>> > >>>>>>>> At that point, everything was set up and working for me. The > >>>>>>>> miniconda install offers to add paths to .bashrc, but I don't > >>>>>>>> think > >>>>>>>> we > >>>>>>>> should go down that route. That being said, this is also a > >>>>>>>> possible > >>>>>>>> point of friction. > >>>>>>>> > >>>>>>>> One nice thing is that this also completely works with the > >>>>>>>> full > >>>>>>>> anaconda; if someone wants everything that is in the anaconda > >>>>>>>> install, > >>>>>>>> they can even simply do "conda install anaconda" from the > >>>>>>>> command > >>>>>>>> line > >>>>>>>> to get it. But the stripped down subset is the default. > >>>>>>>> > >>>>>>>> If anyone has a chance to try this out and has feedback, I'd > >>>>>>>> greatly > >>>>>>>> appreciate it! I think Nathan has done something very > >>>>>>>> similar > >>>>>>>> for > >>>>>>>> OSX. I've also put a couple simple conda recipes here: > >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use > >>>>>>>> as a > >>>>>>>> basis > >>>>>>>> for distributing builds and setting them up on buildbots and > >>>>>>>> the > >>>>>>>> like. > >>>>>>>> I'm pretty optimistic about this. > >>>>>>>> > >>>>>>>> -Matt > >>>>>>>> > >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum > >>>>>>>> <nathan12343@gmail.com> wrote: > >>>>>>>>> I think to get everything working in a sustainable fashion, > >>>>>>>>> we > >>>>>>>>> would need > >>>>>>>>> buildbots for all platform combinations that we want to > >>>>>>>>> support, so > >>>>>>>>> all > >>>>>>>>> permutations of the (32/64 bit, linux / OS X / windows, > >>>>>>>>> py27/py3.3) tuple. > >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit
> >>>>>>>>> 64 > >>>>>>>>> bit > >>>>>>>>> OS X > >>>>>>>>> (not totally clear if OS X version matters), and 32 and 64 > >>>>>>>>> bit > >>>>>>>>> windows. > >>>>>>>>> > >>>>>>>>> Another option is to rely on conda build, which compiles > >>>>>>>>> everything > >>>>>>>>> from > >>>>>>>>> source. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory < s@skory.us> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I have less of a skin in this than I used to, but I'd
On Tue, Sep 10, 2013 at 2:09 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote: tracker. that this this as think script. linux, like
> >>>>>>>>>> to > >>>>>>>>>> raise > >>>>>>>>>> the issue of Windows & package managers. For example, > >>>>>>>>>> Anaconda > >>>>>>>>>> is > >>>>>>>>>> available for Windows - would that mean that yt might "just > >>>>>>>>>> work" > >>>>>>>>>> on > >>>>>>>>>> Windows? Or the opposite, and it would require a great deal > >>>>>>>>>> of > >>>>>>>>>> effort > >>>>>>>>>> to get all the various things we expect to be .so's to work > >>>>>>>>>> as > >>>>>>>>>> .dll's > >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)? > >>>>>>>>>> > >>>>>>>>>> I don't know the answers to these questions, but I think > >>>>>>>>>> it's > >>>>>>>>>> worth > >>>>>>>>>> thinking about. > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Stephen Skory > >>>>>>>>>> s@skory.us > >>>>>>>>>> http://stephenskory.com/ > >>>>>>>>>> 510.621.3687 (google voice) > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> yt-dev mailing list > >>>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>>> > >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> yt-dev mailing list > >>>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> yt-dev mailing list > >>>>>>>> yt-dev@lists.spacepope.org > >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> yt-dev mailing list > >>>>>>> yt-dev@lists.spacepope.org > >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> yt-dev mailing list > >>>>>> yt-dev@lists.spacepope.org > >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> yt-dev mailing list > >>>>> yt-dev@lists.spacepope.org > >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-dev mailing list > >>>> yt-dev@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>> > >>> > >>> > >>> _______________________________________________ > >>> yt-dev mailing list > >>> yt-dev@lists.spacepope.org > >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>> > >> > >> > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >> > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org