Nathan,

I'm sorry. I was not at all clear. Thanks for pointing out that FFTW could have been built against the libm that conda provides. The problem I was trying to get at is that there are probably a lot of scenarios where we *don't* want to rely on whatever library conda provides but we *do* want to link that library into the yt python somehow. I was thinking specifically about deploying on supercomputing resources where, for example, we'd want mpi4py to interface NOT with whatever mpi library conda provides but with the one the supercomputing center provides. Another key example (perhaps not super relevant to yt at the moment) is linear algebra libraries. I guess what I was really asking is if there is a way to make conda talk to system libraries better than it does, or if that's simply not part of how it is intended to operate. From what I understand, conda is designed to deploy python and its scientific appendages easily on a given hardware resource; I would think this would include interfacing with provided high performance libraries, but perhaps that is just not part of their use case.

If it is the case that conda cannot help with this problem, I wonder how this is going to shake out for yt, since we need to *both* have an easy install for a diverse set of people where performance may not be driven by external libraries, but we also have a situation where it might: for example, we increasingly will want to do analysis on machines with somewhat unusual MPI implementations (kraken, for example). Relying on the centers to deploy these resources is one way of dealing with it, but perhaps not the best one given the rapid updates to yt.

j


On Tue, Oct 22, 2013 at 11:59 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
I believe the issue is that your FFTW wrappers are linked against your system libm while you should be linking then against the libraries provided by conda if you want to use it with that python install.

You might want to consider writing a conda recipe for your wrappers.  Building a conda package from such a recipe will ensure that it is linking with anaconda's libraries since conda package builds happen in an isolated environment.

I agree that there are issues on Linux with anaconda and binary compatibility and that it's not totally obvious to newcomers how to use anaconda python if they also want to build their own packages under it.  However, given that it simplifies the installation process and gives users an out-of-the box yt install with significantly less hassle than the install script and with very little maintenance on our end, I still think it's the way to go moving forward.

If you're going to be doing a lot of development, anaconda's python may not be the best choice.  Instead, you might consider building yt and all it's dependencies using the Ubuntu's system python and packages.

Cheers,

Nathan

On Tuesday, October 22, 2013, j s oishi wrote:
daikaiju:/data/../96_Ra10000 [10:39]$ python 2d_gmode_kx1_kz1.py                                                                                              
Traceback (most recent call last):
  File "2d_gmode_kx1_kz1.py", line 26, in <module>
    from dedalus.mods import *
  File "/home/jsoishi/hg-projects/dedalus/dedalus/mods.py", line 30, in <module>
    from dedalus.utils.logger import mylog
  File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/logger.py", line 30, in <module>
    from dedalus.utils.parallelism import com_sys
  File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py", line 52, in <module>
    com_sys = CommunicationSystem()
  File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py", line 43, in __init__
    from dedalus.utils.fftw import fftw
ImportError: /home/jsoishi/build/yt-conda/bin/../lib/libm.so.6: version `GLIBC_2.15' not found (required by /home/jsoishi/build/lib/libfftw3.so.3)



On Tue, Oct 22, 2013 at 11:39 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Jeff,

Can you paste the error messages you're seeing?

Nathan


On Tuesday, October 22, 2013, j s oishi wrote:
Hi all,

So I've been playing with the new conda installed yt. In principle, I really like the idea: a simple, completely isolated python distribution that does dependency management. However, I'm running into some problems that are potentially big problems for yt down the road. My issue is that I'm trying to install Dedalus into the yt-provided conda install. However, to do that, I need to interface with external libraries--ones that I do *not* want to have built as conda packages. The two that come up first are MPI and FFTW. Since the Dedalus FFTW bindings are actually used in yt for its FFT analysis, this is relevant for yt too.

On a supercomputing center, one would very much want to use their MPI and FFT libraries, rather than some binary blob from conda, I think we need to figure out how to do this. The problem I'm getting specifically is that if I build FFTW (for example) myself, I want to link against the system libm. But conda provides its OWN libm, which, it turns out is not GLIBC compatible with what FFTW was built against. I believe this is because libm from conda was not built against the same glibc as my libm, which FFTW was built against. I don't know how to solve this problem, since I don't really understand dynamic libraries, nor conda. Does anyone have any ideas about this? Is this something I should take up with the conda list?

My apologies if this is unclear. I am not fully sure I understand what is happening myself.

thanks,

j

_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org