Hi Christine, On Thu, Apr 11, 2013 at 7:29 PM, Christine Simpson <csimpson@astro.columbia.edu> wrote:
Thanks so much for your help!
So I did some tests. I ran each test on 3 processors for 10 cycles and set PythonTopGridSkip = 1. The hierarchy is static (so no amr).
Okay, that's good to hear -- it eliminates some common problems.
type 1:
import numpy as np
def main(): print "Type 1"
This worked. Type 1 printed a total of 60 times, which is ncycles*nproc*2 (for the 2 calls to python, one at the end of evolve level and the one at the end of evolve hierarchy).
type 2:
from yt.mods import *
def main: print "Type 2"
This failed. The error I get is
Successfully read in parameter file StarParticleTest.enzo. INITIALIZATION TIME = 4.38757896e-01 yt : [INFO ] 2013-04-11 13:14:08,924 Global parallel computation enabled: 1 / 3 yt : [INFO ] 2013-04-11 13:14:08,924 Global parallel computation enabled: 2 / 3 yt : [INFO ] 2013-04-11 13:14:08,924 Global parallel computation enabled: 0 / 3 MPI process (rank: 0) terminated unexpectedly on trestles-1-10.local Exit code -5 signaled from trestles-1-10
Awesome. This tells us a *lot* of information. For starters, importing numpy and having that fail is a very common use case that can be difficult to fix. Fortunately, that's not the case here! :) So here's a good start -- in the directory ~/.yt you may or may not have a file named 'config'. If you add to that file something like: [yt] loglevel: 1 then you can get a lot more info about how things proceed. For me it looks like this: http://paste.yt-project.org/show/3357/ The next step is to put print statements in yt/mods.py around these areas: http://paste.yt-project.org/show/3358/ Speciffically, I'm curious if it gets past startup tasks. After that, there is a series of import blocks that look like: from yt.something \ import flux_capacitor or something. Each of these blocks might be grabbing something. Now, one very *last* thing is, can you run one last script identical to the ones before, except test to see if h5py can be imported? It occurs to me that's another source of error that might be pernicious. I'm *really* sorry this is so annoying -- getting this up and running is a huge priority for me! For what it's worth, part of the issue here is that yt+sim code have to be in the same executable; the next generation of in situ stuff will hopefully avoid this, and thus make problems like this ... erased. But that's a year or more off. Anyhow, I'm really interested in getting this working for you, so please let me know if any of that helps at all -- the unexplained error (-5) doesn't help narrow it down too much, unfortunately. Thanks again, Matt
type 3:
type 1:
from mpi4py import MPI
def main(): print "Type 3"
This works and prints out Type 3 a total of 60 times as in case 1.
One last thing to check is the output of ldd on the MPI.so file inside your yt site-packages directory, under the mpi4py subdirectory, versus the output of ldd for enzo.exe. These should both link to the same MPI library.
I'm not sure what the MPI library is called, but all the directory files listed for MPI.so from the yt site-packages directory and for the enzo executable look the same. They are pasted below.
So it looks like it is something with yt.mods/yt.pmods? (I wasn't quite sure which one to use, by the way). So I can try putting some print statements in that file to see how far it gets. Are there likely problem places that I should focus on?
Thanks Christine
For MPI.so in the yt site-packages directory (yt-x86_64-shared)[csimpson@trestles-login1 mpi4py]$ ldd MPI.so libpython2.7.so.1.0 => /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0 (0x00002b9930541000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b993091d000) librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x00002b9930b39000) libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002b9930d3e000) libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b9930f4b000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b9931152000) librt.so.1 => /lib64/librt.so.1 (0x00002b9931356000) libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so (0x00002b993155f000) libsvml.so => /opt/intel/Compiler/11.1/072/lib/intel64/libsvml.so (0x00002b99318f4000) libm.so.6 => /lib64/libm.so.6 (0x00002b9931b0a000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b9931d8d000) libintlc.so.5 => /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5 (0x00002b9931f9c000) libc.so.6 => /lib64/libc.so.6 (0x00002b99320da000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b9932431000) /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)
For enzo.exe: (yt-x86_64-shared)[csimpson@trestles-login1 mpi4py]$ ldd ~/temp_mom/enzo-dev-mom/src/enzo/enzo.exe libhdf5.so.7 => /home/csimpson/yt-x86_64-shared/lib/libhdf5.so.7 (0x00002b9496c96000) libifcore.so.5 => /opt/intel/Compiler/11.1/072/lib/intel64/libifcore.so.5 (0x00002b9497139000) libifport.so.5 => /opt/intel/Compiler/11.1/072/lib/intel64/libifport.so.5 (0x00002b94973b1000) libpython2.7.so.1.0 => /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0 (0x00002b94974ea000) libreadline.so.5 => /usr/lib64/libreadline.so.5 (0x0000003d96c00000) libtermcap.so.2 => /lib64/libtermcap.so.2 (0x000000346bc00000) libutil.so.1 => /lib64/libutil.so.1 (0x0000003452200000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000000344ca00000) librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x000000344ce00000) libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x000000344c200000) libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b94978c8000) libdl.so.2 => /lib64/libdl.so.2 (0x000000344c600000) librt.so.1 => /lib64/librt.so.1 (0x000000344da00000) libm.so.6 => /lib64/libm.so.6 (0x00002b9497acf000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000035f8a00000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000035f8e00000) libc.so.6 => /lib64/libc.so.6 (0x000000344be00000) libz.so.1 => /home/csimpson/yt-x86_64-shared/lib/libz.so.1 (0x00002b9497d53000) libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so (0x00002b9497f6a000) libintlc.so.5 => /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5 (0x00002b94982fe000) /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org