Hello yt people, I'm trying to install current yt-3.3dev on Blue Waters at NCSA, using the install_script.sh route. After hacking around the fact that it has an obsolete & unsafe openssl library as the default (I needed to pass LDFLAGS/CPPFLAGS to the python build to set the -L and -rpath to the non-default usable openssl), I'm having lots of trouble getting other component installs to work. The yt_install.log has things like Processing dependencies for matplotlib==1.4.3 Searching for pytz Reading https://pypi.python.org/simple/pytz/ Download error on https://pypi.python.org/simple/pytz/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! Couldn't find index page for 'pytz' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! No local packages or download links found for pytz error: Could not find suitable distribution for Requirement.parse('pytz') Yet when I run what seems like the same install by hand ($DIR/bin/pip -v install pytz) it works fine, without error. I can then re-run install_script.sh, until it reaches the next dependency and fails, etc. Each cycle takes 5-10 minutes so it's pretty tedious. ***=> I don't understand why the internal-to-matplotlib-setup call to pip fails, while it succeeds when I run it by hand. A suspicion: could the CERTIFICATE_VERIFY_FAILED errors be hinting that Blue Waters has an https gateway to the outside world that rewrites SSL certificates? I don't know whether that's true. This page suggested a way to grab the needed root certificate and point env var PIP_CERT at it... http://stackoverflow.com/questions/25981703/pip-install-fails-with-connectio... but trying that didn't help in this case. The yt-installed "pip" works fine when run by hand, but whatever matplotlib's setup.py is running still fails. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Another option that might be easier than all this pain would be to build a Python stack based on something older than 2.7.9. Given all that you've gone through, I think you should be able to hack the install script to do this based on tarballs available from Python.org. On other clusters I've used miniconda, which bundles its own OpenSSL library. I have no idea if that would work on a cray. On Friday, January 8, 2016, Levy, Stuart A <salevy@illinois.edu> wrote:
Hello yt people,
I'm trying to install current yt-3.3dev on Blue Waters at NCSA, using the install_script.sh route.
After hacking around the fact that it has an obsolete & unsafe openssl library as the default (I needed to pass LDFLAGS/CPPFLAGS to the python build to set the -L and -rpath to the non-default usable openssl), I'm having lots of trouble getting other component installs to work. The yt_install.log has things like
Processing dependencies for matplotlib==1.4.3 Searching for pytz Reading https://pypi.python.org/simple/pytz/ Download error on https://pypi.python.org/simple/pytz/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! Couldn't find index page for 'pytz' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! No local packages or download links found for pytz error: Could not find suitable distribution for Requirement.parse('pytz')
Yet when I run what seems like the same install by hand ($DIR/bin/pip -v install pytz) it works fine, without error.
I can then re-run install_script.sh, until it reaches the next dependency and fails, etc. Each cycle takes 5-10 minutes so it's pretty tedious.
***=> I don't understand why the internal-to-matplotlib-setup call to pip fails, while it succeeds when I run it by hand.
A suspicion: could the CERTIFICATE_VERIFY_FAILED errors be hinting that Blue Waters has an https gateway to the outside world that rewrites SSL certificates? I don't know whether that's true. This page suggested a way to grab the needed root certificate and point env var PIP_CERT at it...
http://stackoverflow.com/questions/25981703/pip-install-fails-with-connectio... but trying that didn't help in this case. The yt-installed "pip" works fine when run by hand, but whatever matplotlib's setup.py is running still fails.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Stuart, I ran into exactly this problem, and ultimately managed to get yt-dev installed (and working in parallel) using Anaconda. Instructions are attached; hopefully they'll work for you. --Brian ----------------------------------------------------------------------- Getting yt-3 to work on Blue Waters: 0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later. 1) Install the Anaconda python distribution ( https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path. 2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz). 3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file: [cray] mpicc = cc mpicxx = CC extra_link_args = -shared then, in the top-level mpi4py directory, type: python setup.py build --mpi=cray python setp.py install At this point, you should have a functional version of mpi4py that uses the cray MPI libraries. 4) You can then install yt via source in your home directory: hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.) 5) and, optionally, if you want to install Rockstar using yt from source, you do the following: hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so). On Fri, Jan 8, 2016 at 9:30 PM, Levy, Stuart A <salevy@illinois.edu> wrote:
Hello yt people,
I'm trying to install current yt-3.3dev on Blue Waters at NCSA, using the install_script.sh route.
After hacking around the fact that it has an obsolete & unsafe openssl library as the default (I needed to pass LDFLAGS/CPPFLAGS to the python build to set the -L and -rpath to the non-default usable openssl), I'm having lots of trouble getting other component installs to work. The yt_install.log has things like
Processing dependencies for matplotlib==1.4.3 Searching for pytz Reading https://pypi.python.org/simple/pytz/ Download error on https://pypi.python.org/simple/pytz/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! Couldn't find index page for 'pytz' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.python.org/simple/ Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some packages may not be found! No local packages or download links found for pytz error: Could not find suitable distribution for Requirement.parse('pytz')
Yet when I run what seems like the same install by hand ($DIR/bin/pip -v install pytz) it works fine, without error.
I can then re-run install_script.sh, until it reaches the next dependency and fails, etc. Each cycle takes 5-10 minutes so it's pretty tedious.
***=> I don't understand why the internal-to-matplotlib-setup call to pip fails, while it succeeds when I run it by hand.
A suspicion: could the CERTIFICATE_VERIFY_FAILED errors be hinting that Blue Waters has an https gateway to the outside world that rewrites SSL certificates? I don't know whether that's true. This page suggested a way to grab the needed root certificate and point env var PIP_CERT at it...
http://stackoverflow.com/questions/25981703/pip-install-fails-with-connectio... but trying that didn't help in this case. The yt-installed "pip" works fine when run by hand, but whatever matplotlib's setup.py is running still fails.
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Brian O'Shea -
Levy, Stuart A -
Nathan Goldbaum