Hi Mateusz Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder. By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days... Personally, I set up my python environment on Pleiades using miniconda. -Nathan On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt warning: bitbucket.org certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

does pip come with miniconda? you might need to install that separately. i think it comes with the full anaconda. (fwiw, i installed yt on pleiades by using full anaconda, doing the pip install -U cython, then the hg clone yt etc. that you do). On Mon, Oct 19, 2015 at 4:42 PM, Mateusz Ruszkowski wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt hg clone https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski wrote:

...

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt warning: bitbucket.org certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Nathan/Desika, Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt" with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series: bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda install yt [instead of: miniconda3/bin/conda build conda-recipes/yt] miniconda3/bin/conda install pip hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hg up yt python setup.py develop Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2” I also do “module load python_2.7.2” The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up? Thanks, Mateusz

On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt

Here you want to do "conda install yt" to get all of yt's dependencies. From there you can install the development version of yt. You can "conda install" must python packages.

As desika mentioned, you can also install pip with "conda install pip".

I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.

hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

...

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu mailto:mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt warning: bitbucket.org http://bitbucket.org/ certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

The reason I originally loaded python_2.7.2 was to avoid the error message below. Now that python_2.7.2 is unloaded this message is back ... % module unload python_2.7.2 % python setup.py develop yt currently requires Python version 2.7 certain features may fail unexpectedly and silently with older versions. % As the next step I will try to repeat the whole process from scratch w/o using module load pkgsrc/2015Q2. Instead, I will just use standalone mercurial installation to see if this helps.

On Oct 19, 2015, at 7:31 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 4:11 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan/Desika,

Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt" with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda install yt [instead of: miniconda3/bin/conda build conda-recipes/yt] miniconda3/bin/conda install pip hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hg up yt python setup.py develop

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2” I also do “module load python_2.7.2” The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up?

Yes, that is possible.

You want to make sure that you are running the miniconda python and not the python loaded by the module. You can check by doing "which python". If you're using miniconda, you want to unload the python_2.7.2 module.

Thanks, Mateusz

...

On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt

Here you want to do "conda install yt" to get all of yt's dependencies. From there you can install the development version of yt. You can "conda install" must python packages.

As desika mentioned, you can also install pip with "conda install pip".

I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.

hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

...

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu mailto:mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt warning: bitbucket.org http://bitbucket.org/ certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

You were right that I didn’t have python from miniconda in my PATH and so I was picking up the (old) system version of python. I prepended PATH variable and “which python” now points to the right version of python from miniconda. Nevertheless, the problem remains as illustrated below. I am puzzled why this is happening ... % pwd /nobackupnfs2/mruszkow % ls Miniconda3-latest-Linux-x86_64.sh % bash Miniconda3-latest-Linux-x86_64.sh % setenv PATH /nobackupnfs2/mruszkow/miniconda3/bin:${PATH} % echo $PATH % which python % conda install yt % module load pkgsrc/2015Q2 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt % ls Miniconda3-latest-Linux-x86_64.sh miniconda3 yt % pwd /nobackupnfs2/mruszkow % cd yt/ % hg up yt % which python /nobackupnfs2/mruszkow/miniconda3/bin/python % python setup.py develop Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

On Oct 19, 2015, at 7:53 PM, Nathan Goldbaum wrote:

You don't have the python from miniconda in your PATH and are picking up a version of python 2.6 or older from the operating system on Pleiades.

What's the output of "which python"? If the path returned by that command doesn't have "miniconda" in it, it's the wrong python executable. You might need to manually add miniconda's bin directory to your PATH environment variable.

On Mon, Oct 19, 2015 at 4:51 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

The reason I originally loaded python_2.7.2 was to avoid the error message below. Now that python_2.7.2 is unloaded this message is back ...

% module unload python_2.7.2 % python setup.py develop yt currently requires Python version 2.7 certain features may fail unexpectedly and silently with older versions. %

As the next step I will try to repeat the whole process from scratch w/o using module load pkgsrc/2015Q2. Instead, I will just use standalone mercurial installation to see if this helps.

...

On Oct 19, 2015, at 7:31 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 4:11 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan/Desika,

Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt" with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda install yt [instead of: miniconda3/bin/conda build conda-recipes/yt] miniconda3/bin/conda install pip hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hg up yt python setup.py develop

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2” I also do “module load python_2.7.2” The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up?

Yes, that is possible.

You want to make sure that you are running the miniconda python and not the python loaded by the module. You can check by doing "which python". If you're using miniconda, you want to unload the python_2.7.2 module.

Thanks, Mateusz

...

On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt

Here you want to do "conda install yt" to get all of yt's dependencies. From there you can install the development version of yt. You can "conda install" must python packages.

As desika mentioned, you can also install pip with "conda install pip".

I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.

hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

...

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu mailto:mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt warning: bitbucket.org http://bitbucket.org/ certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Good call. It appears that cython was not included in miniconda3 (and was not present under miniconda3/bin/): % conda update cython Error: package 'cython' is not installed in /nobackupnfs2/mruszkow/miniconda3 % python test.py Traceback (most recent call last): File "test.py", line 1, in <module> import cython ImportError: No module named ‘python' So I ran: conda install cython [ now miniconda3/bin/ does include cython ] % python test.py 0.23.4 /nobackupnfs2/mruszkow/miniconda3/lib/python3.4/site-packages/cython.py cd yt python setup.py develop [ appears to have ran properly and, after a long list of warnings, I got: ] [….] Adding yt 3.3.dev0 to easy-install.pth file Installing yt script to /nobackupnfs2/mruszkow/miniconda3/bin Installing iyt script to /nobackupnfs2/mruszkow/miniconda3/bin Installed /nobackupnfs2/mruszkow/yt Processing dependencies for yt==3.3.dev0 Finished processing dependencies for yt==3.3.dev0 % So, how do I activate this installation?

On Oct 19, 2015, at 10:06 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 6:52 PM, Mateusz Ruszkowski wrote:

You were right that I didn’t have python from miniconda in my PATH and so I was picking up the (old) system version of python. I prepended PATH variable and “which python” now points to the right version of python from miniconda. Nevertheless, the problem remains as illustrated below. I am puzzled why this is happening ...

% pwd /nobackupnfs2/mruszkow % ls Miniconda3-latest-Linux-x86_64.sh % bash Miniconda3-latest-Linux-x86_64.sh % setenv PATH /nobackupnfs2/mruszkow/miniconda3/bin:${PATH} % echo $PATH % which python % conda install yt % module load pkgsrc/2015Q2 % hg clone https://bitbucket.org/yt_analysis/yt % ls Miniconda3-latest-Linux-x86_64.sh miniconda3 yt % pwd /nobackupnfs2/mruszkow % cd yt/ % hg up yt % which python /nobackupnfs2/mruszkow/miniconda3/bin/python % python setup.py develop Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Does it persist if you do "conda update cython"? There should be builds of Cython 0.23.4 on linux.

If it does persist, can you paste the output of

import cython print(cython.__version__) print(cython.__file__)

Using the miniconda python.

Sorry this is so complicated, unfortunately this is wrapped up with the general awfulness of packaging in python and one of several things could be going wrong.

...

On Oct 19, 2015, at 7:53 PM, Nathan Goldbaum wrote:

You don't have the python from miniconda in your PATH and are picking up a version of python 2.6 or older from the operating system on Pleiades.

What's the output of "which python"? If the path returned by that command doesn't have "miniconda" in it, it's the wrong python executable. You might need to manually add miniconda's bin directory to your PATH environment variable.

On Mon, Oct 19, 2015 at 4:51 PM, Mateusz Ruszkowski wrote:

The reason I originally loaded python_2.7.2 was to avoid the error message below. Now that python_2.7.2 is unloaded this message is back ...

% module unload python_2.7.2 % python setup.py develop yt currently requires Python version 2.7 certain features may fail unexpectedly and silently with older versions. %

As the next step I will try to repeat the whole process from scratch w/o using module load pkgsrc/2015Q2. Instead, I will just use standalone mercurial installation to see if this helps.

...

On Oct 19, 2015, at 7:31 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 4:11 PM, Mateusz Ruszkowski wrote:

Hi Nathan/Desika,

Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt" with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda install yt [instead of: miniconda3/bin/conda build conda-recipes/yt] miniconda3/bin/conda install pip hg clone https://bitbucket.org/yt_analysis/yt cd yt/ hg up yt python setup.py develop

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2” I also do “module load python_2.7.2” The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up?

Yes, that is possible.

You want to make sure that you are running the miniconda python and not the python loaded by the module. You can check by doing "which python". If you're using miniconda, you want to unload the python_2.7.2 module.

Thanks, Mateusz

...

On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt

Here you want to do "conda install yt" to get all of yt's dependencies. From there you can install the development version of yt. You can "conda install" must python packages.

As desika mentioned, you can also install pip with "conda install pip".

I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.

hg clone https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

...

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt warning: bitbucket.org certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

It works :) Thanks so much for your help!

On Oct 19, 2015, at 11:48 PM, Nathan Goldbaum wrote:

On Mon, Oct 19, 2015 at 8:43 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Good call. It appears that cython was not included in miniconda3 (and was not present under miniconda3/bin/):

% conda update cython Error: package 'cython' is not installed in /nobackupnfs2/mruszkow/miniconda3 % python test.py Traceback (most recent call last): File "test.py", line 1, in <module> import cython ImportError: No module named ‘python'

So I ran:

conda install cython [ now miniconda3/bin/ does include cython ] % python test.py 0.23.4 /nobackupnfs2/mruszkow/miniconda3/lib/python3.4/site-packages/cython.py

cd yt python setup.py develop

[ appears to have ran properly and, after a long list of warnings, I got: ]

[….] Adding yt 3.3.dev0 to easy-install.pth file Installing yt script to /nobackupnfs2/mruszkow/miniconda3/bin Installing iyt script to /nobackupnfs2/mruszkow/miniconda3/bin

Installed /nobackupnfs2/mruszkow/yt Processing dependencies for yt==3.3.dev0 Finished processing dependencies for yt==3.3.dev0 %

So, how do I activate this installation?

No need to activate, just make sure miniconda's python is in your path. You should be able to "import yt" in miniconda's python right now.

...

On Oct 19, 2015, at 10:06 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 6:52 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

You were right that I didn’t have python from miniconda in my PATH and so I was picking up the (old) system version of python. I prepended PATH variable and “which python” now points to the right version of python from miniconda. Nevertheless, the problem remains as illustrated below. I am puzzled why this is happening ...

% pwd /nobackupnfs2/mruszkow % ls Miniconda3-latest-Linux-x86_64.sh % bash Miniconda3-latest-Linux-x86_64.sh % setenv PATH /nobackupnfs2/mruszkow/miniconda3/bin:${PATH} % echo $PATH % which python % conda install yt % module load pkgsrc/2015Q2 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt % ls Miniconda3-latest-Linux-x86_64.sh miniconda3 yt % pwd /nobackupnfs2/mruszkow % cd yt/ % hg up yt % which python /nobackupnfs2/mruszkow/miniconda3/bin/python % python setup.py develop Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Does it persist if you do "conda update cython"? There should be builds of Cython 0.23.4 on linux.

If it does persist, can you paste the output of

import cython print(cython.__version__) print(cython.__file__)

Using the miniconda python.

Sorry this is so complicated, unfortunately this is wrapped up with the general awfulness of packaging in python and one of several things could be going wrong.

...

On Oct 19, 2015, at 7:53 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

You don't have the python from miniconda in your PATH and are picking up a version of python 2.6 or older from the operating system on Pleiades.

What's the output of "which python"? If the path returned by that command doesn't have "miniconda" in it, it's the wrong python executable. You might need to manually add miniconda's bin directory to your PATH environment variable.

On Mon, Oct 19, 2015 at 4:51 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

The reason I originally loaded python_2.7.2 was to avoid the error message below. Now that python_2.7.2 is unloaded this message is back ...

% module unload python_2.7.2 % python setup.py develop yt currently requires Python version 2.7 certain features may fail unexpectedly and silently with older versions. %

As the next step I will try to repeat the whole process from scratch w/o using module load pkgsrc/2015Q2. Instead, I will just use standalone mercurial installation to see if this helps.

...

On Oct 19, 2015, at 7:31 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 4:11 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan/Desika,

Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt" with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda install yt [instead of: miniconda3/bin/conda build conda-recipes/yt] miniconda3/bin/conda install pip hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hg up yt python setup.py develop

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2” I also do “module load python_2.7.2” The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up?

Yes, that is possible.

You want to make sure that you are running the miniconda python and not the python loaded by the module. You can check by doing "which python". If you're using miniconda, you want to unload the python_2.7.2 module.

Thanks, Mateusz

...

On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi Nathan,

Thanks for your help. I think also this almost works ... I downloaded the latest miniconda and followed this procedure:

bash Miniconda3-latest-Linux-x86_64.sh miniconda3/bin/conda build conda-recipes/yt

Here you want to do "conda install yt" to get all of yt's dependencies. From there you can install the development version of yt. You can "conda install" must python packages.

As desika mentioned, you can also install pip with "conda install pip".

I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.

hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt cd yt/ hy up yt python setup.py develop

This resulted in:

Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython

which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?

Thanks, Mateusz

...

On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum mailto:nathan12343@gmail.com> wrote:

Hi Mateusz

Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.

By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...

Personally, I set up my python environment on Pleiades using miniconda.

-Nathan

On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski mailto:mateuszr@umich.edu> wrote:

Hi,

I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation?

Thanks, Mateusz

------------------------------------------------ Dr. Mateusz Ruszkowski Associate Professor

University of Michigan Department of Astronomy 301C West Hall 1085 South University Ave. Ann Arbor, MI 48109-1107, USA

e-mail: mateuszr@umich.edu mailto:mateuszr@umich.edu

————————————————————————

% module load pkgsrc/2014Q3 % module load python/2.7.9 % hg clone https://bitbucket.org/yt_analysis/yt https://bitbucket.org/yt_analysis/yt warning: bitbucket.org http://bitbucket.org/ certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting) destination directory: yt requesting all changes adding changesets adding manifests adding file changes added 17568 changesets with 36510 changes to 3053 files (+3 heads) updating to bookmark @ on branch stable 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved % cd yt % hg update yt 317 files updated, 0 files merged, 148 files removed, 0 files unresolved (leaving bookmark @) % python setup.py install --user --prefix=/nobackupnfs2/mruszkow Cython is a build-time requirement for the source tree of yt. Please either install yt from a provided, release tarball, or install Cython (version 0.22 or higher). You may be able to accomplish this by typing: pip install -U Cython % pip install -U Cython Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=15... Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython Unable to find pgen, not compiling formal grammar.

warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests' warning: no files found matching '*.pxd' under directory 'Cython/Utility' Installing collected packages: Cython Found existing installation: Cython 0.21.1 Uninstalling Cython: Cleaning up... Exception: Traceback (most recent call last): File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install requirement.uninstall(auto_confirm=True) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall paths_to_remove.remove(auto_confirm) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove renames(path, new_path) File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames shutil.move(old, new) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree os.remove(fullname) OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'

Storing debug log for failure in /u/mruszkow/.pip/pip.log %

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org