Having trouble running yt on kraken
Hi all, I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I have on kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file /ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0 aborting job: (null) I can run a non-parallel interactive job (./try1.py) without any problem. My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20) Thanks! Yuan
Hi Yuan, This could be a result of using different version of yt on the two different platforms. Can you check and see which version of yt you're using on kraken? I believe there are modules for versions 2.2, 2.3, and the development branch on kraken. Also, the variable "slice" is already used in python for something else. If slice is apart of your script, you may want to rename that to something else. If your script only works on the most recent version of yt, you'll probably want to try to use the yt/dev module on Kraken. Britton On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi all,
I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I have on kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments
If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file /ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0 aborting job: (null)
I can run a non-parallel interactive job (./try1.py) without any problem.
My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20)
Thanks! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Britton, Thanks for your suggestion! Now the problem has been semi-solved. I have checked my script and I have this "module load yt/dev/", so I guess I am using yt/dev, which is what I use on other machines. There is indeed a problem with "slice". I had all my functions defined inside one file called profile.py (my bad: I started with profiles and kept adding things without changing the name of it). After I changed that file to "yuan.py", the error I got was: ...... Traceback (most recent call last): File "/var/spool/alps/2362242/try1.py", line 3, in <module> import yuan as yuan ImportError: No module named yuan ...... So Kraken did not see my file. It did not complain before because there was a built-in function in python called profile. Now after I added this to my script: import sys sys.path.append('/lustre/scratch/yuan/Thermal09/') Kraken could finally import my functions! The reason why I said that it was only semi-solved is because now I am getting "MemoryError". I will try to use more nodes, but it may never work :). How many nodes do you use for each job, Britton? Thanks! Yuan On Thu, Jul 26, 2012 at 6:39 PM, Britton Smith <brittonsmith@gmail.com>wrote:
Hi Yuan,
This could be a result of using different version of yt on the two different platforms. Can you check and see which version of yt you're using on kraken? I believe there are modules for versions 2.2, 2.3, and the development branch on kraken.
Also, the variable "slice" is already used in python for something else. If slice is apart of your script, you may want to rename that to something else.
If your script only works on the most recent version of yt, you'll probably want to try to use the yt/dev module on Kraken.
Britton
On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi all,
I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I have on kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments
If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file /ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0 aborting job: (null)
I can run a non-parallel interactive job (./try1.py) without any problem.
My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20)
Thanks! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Yuan, On Sat, Jul 28, 2012 at 6:41 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi Britton,
Thanks for your suggestion! Now the problem has been semi-solved.
I have checked my script and I have this "module load yt/dev/", so I guess I am using yt/dev, which is what I use on other machines.
There is indeed a problem with "slice". I had all my functions defined inside one file called profile.py (my bad: I started with profiles and kept adding things without changing the name of it). After I changed that file to "yuan.py", the error I got was: ......
Traceback (most recent call last): File "/var/spool/alps/2362242/try1.py", line 3, in <module> import yuan as yuan ImportError: No module named yuan ......
So Kraken did not see my file. It did not complain before because there was a built-in function in python called profile.
Now after I added this to my script: import sys sys.path.append('/lustre/scratch/yuan/Thermal09/')
Kraken could finally import my functions!
The reason why I said that it was only semi-solved is because now I am getting "MemoryError". I will try to use more nodes, but it may never work :).
What operation are you doing, and how big is your dataset? And, does your data have multiple root level tiles? -Matt
How many nodes do you use for each job, Britton?
Thanks! Yuan
On Thu, Jul 26, 2012 at 6:39 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Yuan,
This could be a result of using different version of yt on the two different platforms. Can you check and see which version of yt you're using on kraken? I believe there are modules for versions 2.2, 2.3, and the development branch on kraken.
Also, the variable "slice" is already used in python for something else. If slice is apart of your script, you may want to rename that to something else.
If your script only works on the most recent version of yt, you'll probably want to try to use the yt/dev module on Kraken.
Britton
On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi all,
I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I have on kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments
If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file /ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0 aborting job: (null)
I can run a non-parallel interactive job (./try1.py) without any problem.
My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20)
Thanks! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matt, I was making thin projections. Each output dataset (DD****) is about 14G. What are "multiple root level tiles"? I have 256^3 root grids... I just submitted another job asking for more nodes. I will let you know how that goes (in two days). Thanks! Yuan On Sat, Jul 28, 2012 at 7:24 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Yuan,
On Sat, Jul 28, 2012 at 6:41 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi Britton,
Thanks for your suggestion! Now the problem has been semi-solved.
I have checked my script and I have this "module load yt/dev/", so I guess I am using yt/dev, which is what I use on other machines.
There is indeed a problem with "slice". I had all my functions defined inside one file called profile.py (my bad: I started with profiles and kept adding things without changing the name of it). After I changed that file to "yuan.py", the error I got was: ......
Traceback (most recent call last): File "/var/spool/alps/2362242/try1.py", line 3, in <module> import yuan as yuan ImportError: No module named yuan ......
So Kraken did not see my file. It did not complain before because there was a built-in function in python called profile.
Now after I added this to my script: import sys sys.path.append('/lustre/scratch/yuan/Thermal09/')
Kraken could finally import my functions!
The reason why I said that it was only semi-solved is because now I am getting "MemoryError". I will try to use more nodes, but it may never work :).
What operation are you doing, and how big is your dataset? And, does your data have multiple root level tiles?
-Matt
How many nodes do you use for each job, Britton?
Thanks! Yuan
On Thu, Jul 26, 2012 at 6:39 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Yuan,
This could be a result of using different version of yt on the two different platforms. Can you check and see which version of yt you're
on kraken? I believe there are modules for versions 2.2, 2.3, and the development branch on kraken.
Also, the variable "slice" is already used in python for something else. If slice is apart of your script, you may want to rename that to something else.
If your script only works on the most recent version of yt, you'll probably want to try to use the yt/dev module on Kraken.
Britton
On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi all,
I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I
have on
kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments
If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file
/ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c
at line 115: 0 aborting job: (null)
I can run a non-parallel interactive job (./try1.py) without any
using problem.
My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20)
Thanks! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Yuan, On Sat, Jul 28, 2012 at 8:22 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi Matt,
I was making thin projections. Each output dataset (DD****) is about 14G. What are "multiple root level tiles"? I have 256^3 root grids...
Hm, you should be able to project no problem, even in serial on a laptop. I think the problem might be in an option that we recently changed, which was over-allocated memory for operations like these. We're putting out a release very soon (over on yt-dev we're even talking about drafting the release announcement!) which would be a good time to upgrade. The issue that was happening in the past was that all grids for a given data source were being loaded at initializatio nof the projection; plus, recent fixes to how the projection was reduced in parallel should reduce peak memory usage but keep it also steady at that peak. This also sounds like a good use case for a recipe in the cookbook that's going out with the documentation in the new release. Would you mind sending me a description or script (off-list) that I could convert to use the new stuff in the next release, which should help out with memory? We've done a lot to improve analysis of many outputs at once, and in particularly for doing this in parallel. I'd love to add this as a worked example. Here are a few using the new features in 2.4 (which are all in the development branch -- everything's developed in the open) that might show how we might be able to reduce memory overhead and speed up overall computation time. http://yt-project.org/docs/2.4/cookbook/calculating_information.html#time-se... http://yt-project.org/docs/2.4/advanced/parallel_computation.html#an-advance...
I just submitted another job asking for more nodes. I will let you know how that goes (in two days).
Cool. :) -Matt
Thanks! Yuan
On Sat, Jul 28, 2012 at 7:24 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Yuan,
On Sat, Jul 28, 2012 at 6:41 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi Britton,
Thanks for your suggestion! Now the problem has been semi-solved.
I have checked my script and I have this "module load yt/dev/", so I guess I am using yt/dev, which is what I use on other machines.
There is indeed a problem with "slice". I had all my functions defined inside one file called profile.py (my bad: I started with profiles and kept adding things without changing the name of it). After I changed that file to "yuan.py", the error I got was: ......
Traceback (most recent call last): File "/var/spool/alps/2362242/try1.py", line 3, in <module> import yuan as yuan ImportError: No module named yuan ......
So Kraken did not see my file. It did not complain before because there was a built-in function in python called profile.
Now after I added this to my script: import sys sys.path.append('/lustre/scratch/yuan/Thermal09/')
Kraken could finally import my functions!
The reason why I said that it was only semi-solved is because now I am getting "MemoryError". I will try to use more nodes, but it may never work :).
What operation are you doing, and how big is your dataset? And, does your data have multiple root level tiles?
-Matt
How many nodes do you use for each job, Britton?
Thanks! Yuan
On Thu, Jul 26, 2012 at 6:39 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Hi Yuan,
This could be a result of using different version of yt on the two different platforms. Can you check and see which version of yt you're using on kraken? I believe there are modules for versions 2.2, 2.3, and the development branch on kraken.
Also, the variable "slice" is already used in python for something else. If slice is apart of your script, you may want to rename that to something else.
If your script only works on the most recent version of yt, you'll probably want to try to use the yt/dev module on Kraken.
Britton
On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi all,
I have been having trouble running yt on kraken. I am using the same script that I use on trestles (where it sort of works). The issue I have on kraken is that when I submit a non-interactive job, I get this error: + aprun -n 12 ./try1.py --parallel Traceback (most recent call last): File "/var/spool/alps/2310447/try1.py", line 3, in <module> slice(imin=12, imax=47, width=20) TypeError: slice() does not take keyword arguments
If I run an interactive job (./try1.py --parallel), I get this error: yuan@krakenpf2:~/scratch/Thermal09> Assertion failed in file
/ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c at line 115: 0 aborting job: (null)
I can run a non-parallel interactive job (./try1.py) without any problem.
My script looks like this: #!/lustre/scratch/proj/sw/yt/dev/bin/python from profile import * slice(imin=41, imax=47, width=20)
Thanks! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Britton Smith -
Matthew Turk -
Yuan Li