MemoryError when using BinnedProfile1D
Hi yt users, I have been having trouble using BinnedProfile1D. I am using the same script that seemed to be working before. Now it is giving me "MemoryError": http://paste.yt-project.org/show/A67caGKsv6mFqA83MZoZ/ I can still use it on a low resolution run without a problem. I am not sure why it needs so much memory (I am already asking for 30 gb of memory). If I do a similar analysis with plotcollection and add_profile_sphere for the same data, I do not need to ask for much memory at all. I thought BinnedProfile1D would need less memory than add_profile_sphere. Please let me know if I was wrong and the job simply requires a lot of memory. What I am trying to do is just very basic analysis: ... sp=pf.h.sphere([0.5, 0.5, 0.5], rmax/pf['kpc']) sp_in=sp.cut_region(["grid['Theta']>0.6"]) prof=BinnedProfile1D(sp_in, 128, "Radiuskpc", 1e-2,rmax) r=prof["Radiuskpc"] prof.add_fields("Temperature",weight="CellMassMsun") ... It usually fails at the temperature part. Please let me know if you have any ideas or suggestions. Thank you! Yuan
Hi Yuan, On Thu, Jul 18, 2013 at 9:43 PM, Yuan Li <bear0980@gmail.com> wrote:
Hi yt users,
I have been having trouble using BinnedProfile1D. I am using the same script that seemed to be working before. Now it is giving me "MemoryError": http://paste.yt-project.org/show/A67caGKsv6mFqA83MZoZ/
I can still use it on a low resolution run without a problem.
I am not sure why it needs so much memory (I am already asking for 30 gb of memory). If I do a similar analysis with plotcollection and add_profile_sphere for the same data, I do not need to ask for much memory at all. I thought BinnedProfile1D would need less memory than add_profile_sphere. Please let me know if I was wrong and the job simply requires a lot of memory.
What I am trying to do is just very basic analysis: ... sp=pf.h.sphere([0.5, 0.5, 0.5], rmax/pf['kpc']) sp_in=sp.cut_region(["grid['Theta']>0.6"]) prof=BinnedProfile1D(sp_in, 128, "Radiuskpc", 1e-2,rmax) r=prof["Radiuskpc"] prof.add_fields("Temperature",weight="CellMassMsun")
Hm, this looks okay to me. The only two things that I think could be a problem: * Have you tried it on the sp object instead of sp_in? It may be that the cut_region is not operating in a memory conservative way. * Which version are you on? (yt instinfo should tell you) A month or so ago we identified (thanks to Mike Kuhlen and Nathan Goldbaum) a way to dramatically reduce memory used in radius fields. That may be relevant here. -Matt
...
It usually fails at the temperature part. Please let me know if you have any ideas or suggestions.
Thank you! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matt, I thought about the first one too. It did not seem to matter whether or not I cut the region. yt instinfo gives me this: The current version of the code is: e3e4f4578504 I have just updated yt again, and I am still getting the same error message. I did manage to make the plots with 64 gb memory, so I guess it really was just a memory issue. It would be great if BinnedProfile1D could be more memory conservative, but for now, I will use add_profile_sphere whenever possible. Thanks! Yuan On Fri, Jul 19, 2013 at 8:16 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Yuan,
Hi yt users,
I have been having trouble using BinnedProfile1D. I am using the same
On Thu, Jul 18, 2013 at 9:43 PM, Yuan Li <bear0980@gmail.com> wrote: script
that seemed to be working before. Now it is giving me "MemoryError": http://paste.yt-project.org/show/A67caGKsv6mFqA83MZoZ/
I can still use it on a low resolution run without a problem.
I am not sure why it needs so much memory (I am already asking for 30 gb of memory). If I do a similar analysis with plotcollection and add_profile_sphere for the same data, I do not need to ask for much memory at all. I thought BinnedProfile1D would need less memory than add_profile_sphere. Please let me know if I was wrong and the job simply requires a lot of memory.
What I am trying to do is just very basic analysis: ... sp=pf.h.sphere([0.5, 0.5, 0.5], rmax/pf['kpc']) sp_in=sp.cut_region(["grid['Theta']>0.6"]) prof=BinnedProfile1D(sp_in, 128, "Radiuskpc", 1e-2,rmax) r=prof["Radiuskpc"] prof.add_fields("Temperature",weight="CellMassMsun")
Hm, this looks okay to me. The only two things that I think could be a problem:
* Have you tried it on the sp object instead of sp_in? It may be that the cut_region is not operating in a memory conservative way. * Which version are you on? (yt instinfo should tell you) A month or so ago we identified (thanks to Mike Kuhlen and Nathan Goldbaum) a way to dramatically reduce memory used in radius fields. That may be relevant here.
-Matt
...
It usually fails at the temperature part. Please let me know if you have any ideas or suggestions.
Thank you! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Yuan, Oh, somehow I completely spaced on this, but BP1D is by default not memory conservative but a_p_s is. Supply the argument: lazy_reader=True when you create the BP1D, and it should fix it. Sorry about that! Matt On Jul 19, 2013 3:43 PM, "Yuan Li" <bear0980@gmail.com> wrote:
Hi Matt,
I thought about the first one too. It did not seem to matter whether or not I cut the region.
yt instinfo gives me this: The current version of the code is: e3e4f4578504
I have just updated yt again, and I am still getting the same error message. I did manage to make the plots with 64 gb memory, so I guess it really was just a memory issue. It would be great if BinnedProfile1D could be more memory conservative, but for now, I will use add_profile_sphere whenever possible.
Thanks! Yuan
On Fri, Jul 19, 2013 at 8:16 AM, Matthew Turk <matthewturk@gmail.com>wrote:
Hi Yuan,
Hi yt users,
I have been having trouble using BinnedProfile1D. I am using the same
On Thu, Jul 18, 2013 at 9:43 PM, Yuan Li <bear0980@gmail.com> wrote: script
that seemed to be working before. Now it is giving me "MemoryError": http://paste.yt-project.org/show/A67caGKsv6mFqA83MZoZ/
I can still use it on a low resolution run without a problem.
I am not sure why it needs so much memory (I am already asking for 30 gb of memory). If I do a similar analysis with plotcollection and add_profile_sphere for the same data, I do not need to ask for much memory at all. I thought BinnedProfile1D would need less memory than add_profile_sphere. Please let me know if I was wrong and the job simply requires a lot of memory.
What I am trying to do is just very basic analysis: ... sp=pf.h.sphere([0.5, 0.5, 0.5], rmax/pf['kpc']) sp_in=sp.cut_region(["grid['Theta']>0.6"]) prof=BinnedProfile1D(sp_in, 128, "Radiuskpc", 1e-2,rmax) r=prof["Radiuskpc"] prof.add_fields("Temperature",weight="CellMassMsun")
Hm, this looks okay to me. The only two things that I think could be a problem:
* Have you tried it on the sp object instead of sp_in? It may be that the cut_region is not operating in a memory conservative way. * Which version are you on? (yt instinfo should tell you) A month or so ago we identified (thanks to Mike Kuhlen and Nathan Goldbaum) a way to dramatically reduce memory used in radius fields. That may be relevant here.
-Matt
...
It usually fails at the temperature part. Please let me know if you have any ideas or suggestions.
Thank you! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matt, Thank you! Setting lazy_reader helped a lot. I don't know why I did not try that before! Thanks, Yuan On Fri, Jul 19, 2013 at 3:44 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Yuan,
Oh, somehow I completely spaced on this, but BP1D is by default not memory conservative but a_p_s is.
Supply the argument:
lazy_reader=True
when you create the BP1D, and it should fix it.
Sorry about that!
Matt On Jul 19, 2013 3:43 PM, "Yuan Li" <bear0980@gmail.com> wrote:
Hi Matt,
I thought about the first one too. It did not seem to matter whether or not I cut the region.
yt instinfo gives me this: The current version of the code is: e3e4f4578504
I have just updated yt again, and I am still getting the same error message. I did manage to make the plots with 64 gb memory, so I guess it really was just a memory issue. It would be great if BinnedProfile1D could be more memory conservative, but for now, I will use add_profile_sphere whenever possible.
Thanks! Yuan
On Fri, Jul 19, 2013 at 8:16 AM, Matthew Turk <matthewturk@gmail.com>wrote:
Hi Yuan,
Hi yt users,
I have been having trouble using BinnedProfile1D. I am using the same
On Thu, Jul 18, 2013 at 9:43 PM, Yuan Li <bear0980@gmail.com> wrote: script
that seemed to be working before. Now it is giving me "MemoryError": http://paste.yt-project.org/show/A67caGKsv6mFqA83MZoZ/
I can still use it on a low resolution run without a problem.
I am not sure why it needs so much memory (I am already asking for 30 gb of memory). If I do a similar analysis with plotcollection and add_profile_sphere for the same data, I do not need to ask for much memory at all. I thought BinnedProfile1D would need less memory than add_profile_sphere. Please let me know if I was wrong and the job simply requires a lot of memory.
What I am trying to do is just very basic analysis: ... sp=pf.h.sphere([0.5, 0.5, 0.5], rmax/pf['kpc']) sp_in=sp.cut_region(["grid['Theta']>0.6"]) prof=BinnedProfile1D(sp_in, 128, "Radiuskpc", 1e-2,rmax) r=prof["Radiuskpc"] prof.add_fields("Temperature",weight="CellMassMsun")
Hm, this looks okay to me. The only two things that I think could be a problem:
* Have you tried it on the sp object instead of sp_in? It may be that the cut_region is not operating in a memory conservative way. * Which version are you on? (yt instinfo should tell you) A month or so ago we identified (thanks to Mike Kuhlen and Nathan Goldbaum) a way to dramatically reduce memory used in radius fields. That may be relevant here.
-Matt
...
It usually fails at the temperature part. Please let me know if you have any ideas or suggestions.
Thank you! Yuan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (2)
-
Matthew Turk -
Yuan Li