Thank you for the great response.

Unfortunately I'm not able to share the data at this point.
I tried switching from using a timeseries to using the path to each output folder as you suggested.

It seems that even if I strip my code down to the bare essentials where I just load each sim in a for loop and load the field list, it keeps every sim in memory as long as the for loop is still running. It does drop them all at the end of the loop but I would like each to be dropped when the next one loads.

However, if I try saving a projection plot, for each run through the loop, it does not drop the simulations when the loop finishes.
By saving I mean both saving the plot to disk and keeping the figure object in an array for later use.

Does this help at all?


The axis names I was referring to was the axes on the ProfilePlot object, not the ProjectionPlot one. Specifically the 1D profile one with a simple x and y axis. Could you please point me in the right direction?


Best,
Andreas Ellewsen

On Wed, 14 Dec 2016 at 16:18 Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Wed, Dec 14, 2016 at 5:30 AM, Thor Andreas Seiff Ellewsen <tellewsen@gmail.com> wrote:
Hi all,

I'm working on a project where I'm comparing different RAMSES simulations.
However I'm running into a problem with memory.
It seems that yt loads the whole simulation into memory even though I'm only interested in a number of dark matter halos which I have the coordinates for.

Yes, this is a limitation of our current approach for octree data, we must build a global octree for indexing purposes. For large simulations this can be a prohibitive memory cost. We have long-term plans to fix this but as far as I know no one is actively working on it.
 

It also seems like it doesn't drop the first simulation out of memory when I'm done making the figures I want for that one. 

This quickly fills up the memory.

You can find the relevant parts of my code at the pastebin at http://paste.yt/p4946.html
I'm running yt version 3.3.2

What am I doing wrong?

In principle this shouldn't happen but it's entirely possible there is a bug somewhere. In the python memory model, an object is retained in memory until no other object have a reference to it, at which point it is garbage collected. I don't see anything in your script that would cause references to old datasets to be retained, so it might be the case that yt is inadvertantly retaining a reference to datasets in a ramses time series when it shouldn't be or if we need the reference that we should be using a weakref. Unfortunately it looks like we don't have any Ramses time series datasets available in our repository of public test datasets (http://yt-project.org/data/) so I don't have a way of reproducing this behavior. Is there any chance you can make a ramses time series available for testing? You might find the yt curldrop to be useful for sharing a large dataset:


A test that this is isolated to the time series object would be to instead use a loop over filenames, loading a single filename in each loop iteration, and then explicitly deleting the dataset object using the `del` statement at the end of the loop. In principle this shouldn't be necessary but it might help reveal where the issue is localized to.
 

Also, is there a way to change the axis names?
I'm making some dark matter density profiles using the ('deposit','io_density') field and this gives IO Density on the y axis, but I just want it to say Dark matter density so my readers actually understand what I'm talking about.


Thanks for helping

Best,
Andreas Ellewsen



_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org