Re: [Yt-dev] Parameter file store

So no problems?  I should feel free to apply the patch to trunk?

Seems good so far. I re-read and re-plotted some clumps I made a while ago, and it matches a plot I made about a week ago. Patch away. d.

On Fri, Mar 6, 2009 at 9:32 AM, david collins wrote:

...

Dude.

Looking at time per parameter file read, reading the first 600 steps from one sim.

Before the fix, it monotonically increased from 0.01 second/read to 3 seconds/read.

After the fix, it's roughly constant at 0.02.  Huge improvement in performance.  That's really going to speed things up when I make my core tracking analysis.  Thanks!

d.

On Thu, Mar 5, 2009 at 10:49 PM, Matthew Turk wrote:

...

Any luck?

On Tue, Mar 3, 2009 at 3:19 PM, Matthew Turk wrote:

...

Hi Dave,

Here's a patch:

http://paste.enzotools.org/show/68/

You can get this with yt_lodgeit.py --download=68 > patch and apply it with patch -p1 < patch .  Conversely, you can download it from the bitbucket repo or the mirror on hg.enzotools.org.  It defaults to keeping the last 300 touched files around.

If this works I'll merge back to trunk.  You can test it just by instantiated a crapton of them in a row.

-Matt

On Tue, Mar 3, 2009 at 2:25 PM, Matthew Turk wrote:

...

That's right, it's completely re-creatable.

On Tue, Mar 3, 2009 at 2:19 PM, david collins wrote:

...

Sure.  So one would need to re-instantiate if something gets bumped off?

d.

On Tue, Mar 3, 2009 at 2:15 PM, Matthew Turk wrote: > Okay, how about this. > > We make a 'freshness' column in the CSV.  This is a timestamp of the > last time the parameter file was used.  This can be the sort field, > too.  We'll set a variable -- default it to, say, 300 -- that includes > the freshness cutoff in number of parameter files.  Thus, sorted by > freshness, we take the last N, where N is set in the parameter file. > > Sound good? > > On Tue, Mar 3, 2009 at 1:47 PM, david collins wrote: >>> Wow, you've been analyzing 10,000 parameter files?  Okay. >> >> Ci.  Making time sequences.  Hundreds/thousands of snapshots from a >> dozen sims, and comparing.   (This is why uber) >> >> Deleting parameter_files.csv is a good workaround for me now, so don't >> spend more than 45 seconds on it.  I did this recently, so it's not >> huge, only 68 Mb. >> >> kraken /nics/b/home/collins/.yt/parameter_files.csv >> >> d. >> >>> >>> One thing I can do is implement a maximum number in the store.  Then >>> we can only guarantee objects will be automatically retrieved (without >>> having *recently* instantiated the PF) if they were in the last N, >>> where N is the number of files we keep in the store. >>> >>> Can you send me one of your parameter file stores?  Or put it on kraken? >>> >>> -Matt >>> - Show quoted text - >>> On Tue, Mar 3, 2009 at 1:04 PM, david collins wrote: >>>> Hey-- >>>> >>>> Not sure if this belongs on the dev list-- >>>> >>>> I've been reading and analyzing large-ish ( O(10,000) ) parameter >>>> files, and the parameter_file.csv gets large (300 Mb), which isn't a >>>> problem, but when that happens, doing anything (even importing YT) >>>> takes forever (> 1 minute to import yt.lagos). >>>> >>>> I did install the "unboudned growth" fix you sent, still happens. >>>> >>>> I just noticed that it  happens after your fix yesterday, or I'da said >>>> something earlier. >>>> >>>> d. >>>> >>> _______________________________________________ >>> Yt-dev mailing list >>> Yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> _______________________________________________ >> Yt-dev mailing list >> Yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ > Yt-dev mailing list > Yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org