When can we turn off serialization by default?
Hi all, I'd like to turn off serialization by default. This includes: 1) Projections 2) Grid hierarchy 3) Field lists and maybe other things. In general, I think it's a bad idea to just litter directories with new files, and specifically I think it's a bad idea to store 2&3 since they are a huge source of errors for new and existing users. My preference would be to disable all auto-serialization of anything and make that not possible, mandating explicit, manual saves of data instead. However I recognize that is likely unpopular, as even though my simulations have a lot of data on disk, I know that they also take a relatively short time to project. So when can we make it disabled-by-default? 2.6? 3.0? Now? -Matt
I think we can make it disabled-by-default now. I think we may want to update the docs on saving projections. It's not clear to me how to save a projection from: http://yt-project.org/doc/analyzing/objects.html#storing-and-loading-objects and I couldn't find a better doc page for projection/2d data object storage. Anyways, I'm +1 on disabling serialization by default, and would be +0 on getting rid of it completely. Sam On Fri, Apr 19, 2013 at 8:31 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
I'd like to turn off serialization by default. This includes:
1) Projections 2) Grid hierarchy 3) Field lists
and maybe other things. In general, I think it's a bad idea to just litter directories with new files, and specifically I think it's a bad idea to store 2&3 since they are a huge source of errors for new and existing users.
My preference would be to disable all auto-serialization of anything and make that not possible, mandating explicit, manual saves of data instead. However I recognize that is likely unpopular, as even though my simulations have a lot of data on disk, I know that they also take a relatively short time to project.
So when can we make it disabled-by-default? 2.6? 3.0? Now?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that? -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at. -Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for doing it now. On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
Dave, out of curiosity, how long does a projection of one of your datasets take? On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for doing it now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I have a small handful of use cases where it would be nice to save a serialized projection. The primary case I'm thinking of is making projections on a supercomputer and downloading the serialized files locally to make images. However, if we're totally rethinking what yt should be doing on its own with writing files, I am totally in favor of making it an explicit, opt-in thing. Is it simple enough to set the serialization keyword default to False? In general, performing analysis on supercomputers and then downloading reduced data to my laptop to make figures is how I do almost everything. On Fri, Apr 19, 2013 at 11:18 AM, Matthew Turk <matthewturk@gmail.com>wrote:
Dave, out of curiosity, how long does a projection of one of your datasets take?
On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for doing
it
now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I just did a test. 512^3 with 4 levels everywhere. On my desktop, it took 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is not that fast today. This is, of course, and extreme example, but it isn't by any means rare for that system. 78 seconds isn't too bad, but today I'll be looking at >20 datasets, so it adds up. I definitely am in favor of an opt-in system, though, as many applications won't be this shape or on this disk. Or even an explicit call, such as ProjectinPlot.serialize() pf = load(fname) t0=time.time() proj=ProjectionPlot(pf,2,'Density') t1=time.time() print t1-t0 Parsing Hierarchy100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Initializing tree 0 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 0 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:03:39 Initializing tree 1 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 1 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:12 Initializing tree 2 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 2 / 4 80% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| | ETA: 00:00:52 Projecting level 2 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:17 Initializing tree 3 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 3 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:02:58 Initializing tree 4 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 4 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:06 1167.0001719 On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Dave, out of curiosity, how long does a projection of one of your datasets take?
On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for doing
it
now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher@gmail.com> wrote:
I just did a test. 512^3 with 4 levels everywhere. On my desktop, it took 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is not that fast today. This is, of course, and extreme example, but it isn't by any means rare for that system. 78 seconds isn't too bad, but today I'll be looking at >20 datasets, so it adds up.
I understand, 78 seconds will definitely add up. I'm impressed by the difference between Kraken and local. Or maybe flabberghasted? Either way, it's definitely something.
I definitely am in favor of an opt-in system, though, as many applications won't be this shape or on this disk. Or even an explicit call, such as ProjectinPlot.serialize()
Okay, cool. I think that'd be a good one, like Britton mentioned too.
pf = load(fname) t0=time.time() proj=ProjectionPlot(pf,2,'Density') t1=time.time() print t1-t0 Parsing Hierarchy100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Initializing tree 0 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 0 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:03:39 Initializing tree 1 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 1 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:12 Initializing tree 2 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 2 / 4 80% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| | ETA: 00:00:52 Projecting level 2 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:17 Initializing tree 3 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 3 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:02:58 Initializing tree 4 / 4100% |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 Projecting level 4 / 4 100% ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Time: 00:04:06 1167.0001719
On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Dave, out of curiosity, how long does a projection of one of your datasets take?
On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for doing it now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I am not surprised--Kraken is the worst cluster on the planet. On Fri, Apr 19, 2013 at 10:26 AM, Matthew Turk <matthewturk@gmail.com>wrote:
On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher@gmail.com> wrote:
I just did a test. 512^3 with 4 levels everywhere. On my desktop, it took 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is not that fast today. This is, of course, and extreme example, but it isn't by any means rare for that system. 78 seconds isn't too bad, but today I'll be looking at >20 datasets, so it adds up.
I understand, 78 seconds will definitely add up. I'm impressed by the difference between Kraken and local. Or maybe flabberghasted? Either way, it's definitely something.
I definitely am in favor of an opt-in system, though, as many
applications
won't be this shape or on this disk. Or even an explicit call, such as ProjectinPlot.serialize()
Okay, cool. I think that'd be a good one, like Britton mentioned too.
pf = load(fname) t0=time.time() proj=ProjectionPlot(pf,2,'Density') t1=time.time() print t1-t0 Parsing Hierarchy100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Initializing tree 0 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 0 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:03:39 Initializing tree 1 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 1 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:12 Initializing tree 2 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 2 / 4 80%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ETA: 00:00:52 Projecting level 2 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:17 Initializing tree 3 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 3 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:02:58 Initializing tree 4 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 4 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:06 1167.0001719
On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Dave, out of curiosity, how long does a projection of one of your
datasets
take?
On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the current behavior, or some reasonable facsimile, is still available I'm for
doing
it now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com> wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote:
Just to play devil's advocate here, what would it take to "fix" serialization? Adding a checksum so that data changes can be detected? More than that?
That would partly fix the issue of fields/hierarchy changes not being detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
Can we do it yesterday???!? On Fri, Apr 19, 2013 at 2:02 PM, Cameron Hummels <chummels@gmail.com> wrote:
I am not surprised--Kraken is the worst cluster on the planet.
On Fri, Apr 19, 2013 at 10:26 AM, Matthew Turk <matthewturk@gmail.com>wrote:
On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher@gmail.com> wrote:
I just did a test. 512^3 with 4 levels everywhere. On my desktop, it took 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is not that fast today. This is, of course, and extreme example, but it isn't by any means rare for that system. 78 seconds isn't too bad, but today I'll be looking at >20 datasets, so it adds up.
I understand, 78 seconds will definitely add up. I'm impressed by the difference between Kraken and local. Or maybe flabberghasted? Either way, it's definitely something.
I definitely am in favor of an opt-in system, though, as many
applications
won't be this shape or on this disk. Or even an explicit call, such as ProjectinPlot.serialize()
Okay, cool. I think that'd be a good one, like Britton mentioned too.
pf = load(fname) t0=time.time() proj=ProjectionPlot(pf,2,'Density') t1=time.time() print t1-t0 Parsing Hierarchy100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Initializing tree 0 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 0 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:03:39 Initializing tree 1 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 1 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:12 Initializing tree 2 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 2 / 4 80%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ETA: 00:00:52 Projecting level 2 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:17 Initializing tree 3 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 3 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:02:58 Initializing tree 4 / 4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00 Projecting level 4 / 4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:06 1167.0001719
On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Dave, out of curiosity, how long does a projection of one of your
datasets
take?
On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher@gmail.com> wrote:
I use serialized projections quite frequently. As long as the
current
behavior, or some reasonable facsimile, is still available I'm for doing it now.
On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk@gmail.com
wrote:
On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s@skory.us> wrote: > Just to play devil's advocate here, what would it take to "fix" > serialization? Adding a checksum so that data changes can be > detected? > More than that?
That would partly fix the issue of fields/hierarchy changes not
being
detected (unless you manually mess with the data). But it wouldn't fix the deeper problem, which is that a) we scatter files willy nilly about the directory, which I am coming to feel is a really gross violation of expectations, and b) the process of auto-serialization doesn't save a huge amount of time in most cases. JohnW and I spitballed last fall about some of the biggest hierarchies he's dealt with in Enzo and I promised to write a Cython parser, which I never succeeded at.
-Matt
> > > > -- > Stephen Skory > s@skory.us > http://stephenskory.com/ > 510.621.3687 (google voice) > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer.
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (7)
-
Britton Smith -
Cameron Hummels -
david collins -
j s oishi -
Matthew Turk -
Sam Skillman -
Stephen Skory