Re: [Yt-dev] A fun interface for data

This is a super cool idea, especially given what I saw when you demo'd it to me, Matt. So what you're proposing is a sort of "standalone" version of the applet that allows pan-n-scan in our plot windows, yes? One so that a person could just go to a remote website that had the data on it and be able to move around in that data interactively for presentation to an audience, or basically like a googlemaps interace for a few datasets? I think this would be very useful, however, we need to take things one step at a time, I think. Why don't we implement this in the existing reason for now, and see how that works. After all, reason will be able to deal with more than just projections of the dataset and more than just looking at it from one perspective, right? Correct me if I'm wrong, but what you're describing is a scaled down version of this which would only be able to look at the tiles for one perspective's projections? Because if we want to do more than one perspective, or more than one type of data (slices, projections, off-axis projections, etc.), then we need to store more tiles, right? Then we need more room for the data, and we need a means in the window of switching between the various views and modes, and then it becomes closer and closer to the original reason. So what I think we should do is first focus on implementing it with the full suite of tools that we're doing in reason, and THEN if things are slow for the full reason implementation, we could make a reason-light version that scales down the feature set to only pan-n-scan for one mode and one perspective. What do you think? Oh, btw, I totally want to be involved in this, because it is super awesome. Cameron On 05/18/2011 01:52 PM, Matthew Turk wrote:

Thanks, Sam!

As a quick note, I changed all of the PNG writing to not use tempfile but instead write directly to in-memory, which was a pretty glaring problem waiting to rear its head.

I was idly thinking about this yesterday, and after some conversations I've had with Tom and others I think that one could imagine a very cool project coming out of this. The idea would be that the minimum amount of information to do an entire pan-n-scan is really just the 5xN array for a given projection or slice: (px, py, pdx, pdy, z). This is how it currently works -- it simply repixelizes that array for each tile segment, and returns a PNG of that newly created tile. The storage on the backend is really quite small; these 5xN arrays are much more efficient than creating all the possible tiles in advance. They're not free in terms of web-server memory, like the tiles would be, but they're quite inexpensive. I believe that even for the 1024^3 L7 dataset it was on the order of a hundred megs or so.

So imagine having a server that lived somewhere that stored a bunch of these 5xN arrays in memory, along with appropriate metadata. Go to the frontend of the server, select one to view, and then having that dataset pop up. Although they aren't in the mapserver command, one can supply annotations and even drawings with the leaflet library.

(I've basically just described a number of existing services. Except, ours would be based on simulation data exclusively and would probably be able to scale to having many, many datasets!)

You'd have to implement a frontend, a backend that stored these datasets, and likely have some kind of garbage collection -- if the last request for a given dataset was more than N minutes ago, remove it from memory. I think this could be a viable, fun project, and would be really excellent to use as supplemental data for papers and presentations -- particularly for star formation and cosmology, where there are a number of points of interest within relatively high dynamical range.

Anyway, if someone is interested in working on this, it could be VERY fun and possibly quite straightforward to implement. I will confess I don't know enough about web apps, server deployment and database backends to move forward on it, but I'd be more than happy to help out with the data transport and visualization aspects, which I think the mapserver already has a good start on. Coming up with money to run this on, say, EC2 is probably also possible; it may even be reasonable to think we could make this a project that people in the community could upload to and use.

-Matt

On Wed, May 18, 2011 at 1:37 PM, Sam Skillman wrote:

...

Matt, This is awesome, and I'm definitely a +1 on rolling this into reason in the future. I'm very tempted to throw this up on a big set of displays, and up the window size :). Well done! Sam On Wed, May 18, 2011 at 1:42 AM, Matthew Turk wrote:

...

Thanks for the positive feedback, Stephen -- I'm pretty excited about this, too.

As a stopgap before it gets rolled into reason, which may end up being sort of tricky, I have added a "mapserver" command to the yt command line utility. If you update your installation, you can do:

yt mapserver DD0030/DD0030

and it'll spawn on http://127.0.0.1:8080/ , which you can then open in your browser. There are a couple options for this command, too -- axis, field, projection, and weight, which you can see with --help.

-Matt

On Tue, May 17, 2011 at 4:15 PM, Stephen Skory wrote:

...

Hi Matt,

...

I'd love it if people could download and try it out. If you have yt installed, then you don't need anything other than this repo: I just gave it a go on a couple datasets and it worked pretty well! The periodicity is handled interestingly - just like Google Maps where you can pan left and right and get the same thing, but up and down cuts off the data. Really neat!

-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu PGP: 0x06F886E3