Just want to say that the tweets are from me alone and shouldn't be taken as the opinion of the community. Thank you for attempting to correct some of the issues I noticed. This is the first I'm hearing about trouble working with very large Enzo datasets, it would be great to get more info about that on our bug tracker. We do actually have the ability to overplot different fields on the same volume rendering. Admittedly this is not documented terribly well, but all one needs to do is add multiple VolumeSource instances to the same scene. We're definitely aware of issues with working with large SPH datasets, the next major release will correct some of those deficiencies. Volume rendering for SPH data won't be in the initial release but I think will be much easier to implement a pure-SPH volume renderer following e.g. what the splotch code does after the release. On Mon, Apr 8, 2019 at 10:40 AM Joseph Smidt wrote:

Hey everyone,

This tweet [1] was brought to my attention. There is good criticism in that thread so let me respond:

1. It is ridiculous the YT paper is not cited. It defiantly was last I checked. How that was dropped is beyond me. I will make sure that is somehow fixed.

2. The "single slice plot of a single time step took 13 minutes" is also misleading and is a mistake that I will make sure is somehow corrected. I ran the simulations and wrote those sections and someone else ran the plotting routines - who is admittedly a Paraview developer - and I should have checked this text more closely. They were running both YT and Paraview from their Apple Macbook. The 13 minutes was how long it took that Macbook to volume render the dump versus Paraview, also running on a Macbook but using an HPC set of nodes in the backend. It's a feature of Paraview that you can do this.

Thus it is very misleading, I am embarrassed I missed it, and will make sure it is corrected.

3. That said, there are some real deficiencies in YT for HPC-class datasets

- The ability to overlay several variables using different color schemes on the same plot. There are times when having a temperature field and metallically field - for example - overlaying a density field is helpful. This was mentioned in the paper.

- The ability to volume render interactively until you get a plot exactly how you want it. With YT you have to keep running a script over and over again until it looks right. With paraview - even with a 26 Tb dump on our classified systems - files can be open and rotated and modified in real time to be exactly how you want in like 5 minutes. This was the cumbersome comment.

- Paraview's ability to open up a huge file - again, we have a 26 terabyte file on our classified side we used Paraview - and let the clusters do all the work in minutes from the comfort of your personal Macbook. A file that YT was unable to ever fit in memory for volume rendering no matter how many nodes we tried. This failure was not mentioned in the paper, even though some working with such huge datsets may be interested.

- Another omission from the paper is we did the same thing with gadget data, that is arguably used more than Enzo. Long story short, YT could never volume render the data, a known issue for at least 5 years [2], whereas paraview both reads and volume renders gadget data natively. Again, there is no reason to point this out in print even though some would be interested.

- The issue with Paraview not reading Enzo natively was true at the time of the analysis but is no longer true. There is a development version by the above developer that reads Enzo.

- Some of you work with Alex Gagliano. Though he has been working on this water network, he has nothing to do with how any of this paper is worded. Please do not give him grief for what is in it. He has a future paper on the subject and I guarantee these problems will not be repeated.

Anyway, I think YT is a fine product. I use it almost every day and encourage all students to use it. I am embarrassed this was lost from the paper and will work to restore it. But at the same time, to deny that Paraview comes with many advantages - especially as the world becomes more huge 3D TB sized datasets - is a little naive.

[1] https://twitter.com/njgoldbaum/status/1113163672120119298

[2] http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2014-June/005272...

-- ------------------------------------------------------------------------ Joseph Smidt

Theoretical Division P.O. Box 1663, Mail Stop B283 Los Alamos, NM 87545 Office: 505-665-9752 Fax: 505-667-1931 _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

I've opened #2232 https://github.com/yt-project/yt/issues/2232 to improve our docs on overplotting different fields on the same volume rendering, so hopefully that feature will be more accessible/visible to users in the future! On Mon, Apr 8, 2019 at 11:22 AM Matthew Turk wrote:

Hi Joseph,

Thanks for writing. Having been on SC viz papers, and also seeing both the stage-front and back-stage of doing press releases, etc, I can definitely see how this could happen!

When I initially saw the paper, my feelings were a bit hurt -- it felt like yt was being "punched down" at. After time and reflection my personal feelings on it mellowed, but I still feel like there was a missed opportunity for a deeper collaboration. Figuring out how to bring the different types of communities together (particularly since the means of production of software in the two relevant communities are so vastly different) would be an interesting opportunity, but it felt like that was sidestepped here.

I'm not going to respond *here* to the point-by-point items you note above, but I will end by saying that I do think that there's a bigger conversation we should have, and that until folks in the yt community can have a discussion about the flaws *without* getting defensive when those flaws are held up to other software packages (and I'm talking about myself here -- I am *not* referring to anyone else) we will be limited in our growth opportunities.

I want to end by noting something else. You mention that some of us work with Alex -- that's true! And in fact, it was during group meeting that I discovered this paper, and mentioned it, sharing my annoyance with how it discussed yt. I privately apologized to Alex afterward and intended to do so in our subsequent group meeting. I will replicate my apology here:

"Hey Alex, I realized that maybe I was a bit too harsh yesterday about that paper. I actually didn't really feel like it was that bad. People write papers -- they don't always say the stuff the people they're talking about would want them to. It's how it goes. I didn't need to be such a jerkwad about it. It was pretty inappropriate, and facetious or not, it was rude and uncalled for on my part. I think I reacted because I felt like they were "punching down" but on reflection, that turned into *me* "punching down" in a way that caught you up in the middle."

-Matt

On Mon, Apr 8, 2019 at 10:40 AM Joseph Smidt wrote:

...

Hey everyone,

This tweet [1] was brought to my attention. There is good criticism

in that thread so let me respond:

...

1. It is ridiculous the YT paper is not cited. It defiantly was last I

checked. How that was dropped is beyond me. I will make sure that is somehow fixed.

...

2. The "single slice plot of a single time step took 13 minutes" is also

misleading and is a mistake that I will make sure is somehow corrected. I ran the simulations and wrote those sections and someone else ran the plotting routines - who is admittedly a Paraview developer - and I should have checked this text more closely. They were running both YT and Paraview from their Apple Macbook. The 13 minutes was how long it took that Macbook to volume render the dump versus Paraview, also running on a Macbook but using an HPC set of nodes in the backend. It's a feature of Paraview that you can do this.

...

Thus it is very misleading, I am embarrassed I missed it, and will

make sure it is corrected.

...

3. That said, there are some real deficiencies in YT for HPC-class

datasets

...

- The ability to overlay several variables using different color

schemes on the same plot. There are times when having a temperature field and metallically field - for example - overlaying a density field is helpful. This was mentioned in the paper.

...

- The ability to volume render interactively until you get a plot

exactly how you want it. With YT you have to keep running a script over and over again until it looks right. With paraview - even with a 26 Tb dump on our classified systems - files can be open and rotated and modified in real time to be exactly how you want in like 5 minutes. This was the cumbersome comment.

...

- Paraview's ability to open up a huge file - again, we have a 26

terabyte file on our classified side we used Paraview - and let the clusters do all the work in minutes from the comfort of your personal Macbook. A file that YT was unable to ever fit in memory for volume rendering no matter how many nodes we tried. This failure was not mentioned in the paper, even though some working with such huge datsets may be interested.

...

- Another omission from the paper is we did the same thing with gadget

data, that is arguably used more than Enzo. Long story short, YT could never volume render the data, a known issue for at least 5 years [2], whereas paraview both reads and volume renders gadget data natively. Again, there is no reason to point this out in print even though some would be interested.

...

- The issue with Paraview not reading Enzo natively was true at the

time of the analysis but is no longer true. There is a development version by the above developer that reads Enzo.

...

- Some of you work with Alex Gagliano. Though he has been working on

this water network, he has nothing to do with how any of this paper is worded. Please do not give him grief for what is in it. He has a future paper on the subject and I guarantee these problems will not be repeated.

...

Anyway, I think YT is a fine product. I use it almost every day and

encourage all students to use it. I am embarrassed this was lost from the paper and will work to restore it. But at the same time, to deny that Paraview comes with many advantages - especially as the world becomes more huge 3D TB sized datasets - is a little naive.

...

[1] https://twitter.com/njgoldbaum/status/1113163672120119298

[2]

http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2014-June/005272...

...

-- ------------------------------------------------------------------------ Joseph Smidt

Theoretical Division P.O. Box 1663, Mail Stop B283 Los Alamos, NM 87545 Office: 505-665-9752 Fax: 505-667-1931 _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

Hey everyone, One last update: I have talked verbally with the coauthors. Mistakes were made but the buck stops with me. There will be at least an errata, possibly a retraction, and I can promise this will never happen again. At least not on any paper I am first author on. Though there are some features I like about Paraview, YT is my goto tool for almost all astrophysics visualization. I like it immensely and want nothing more than to make sure it is promoted going forward. My new summer students will be trained on it from day one, etc... You all do great work and that needs to be admitted. Once again I am sorry this happened. On Mon, Apr 8, 2019 at 11:21 AM Joseph Smidt wrote:

Hey everyone,

Thanks. I will try to improve the paper as best I can now that it is out. I can promise you my goal was not to dump on YT - as I personally use it a lot - as much as to give Paraview some credit as it's what some developers work on here and they were the ones attending the conference. I apologize and for certain will do better by YT in the future.

For example, Alex's papers will definitely promote YT as it's what he uses. He also is very grateful for all the help you give him.

Thanks agains and sorry for the drama this caused.

On Mon, Apr 8, 2019 at 10:37 AM Madicken Munk wrote:

...

I've opened #2232 https://github.com/yt-project/yt/issues/2232 to improve our docs on overplotting different fields on the same volume rendering, so hopefully that feature will be more accessible/visible to users in the future!

On Mon, Apr 8, 2019 at 11:22 AM Matthew Turk wrote:

...

Hi Joseph,

Thanks for writing. Having been on SC viz papers, and also seeing both the stage-front and back-stage of doing press releases, etc, I can definitely see how this could happen!

When I initially saw the paper, my feelings were a bit hurt -- it felt like yt was being "punched down" at. After time and reflection my personal feelings on it mellowed, but I still feel like there was a missed opportunity for a deeper collaboration. Figuring out how to bring the different types of communities together (particularly since the means of production of software in the two relevant communities are so vastly different) would be an interesting opportunity, but it felt like that was sidestepped here.

I'm not going to respond *here* to the point-by-point items you note above, but I will end by saying that I do think that there's a bigger conversation we should have, and that until folks in the yt community can have a discussion about the flaws *without* getting defensive when those flaws are held up to other software packages (and I'm talking about myself here -- I am *not* referring to anyone else) we will be limited in our growth opportunities.

I want to end by noting something else. You mention that some of us work with Alex -- that's true! And in fact, it was during group meeting that I discovered this paper, and mentioned it, sharing my annoyance with how it discussed yt. I privately apologized to Alex afterward and intended to do so in our subsequent group meeting. I will replicate my apology here:

"Hey Alex, I realized that maybe I was a bit too harsh yesterday about that paper. I actually didn't really feel like it was that bad. People write papers -- they don't always say the stuff the people they're talking about would want them to. It's how it goes. I didn't need to be such a jerkwad about it. It was pretty inappropriate, and facetious or not, it was rude and uncalled for on my part. I think I reacted because I felt like they were "punching down" but on reflection, that turned into *me* "punching down" in a way that caught you up in the middle."

-Matt

On Mon, Apr 8, 2019 at 10:40 AM Joseph Smidt wrote:

...

Hey everyone,

This tweet [1] was brought to my attention. There is good

criticism in that thread so let me respond:

...

1. It is ridiculous the YT paper is not cited. It defiantly was last I

checked. How that was dropped is beyond me. I will make sure that is somehow fixed.

...

2. The "single slice plot of a single time step took 13 minutes" is

also misleading and is a mistake that I will make sure is somehow corrected. I ran the simulations and wrote those sections and someone else ran the plotting routines - who is admittedly a Paraview developer - and I should have checked this text more closely. They were running both YT and Paraview from their Apple Macbook. The 13 minutes was how long it took that Macbook to volume render the dump versus Paraview, also running on a Macbook but using an HPC set of nodes in the backend. It's a feature of Paraview that you can do this.

...

Thus it is very misleading, I am embarrassed I missed it, and will

make sure it is corrected.

...

3. That said, there are some real deficiencies in YT for HPC-class

datasets

...

- The ability to overlay several variables using different color

schemes on the same plot. There are times when having a temperature field and metallically field - for example - overlaying a density field is helpful. This was mentioned in the paper.

...

- The ability to volume render interactively until you get a plot

exactly how you want it. With YT you have to keep running a script over and over again until it looks right. With paraview - even with a 26 Tb dump on our classified systems - files can be open and rotated and modified in real time to be exactly how you want in like 5 minutes. This was the cumbersome comment.

...

- Paraview's ability to open up a huge file - again, we have a 26

terabyte file on our classified side we used Paraview - and let the clusters do all the work in minutes from the comfort of your personal Macbook. A file that YT was unable to ever fit in memory for volume rendering no matter how many nodes we tried. This failure was not mentioned in the paper, even though some working with such huge datsets may be interested.

...

- Another omission from the paper is we did the same thing with

gadget data, that is arguably used more than Enzo. Long story short, YT could never volume render the data, a known issue for at least 5 years [2], whereas paraview both reads and volume renders gadget data natively. Again, there is no reason to point this out in print even though some would be interested.

...

- The issue with Paraview not reading Enzo natively was true at the

time of the analysis but is no longer true. There is a development version by the above developer that reads Enzo.

...

- Some of you work with Alex Gagliano. Though he has been working

on this water network, he has nothing to do with how any of this paper is worded. Please do not give him grief for what is in it. He has a future paper on the subject and I guarantee these problems will not be repeated.

...

Anyway, I think YT is a fine product. I use it almost every day and

encourage all students to use it. I am embarrassed this was lost from the paper and will work to restore it. But at the same time, to deny that Paraview comes with many advantages - especially as the world becomes more huge 3D TB sized datasets - is a little naive.

...

[1] https://twitter.com/njgoldbaum/status/1113163672120119298

[2]

http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2014-June/005272...

...

--

------------------------------------------------------------------------

...

Joseph Smidt

Theoretical Division P.O. Box 1663, Mail Stop B283 Los Alamos, NM 87545 Office: 505-665-9752 Fax: 505-667-1931 _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

---

yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org

-- ------------------------------------------------------------------------ Joseph Smidt

Theoretical Division P.O. Box 1663, Mail Stop B283 Los Alamos, NM 87545 Office: 505-665-9752 Fax: 505-667-1931

-- ------------------------------------------------------------------------ Joseph Smidt Theoretical Division P.O. Box 1663, Mail Stop B283 Los Alamos, NM 87545 Office: 505-665-9752 Fax: 505-667-1931