I recently ran into some major slowdowns in going from yt version 3.1 to 3.2 on certain machines, although it wasn't related to my calling the annotate_contour callback. When I tracked it down with pdb, the slowdown was in indexing certain arrays, so maybe there is a similar issue you're encountering. I'm going to track down the changeset that this is tied to this week, and I'll let you know what I find. Cameron On Mon, Aug 10, 2015 at 3:15 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:

Hey Bili,

Just a shot in the dark, but do you have different matplotlib versions installed on your machine and server at UCSD? The contour callback is using some internal machinery in matplotlib that changed relatively recently.

It might also be worthwhile to run your script under cProfile so we get an idea of which function exactly is taking so long. Even better if you can run *that* function under the line_profiler.

-Nathan

On Mon, Aug 10, 2015 at 5:10 PM, Bili Dong <qobilidop@gmail.com> wrote:

...

Hi yt-users,

I’m using yt-3.2 (Changeset = 329b5039b031 stable).

The problem is that the same script that could finish in seconds on my local machine will take much (order of magnitude) longer time to run on the server I’m using. In some cases, it get stuck for hours that I don’t know if it will ever finish. I’ve tracked down the bottleneck to be the `annotate_contour` method. The server should be faster than my local machine. (I’m not exactly sure about how to compare it. But the server is generally much faster in other situations.)

For example, for this script (http://paste.yt-project.org/show/5808/), on my local machine the output shows:

``` yt : [INFO ] 2015-08-10 *14:48:15*,401 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 *14:48:15*,751 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 *14:48:16*,716 Saving plot snapshot_200_Projection_z_density.png ```

However, on the server the output shows: ``` yt : [INFO ] 2015-08-10 *14:51:25*,551 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 *14:51:25*,917 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 *14:56:21*,922 Saving plot snapshot_200_Projection_z_density.png ```

That’s *296* times slower!!!

This is likely to be a very specific problem with the server. But I’d like to know if anyone else has encountered similar problems before so I can learn from your experience. For now, I have no idea what could have gone wrong.

BTW, I’m using TSCC (http://idi.ucsd.edu/computing/) at UC San Diego, if you happen to know it.

Thanks, Bili

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- Cameron Hummels NSF Postdoctoral Fellow Department of Astronomy California Institute of Technology http://chummels.org

The previous output in pastebin is truncated. Here’s the full version http://use.yt/upload/410b970d <http://use.yt/upload/410b970d>. -Bili

On Aug 10, 2015, at 3:41 PM, Bili Dong <qobilidop@gmail.com> wrote:

Hi Nathan,

I checked that the matplotlib versions are the same.

I changed my script to http://paste.yt-project.org/show/5809/ <http://paste.yt-project.org/show/5809/>. The results of cProfile on the server are http://paste.yt-project.org/show/5810/ <http://paste.yt-project.org/show/5810/>. Judging by eye, it seems to be related to the triangulation functionality.

-Bili

...

On Aug 10, 2015, at 3:15 PM, Nathan Goldbaum <nathan12343@gmail.com <mailto:nathan12343@gmail.com>> wrote:

Hey Bili,

Just a shot in the dark, but do you have different matplotlib versions installed on your machine and server at UCSD? The contour callback is using some internal machinery in matplotlib that changed relatively recently.

It might also be worthwhile to run your script under cProfile so we get an idea of which function exactly is taking so long. Even better if you can run *that* function under the line_profiler.

-Nathan

On Mon, Aug 10, 2015 at 5:10 PM, Bili Dong <qobilidop@gmail.com <mailto:qobilidop@gmail.com>> wrote: Hi yt-users,

I’m using yt-3.2 (Changeset = 329b5039b031 stable).

The problem is that the same script that could finish in seconds on my local machine will take much (order of magnitude) longer time to run on the server I’m using. In some cases, it get stuck for hours that I don’t know if it will ever finish. I’ve tracked down the bottleneck to be the `annotate_contour` method. The server should be faster than my local machine. (I’m not exactly sure about how to compare it. But the server is generally much faster in other situations.)

For example, for this script (http://paste.yt-project.org/show/5808/ <http://paste.yt-project.org/show/5808/>), on my local machine the output shows:

``` yt : [INFO ] 2015-08-10 14:48:15,401 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:48:15,751 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:48:16,716 Saving plot snapshot_200_Projection_z_density.png ```

However, on the server the output shows: ``` yt : [INFO ] 2015-08-10 14:51:25,551 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:51:25,917 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:56:21,922 Saving plot snapshot_200_Projection_z_density.png ```

That’s 296 times slower!!!

This is likely to be a very specific problem with the server. But I’d like to know if anyone else has encountered similar problems before so I can learn from your experience. For now, I have no idea what could have gone wrong.

BTW, I’m using TSCC (http://idi.ucsd.edu/computing/ <http://idi.ucsd.edu/computing/>) at UC San Diego, if you happen to know it.

Thanks, Bili

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

I reinstalled matplotlib and it works. Thanks Kacper! You saved my life! -Bili

On Aug 10, 2015, at 4:30 PM, Kacper Kowalik <xarthisius.kk@gmail.com> wrote:

On 08/10/2015 05:10 PM, Bili Dong wrote:

...

Hi yt-users,

I’m using yt-3.2 (Changeset = 329b5039b031 stable).

The problem is that the same script that could finish in seconds on my local machine will take much (order of magnitude) longer time to run on the server I’m using. In some cases, it get stuck for hours that I don’t know if it will ever finish. I’ve tracked down the bottleneck to be the `annotate_contour` method. The server should be faster than my local machine. (I’m not exactly sure about how to compare it. But the server is generally much faster in other situations.)

For example, for this script (http://paste.yt-project.org/show/5808/ <http://paste.yt-project.org/show/5808/>), on my local machine the output shows:

``` yt : [INFO ] 2015-08-10 14:48:15,401 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:48:15,751 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:48:16,716 Saving plot snapshot_200_Projection_z_density.png ```

However, on the server the output shows: ``` yt : [INFO ] 2015-08-10 14:51:25,551 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:51:25,917 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:56:21,922 Saving plot snapshot_200_Projection_z_density.png ```

That’s 296 times slower!!!

This is likely to be a very specific problem with the server. But I’d like to know if anyone else has encountered similar problems before so I can learn from your experience. For now, I have no idea what could have gone wrong.

BTW, I’m using TSCC (http://idi.ucsd.edu/computing/ <http://idi.ucsd.edu/computing/>) at UC San Diego, if you happen to know it.

Thanks, Bili

Hi Bili, make sure you pass "-DNDEBUG" to CPPFLAGS when you compile matplotlib. Without it mpl is a good cpu cycle burner. There's a debug routine in triangulation module that's enabled by default. Cheers, Kacper

...

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi John, Previously I was using the default mpl provided by the system. Then I installed a local version using `pip install --user matplotlib` without specifying -DNDEBUG explicitly, after which the slowdown was gone. I guess somehow the system default version missed the -DNDEBUG flag while compiling. But I don’t know the details. Regards, Bili

On Aug 10, 2015, at 6:54 PM, John Wise <jwise@physics.gatech.edu> wrote:

Hi,

I just wanted to follow-up with a question since -DNDEBUG should be specified by default from the Python installation. Does anyone know why CPPFLAGS wasn't passed into mpl's setup.py?

Bili, did you install yt with the install_script or on top of the system's or module's Python stack?

Thanks, John

On 08/10/2015 09:27 PM, Bili Dong wrote:

...

I reinstalled matplotlib and it works. Thanks Kacper! You saved my life!

-Bili

...

On Aug 10, 2015, at 4:30 PM, Kacper Kowalik <xarthisius.kk@gmail.com> wrote:

On 08/10/2015 05:10 PM, Bili Dong wrote:

...

Hi yt-users,

I’m using yt-3.2 (Changeset = 329b5039b031 stable).

The problem is that the same script that could finish in seconds on my local machine will take much (order of magnitude) longer time to run on the server I’m using. In some cases, it get stuck for hours that I don’t know if it will ever finish. I’ve tracked down the bottleneck to be the `annotate_contour` method. The server should be faster than my local machine. (I’m not exactly sure about how to compare it. But the server is generally much faster in other situations.)

For example, for this script (http://paste.yt-project.org/show/5808/ <http://paste.yt-project.org/show/5808/>), on my local machine the output shows:

``` yt : [INFO ] 2015-08-10 14:48:15,401 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:48:15,751 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:48:16,716 Saving plot snapshot_200_Projection_z_density.png ```

However, on the server the output shows: ``` yt : [INFO ] 2015-08-10 14:51:25,551 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:51:25,917 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:56:21,922 Saving plot snapshot_200_Projection_z_density.png ```

That’s 296 times slower!!!

This is likely to be a very specific problem with the server. But I’d like to know if anyone else has encountered similar problems before so I can learn from your experience. For now, I have no idea what could have gone wrong.

BTW, I’m using TSCC (http://idi.ucsd.edu/computing/ <http://idi.ucsd.edu/computing/>) at UC San Diego, if you happen to know it.

Thanks, Bili

Hi Bili, make sure you pass "-DNDEBUG" to CPPFLAGS when you compile matplotlib. Without it mpl is a good cpu cycle burner. There's a debug routine in triangulation module that's enabled by default. Cheers, Kacper

...

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- John Wise Assistant Professor of Physics Center for Relativistic Astrophysics, Georgia Tech http://cosmo.gatech.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Might be worth sending a message to the triton sysadmins too. On Monday, August 10, 2015, John Wise <jwise@physics.gatech.edu> wrote:

Thanks, Bili. There could have been some loose ends in the system version. I was confused because this shouldn't have happened with the install script, but this sort of clears things up.

Thanks again! John

On 08/10/2015 10:03 PM, Bili Dong wrote:

...

Hi John,

Previously I was using the default mpl provided by the system. Then I installed a local version using `pip install --user matplotlib` without specifying -DNDEBUG explicitly, after which the slowdown was gone. I guess somehow the system default version missed the -DNDEBUG flag while compiling. But I don’t know the details.

Regards, Bili

On Aug 10, 2015, at 6:54 PM, John Wise <jwise@physics.gatech.edu> wrote:

...

Hi,

I just wanted to follow-up with a question since -DNDEBUG should be specified by default from the Python installation. Does anyone know why CPPFLAGS wasn't passed into mpl's setup.py?

Bili, did you install yt with the install_script or on top of the system's or module's Python stack?

Thanks, John

On 08/10/2015 09:27 PM, Bili Dong wrote:

...

I reinstalled matplotlib and it works. Thanks Kacper! You saved my life!

-Bili

On Aug 10, 2015, at 4:30 PM, Kacper Kowalik <xarthisius.kk@gmail.com>

...

wrote:

On 08/10/2015 05:10 PM, Bili Dong wrote:

...

Hi yt-users,

I’m using yt-3.2 (Changeset = 329b5039b031 stable).

The problem is that the same script that could finish in seconds on my local machine will take much (order of magnitude) longer time to run on the server I’m using. In some cases, it get stuck for hours that I don’t know if it will ever finish. I’ve tracked down the bottleneck to be the `annotate_contour` method. The server should be faster than my local machine. (I’m not exactly sure about how to compare it. But the server is generally much faster in other situations.)

For example, for this script (http://paste.yt-project.org/show/5808/ <http://paste.yt-project.org/show/5808/>), on my local machine the output shows:

``` yt : [INFO ] 2015-08-10 14:48:15,401 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:48:15,751 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:48:16,716 Saving plot snapshot_200_Projection_z_density.png ```

However, on the server the output shows: ``` yt : [INFO ] 2015-08-10 14:51:25,551 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800 yt : [INFO ] 2015-08-10 14:51:25,917 Saving plot snapshot_200_Projection_z_density.png yt : [INFO ] 2015-08-10 14:56:21,922 Saving plot snapshot_200_Projection_z_density.png ```

That’s 296 times slower!!!

This is likely to be a very specific problem with the server. But I’d like to know if anyone else has encountered similar problems before so I can learn from your experience. For now, I have no idea what could have gone wrong.

BTW, I’m using TSCC (http://idi.ucsd.edu/computing/ < http://idi.ucsd.edu/computing/>) at UC San Diego, if you happen to know it.

Thanks, Bili

Hi Bili, make sure you pass "-DNDEBUG" to CPPFLAGS when you compile matplotlib. Without it mpl is a good cpu cycle burner. There's a debug routine in triangulation module that's enabled by default. Cheers, Kacper

...

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- John Wise Assistant Professor of Physics Center for Relativistic Astrophysics, Georgia Tech http://cosmo.gatech.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-- John Wise Assistant Professor of Physics Center for Relativistic Astrophysics, Georgia Tech http://cosmo.gatech.edu _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org