Mailman 3 variance versus standard deviation - yt-users - python.org

newer
Scheduled maintenance

variance versus standard deviation

older
How the field ('gas',...

Andrew Cunningham

31 Dec 2016 31 Dec '16

8:50 p.m.

In the following cookbook example: http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance-... the variance along the profile is extracted: variance = prof.variance['gas', 'velocity_magnitude'].value but is then labelled as the standard deviation when the plot is created: plt.loglog(radius, variance, label='Standard Deviation') So, does "prof.variance" return the standard deviation and not the variance? Or, is the plot mislabelled? Should the plot instead be: plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

Attachments:

attachment.html (text/html — 1.8 KB)

Reply

Sign in to reply online Use email software

Show replies by thread

Nathan Goldbaum

1 Jan 1 Jan

10:50 a.m.

Hi Andrew, I think the code is simply using the terms "variance" and "standard deviation" interchangeably here. You can see how the `variance` array attached to the profile object is calculated here: https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183 I haven't looked up the mathematical definitions in detail, but I believe that this is the sample variance at each profile bin (*not* the standard deviation of the mean). If you'd like, please feel free to send a pull request to update that cookbook recipe to label the variance with "variance" instead of "standard deviation". The code for the cookbook recipe is located here in the repository: https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default On Sun, Jan 1, 2017 at 8:30 AM Andrew Cunningham wrote:

In the following cookbook example:

http://yt-project.org/doc/cookbook/simple_plots.html# profiles-with-variance-values

the variance along the profile is extracted:

variance = prof.variance['gas', 'velocity_magnitude'].value

but is then labelled as the standard deviation when the plot is created:

plt.loglog(radius, variance, label='Standard Deviation')

So, does "prof.variance" return the standard deviation and not the variance? Or, is the plot mislabelled? Should the plot instead be:

plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

_______________________________________________

yt-users mailing list

yt-users@lists.spacepope.org

http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Reply

Sign in to reply online Use email software

Andrew Cunningham

3 Jan 3 Jan

5:43 p.m.

Nathan, Thanks for pointing where to look for the variance computation. I see in the profiles.py you linked to, line 192 in "_finalize_storage" all_var = np.sqrt(all_var) which later is used to construct "self.variance". So, after this line, "all_var" is really a standard deviation. Consequently, the example in the cookbook is correct. The variance member of the profiles object is really the standard deviation. There is no "bug" in the cookbook example. Rather, this particular data member in the profile object is just misleadingly named. I don't currently have a dev version of yt setup. I'm more of a humble user than agile developer into the yt innards. I'd be happy to set that up and submit a pull request if that would be helpful, but I'm not sure what the "right" way to fix this is. One could remove the line: all_var = np.sqrt(all_var) so that the variance member is really the variance and update the cookbook example to take the square root inline when making the plot of mean and st dev. Though, that would break user scripts already out in the wild that rely on "variance" really being the standard deviation.

Hi Andrew,

I think the code is simply using the terms "variance" and "standard deviation" interchangeably here.

You can see how the `variance` array attached to the profile object is calculated here:

https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183

I haven't looked up the mathematical definitions in detail, but I believe that this is the sample variance at each profile bin (*not* the standard deviation of the mean).

If you'd like, please feel free to send a pull request to update that cookbook recipe to label the variance with "variance" instead of "standard deviation". The code for the cookbook recipe is located here in the repository:

https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default

On Sat, 31 Dec 2016, Andrew Cunningham wrote:

In the following cookbook example:

http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance -values

the variance along the profile is extracted:

variance = prof.variance['gas', 'velocity_magnitude'].value

but is then labelled as the standard deviation when the plot is created:

plt.loglog(radius, variance, label='Standard Deviation')

So, does "prof.variance" return the standard deviation and not the variance? Or, is the plot mislabelled? Should the plot instead be:

plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

Reply

Sign in to reply online Use email software

Nathan Goldbaum

5:52 p.m.

Ah, I see why you mean. Changing the name here would change the public-facing API, so I'm not sure there is a quick, easy fix here. If you'd like, feel free to open an issue about this so we don't lose track: https://bitbucket.org/yt_analysis/yt/issues/new On Tue, Jan 3, 2017 at 4:43 PM Andrew Cunningham wrote:

Nathan,

Thanks for pointing where to look for the variance computation. I see in

the profiles.py you linked to, line 192 in "_finalize_storage"

all_var = np.sqrt(all_var)

which later is used to construct "self.variance". So, after this line,

"all_var" is really a standard deviation.

Consequently, the example in the cookbook is correct. The variance member

of the profiles object is really the standard deviation. There is no

"bug" in the cookbook example. Rather, this particular data member in the

profile object is just misleadingly named.

I don't currently have a dev version of yt setup. I'm more of a humble

user than agile developer into the yt innards. I'd be happy to set that

up and submit a pull request if that would be helpful, but I'm not sure

what the "right" way to fix this is.

One could remove the line:

all_var = np.sqrt(all_var)

so that the variance member is really the variance and update the cookbook

example to take the square root inline when making the plot of mean and st

dev. Though, that would break user scripts already out in the wild that

rely on "variance" really being the standard deviation.

...
Hi Andrew,

...
...
I think the code is simply using the terms "variance" and "standard

...
deviation" interchangeably here.

...
...
You can see how the `variance` array attached to the profile object is

...
calculated here:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183

...
...
I haven't looked up the mathematical definitions in detail, but I believe

...
that this is the sample variance at each profile bin (*not* the standard

...
deviation of the mean).

...
...
If you'd like, please feel free to send a pull request to update that

...
cookbook recipe to label the variance with "variance" instead of

...
"standard deviation". The code for the cookbook recipe is located here

...
in the repository:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default

On Sat, 31 Dec 2016, Andrew Cunningham wrote:

...
In the following cookbook example:

...
...
http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance

...
-values

...
...
the variance along the profile is extracted:

...
...
variance = prof.variance['gas', 'velocity_magnitude'].value

...
...
but is then labelled as the standard deviation when the plot is created:

...
...
plt.loglog(radius, variance, label='Standard Deviation')

...
...
So, does "prof.variance" return the standard deviation and not the

...
variance? Or, is the plot mislabelled? Should the plot instead be:

...
...
plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

...
...
...
_______________________________________________

yt-users mailing list

yt-users@lists.spacepope.org

http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Reply

Sign in to reply online Use email software

Greg Bryan

4 Jan 4 Jan

12:04 a.m.

I would argue that you should treat this like a bug and just fix it. The definitions are very standard and this violates the rule of least surprise (it also conflicts with numpy usage). This will break some existing scripts but, speaking from painful experience, it is better to correct obvious mistakes earlier rather than later… Cheers, Greg

On Jan 3, 2017, at 5:52 PM, Nathan Goldbaum wrote:

Ah, I see why you mean. Changing the name here would change the public-facing API, so I'm not sure there is a quick, easy fix here.

If you'd like, feel free to open an issue about this so we don't lose track:

https://bitbucket.org/yt_analysis/yt/issues/new https://bitbucket.org/yt_analysis/yt/issues/new

On Tue, Jan 3, 2017 at 4:43 PM Andrew Cunningham mailto:ajcunn@gmail.com> wrote: Nathan,

Thanks for pointing where to look for the variance computation. I see in

the profiles.py you linked to, line 192 in "_finalize_storage"

all_var = np.sqrt(all_var)

which later is used to construct "self.variance". So, after this line,

"all_var" is really a standard deviation.

Consequently, the example in the cookbook is correct. The variance member

of the profiles object is really the standard deviation. There is no

"bug" in the cookbook example. Rather, this particular data member in the

profile object is just misleadingly named.

I don't currently have a dev version of yt setup. I'm more of a humble

user than agile developer into the yt innards. I'd be happy to set that

up and submit a pull request if that would be helpful, but I'm not sure

what the "right" way to fix this is.

One could remove the line:

all_var = np.sqrt(all_var)

so that the variance member is really the variance and update the cookbook

example to take the square root inline when making the plot of mean and st

dev. Though, that would break user scripts already out in the wild that

rely on "variance" really being the standard deviation.

...
Hi Andrew,

...
...
I think the code is simply using the terms "variance" and "standard

...
deviation" interchangeably here.

...
...
You can see how the `variance` array attached to the profile object is

...
calculated here:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183 https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183

...
...
I haven't looked up the mathematical definitions in detail, but I believe

...
that this is the sample variance at each profile bin (*not* the standard

...
deviation of the mean).

...
...
If you'd like, please feel free to send a pull request to update that

...
cookbook recipe to label the variance with "variance" instead of

...
"standard deviation". The code for the cookbook recipe is located here

...
in the repository:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default

On Sat, 31 Dec 2016, Andrew Cunningham wrote:

...
In the following cookbook example:

...
...
http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance

...
-values

...
...
the variance along the profile is extracted:

...
...
variance = prof.variance['gas', 'velocity_magnitude'].value

...
...
but is then labelled as the standard deviation when the plot is created:

...
...
plt.loglog(radius, variance, label='Standard Deviation')

...
...
So, does "prof.variance" return the standard deviation and not the

...
variance? Or, is the plot mislabelled? Should the plot instead be:

...
...
plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

...
...
...
_______________________________________________

yt-users mailing list

yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org

http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Reply

Sign in to reply online Use email software

Nathan Goldbaum

12:12 a.m.

I agree it should be fixed (albeit with backward compatibility shims so profile.variance still works and raises a deprecation warning so that scripts don't break). If one of you could open a bug report about this that would be very much appreciated. On Tue, Jan 3, 2017 at 11:07 PM Greg Bryan wrote:

I would argue that you should treat this like a bug and just fix it. The definitions are very standard and this violates the rule of least surprise (it also conflicts with numpy usage). This will break some existing scripts but, speaking from painful experience, it is better to correct obvious mistakes earlier rather than later…

Cheers, Greg

On Jan 3, 2017, at 5:52 PM, Nathan Goldbaum wrote:

Ah, I see why you mean. Changing the name here would change the public-facing API, so I'm not sure there is a quick, easy fix here.

If you'd like, feel free to open an issue about this so we don't lose track:

https://bitbucket.org/yt_analysis/yt/issues/new

On Tue, Jan 3, 2017 at 4:43 PM Andrew Cunningham wrote:

Nathan,

Thanks for pointing where to look for the variance computation. I see in

the profiles.py you linked to, line 192 in "_finalize_storage"

all_var = np.sqrt(all_var)

which later is used to construct "self.variance". So, after this line,

"all_var" is really a standard deviation.

Consequently, the example in the cookbook is correct. The variance member

of the profiles object is really the standard deviation. There is no

"bug" in the cookbook example. Rather, this particular data member in the

profile object is just misleadingly named.

I don't currently have a dev version of yt setup. I'm more of a humble

user than agile developer into the yt innards. I'd be happy to set that

up and submit a pull request if that would be helpful, but I'm not sure

what the "right" way to fix this is.

One could remove the line:

all_var = np.sqrt(all_var)

so that the variance member is really the variance and update the cookbook

example to take the square root inline when making the plot of mean and st

dev. Though, that would break user scripts already out in the wild that

rely on "variance" really being the standard deviation.

...
Hi Andrew,

...
...
I think the code is simply using the terms "variance" and "standard

...
deviation" interchangeably here.

...
...
You can see how the `variance` array attached to the profile object is

...
calculated here:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/yt/data_objects/profiles.py?at=yt&fileviewer=file-view-default#profiles.py-183

...
...
I haven't looked up the mathematical definitions in detail, but I believe

...
that this is the sample variance at each profile bin (*not* the standard

...
deviation of the mean).

...
...
If you'd like, please feel free to send a pull request to update that

...
cookbook recipe to label the variance with "variance" instead of

...
"standard deviation". The code for the cookbook recipe is located here

...
in the repository:

...
...
https://bitbucket.org/yt_analysis/yt/src/09f0ef297d7068078a021fc8290d9e3519baf82d/doc/source/cookbook/profile_with_variance.py?at=yt&fileviewer=file-view-default

On Sat, 31 Dec 2016, Andrew Cunningham wrote:

...
In the following cookbook example:

...
...
http://yt-project.org/doc/cookbook/simple_plots.html#profiles-with-variance

...
-values

...
...
the variance along the profile is extracted:

...
...
variance = prof.variance['gas', 'velocity_magnitude'].value

...
...
but is then labelled as the standard deviation when the plot is created:

...
...
plt.loglog(radius, variance, label='Standard Deviation')

...
...
So, does "prof.variance" return the standard deviation and not the

...
variance? Or, is the plot mislabelled? Should the plot instead be:

...
...
plt.loglog(radius, np.sqrt(variance), label='Standard Deviation')

...
...
...
_______________________________________________

yt-users mailing list

yt-users@lists.spacepope.org

http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________

yt-users mailing list

yt-users@lists.spacepope.org

http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Reply

Sign in to reply online Use email software

2666

Age (days ago)

2669

Last active (days ago)

Download

5 comments

3 participants

tags

participants (3)

Andrew Cunningham
Greg Bryan
Nathan Goldbaum