From sylvain.corlay at gmail.com  Sat Feb  3 11:11:24 2018
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sat, 3 Feb 2018 17:11:24 +0100
Subject: [Numpy-discussion] Extending C with Python
In-Reply-To: <ee26149f-90a9-a447-41c5-49bd9ad9ae96@seefeld.name>
References: <CAOWawpEP2wNXMTjkdtZeO5vwtVbxm8Yx9dAFv_Zno5i27=XMPQ@mail.gmail.com>
 <CAF6FJitWBbq9rntGk9gL5XAry5AHpjhizT+MYJAu6gFEYMiWjA@mail.gmail.com>
 <CAOWawpFwLAtKw579Qi+BcOYAVLKDk9Sb9_e0WnVOKxr81A+sJg@mail.gmail.com>
 <CALGmxEKjxhN4apbKfnbNzNcBbAoB4WChxU2R301M2hymkKvuyQ@mail.gmail.com>
 <ee26149f-90a9-a447-41c5-49bd9ad9ae96@seefeld.name>
Message-ID: <CAK=Phk4UZVVhfUyCYr4fJCZWDqQJGad8MDdY+wvyj6VAUb_90Q@mail.gmail.com>

You can also check out pybind11, xtensor, and xtensor-python

The latter enables a high-level numpy-like API on the C++ side.

You can check out the numpy to xtensor cheat sheet:

http://xtensor.readthedocs.io/en/latest/numpy.html

Best,

Sylvain


On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld <stefan at seefeld.name> wrote:

> On 31.01.2018 17:58, Chris Barker wrote:
>
> I'm guessing you could use Cython to make this easier.
>
>
> ... or Boost.Python (http://boostorg.github.io/python), which has
> built-in support for NumPy (http://boostorg.github.io/
> python/doc/html/numpy/index.html), and supports both directions:
> extending Python with C++, as well as embedding Python into C++
> applications.
>
>
> [image: Stefan]
>
> --
>
>       ...ich hab' noch einen Koffer in Berlin...
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180203/d0b98559/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.png
Type: image/png
Size: 1478 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180203/d0b98559/attachment.png>

From jalnliu at lbl.gov  Sat Feb  3 13:29:56 2018
From: jalnliu at lbl.gov (Jialin Liu)
Date: Sat, 3 Feb 2018 10:29:56 -0800
Subject: [Numpy-discussion] Extending C with Python
In-Reply-To: <CAK=Phk4UZVVhfUyCYr4fJCZWDqQJGad8MDdY+wvyj6VAUb_90Q@mail.gmail.com>
References: <CAOWawpEP2wNXMTjkdtZeO5vwtVbxm8Yx9dAFv_Zno5i27=XMPQ@mail.gmail.com>
 <CAF6FJitWBbq9rntGk9gL5XAry5AHpjhizT+MYJAu6gFEYMiWjA@mail.gmail.com>
 <CAOWawpFwLAtKw579Qi+BcOYAVLKDk9Sb9_e0WnVOKxr81A+sJg@mail.gmail.com>
 <CALGmxEKjxhN4apbKfnbNzNcBbAoB4WChxU2R301M2hymkKvuyQ@mail.gmail.com>
 <ee26149f-90a9-a447-41c5-49bd9ad9ae96@seefeld.name>
 <CAK=Phk4UZVVhfUyCYr4fJCZWDqQJGad8MDdY+wvyj6VAUb_90Q@mail.gmail.com>
Message-ID: <7E5AECC1-1AF1-4752-B563-165B4235A0F2@lbl.gov>

Thank you guys.

Best,
Jialin 

Sent from my iPhone

> On Feb 3, 2018, at 8:11 AM, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> 
> You can also check out pybind11, xtensor, and xtensor-python
> 
> The latter enables a high-level numpy-like API on the C++ side.
> 
> You can check out the numpy to xtensor cheat sheet:
> 
> http://xtensor.readthedocs.io/en/latest/numpy.html
> 
> Best,
> 
> Sylvain
> 
> 
>> On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld <stefan at seefeld.name> wrote:
>>> On 31.01.2018 17:58, Chris Barker wrote:
>>> I'm guessing you could use Cython to make this easier.
>> 
>> ... or Boost.Python (http://boostorg.github.io/python), which has built-in support for NumPy (http://boostorg.github.io/python/doc/html/numpy/index.html), and supports both directions: extending Python with C++, as well as embedding Python into C++ applications.
>> 
>> 
>> <signature.png>
>> -- 
>> 
>>       ...ich hab' noch einen Koffer in Berlin...
>>     
>> 
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180203/8bbcaa4f/attachment.html>

From jalnliu at lbl.gov  Tue Feb  6 13:40:12 2018
From: jalnliu at lbl.gov (Jialin Liu)
Date: Tue, 6 Feb 2018 10:40:12 -0800
Subject: [Numpy-discussion] Extending C with Python
In-Reply-To: <7E5AECC1-1AF1-4752-B563-165B4235A0F2@lbl.gov>
References: <CAOWawpEP2wNXMTjkdtZeO5vwtVbxm8Yx9dAFv_Zno5i27=XMPQ@mail.gmail.com>
 <CAF6FJitWBbq9rntGk9gL5XAry5AHpjhizT+MYJAu6gFEYMiWjA@mail.gmail.com>
 <CAOWawpFwLAtKw579Qi+BcOYAVLKDk9Sb9_e0WnVOKxr81A+sJg@mail.gmail.com>
 <CALGmxEKjxhN4apbKfnbNzNcBbAoB4WChxU2R301M2hymkKvuyQ@mail.gmail.com>
 <ee26149f-90a9-a447-41c5-49bd9ad9ae96@seefeld.name>
 <CAK=Phk4UZVVhfUyCYr4fJCZWDqQJGad8MDdY+wvyj6VAUb_90Q@mail.gmail.com>
 <7E5AECC1-1AF1-4752-B563-165B4235A0F2@lbl.gov>
Message-ID: <CAOWawpFAGPE0J5CGnmmwz5BD7upFuq_fMe9_UeXr8OJn2SgVjw@mail.gmail.com>

With PyObject_CallMethod(pInstance, method_name, "O", py_dims);

Can I pass in a reference and modify its content in python?

Best,
Jialin

On Sat, Feb 3, 2018 at 10:29 AM, Jialin Liu <jalnliu at lbl.gov> wrote:

> Thank you guys.
>
> Best,
> Jialin
>
> Sent from my iPhone
>
> On Feb 3, 2018, at 8:11 AM, Sylvain Corlay <sylvain.corlay at gmail.com>
> wrote:
>
> You can also check out pybind11, xtensor, and xtensor-python
>
> The latter enables a high-level numpy-like API on the C++ side.
>
> You can check out the numpy to xtensor cheat sheet:
>
> http://xtensor.readthedocs.io/en/latest/numpy.html
>
> Best,
>
> Sylvain
>
>
> On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld <stefan at seefeld.name>
> wrote:
>
>> On 31.01.2018 17:58, Chris Barker wrote:
>>
>> I'm guessing you could use Cython to make this easier.
>>
>>
>> ... or Boost.Python (http://boostorg.github.io/python), which has
>> built-in support for NumPy (http://boostorg.github.io/pyt
>> hon/doc/html/numpy/index.html), and supports both directions: extending
>> Python with C++, as well as embedding Python into C++ applications.
>>
>>
>> <signature.png>
>>
>> --
>>
>>       ...ich hab' noch einen Koffer in Berlin...
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180206/96fd308b/attachment.html>

From ben.v.root at gmail.com  Tue Feb  6 21:57:24 2018
From: ben.v.root at gmail.com (Benjamin Root)
Date: Tue, 6 Feb 2018 21:57:24 -0500
Subject: [Numpy-discussion] improving arange()? introducing fma()?
Message-ID: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>

Note, the following is partly to document my explorations in computing
lat/on grids in numpy lately, in case others come across these issues in
the future. It is partly a plea for some development of numerically
accurate functions for computing lat/lon grids from a combination of
inputs: bounds, counts, and resolutions.

I have been playing around with the decimal package a bit lately, and I
discovered the concept of "fused multiply-add" operations for improved
accuracy. I have come to realize that fma operations could be used to
greatly improve the accuracy of linspace() and arange(). In particular, I
have been needing improved results for computing latitude/longitude grids,
which tend to be done in float32's to save memory (at least, this is true
in data I come across).

Since fma is not available yet in python, please consider the following C
snippet:

```
$ cat test_fma.c
#include<stdlib.h>
#include<math.h>
#include<stdio.h>

int main(){
    float res = 0.01;
    float x0 = -115.0;
    int cnt = 7001;
    float x1 = res * cnt + x0;
    float x2 = fma(res, cnt, x0);
    printf("x1 %.16f  x2 %.16f\n", x1, x2);
    float err1 = fabs(x1 - -44.990000000000000);
    float err2 = fabs(x2 - -44.990000000000000);
    printf("err1 %.5e  err2 %.5e\n", err1, err2);
    return 0;
}
$ gcc test_fma.c -lm -o test_fma
$ ./test_fma
x1 -44.9899978637695312  x2 -44.9900016784667969
err1 2.13623e-06  err2 1.67847e-06
```

And if you do similar for double-precision, the fma still yields
significant accuracy over double precision explicit multiply-add.

Now, to the crux of my problem. It is next to impossible to generate a
non-trivial numpy array of coordinates, even in double precision, without
hitting significant numerical errors. Which has lead me down the path of
using the decimal package (which doesn't play very nicely with numpy
because of the lack of casting rules for it). Consider the following:
```
$ cat test_fma.py
from __future__ import print_function
import numpy as np
res = np.float32(0.01)
cnt = 7001
x0 = np.float32(-115.0)
x1 = res * cnt + x0
print("res * cnt + x0 = %.16f" % x1)
x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
print("linspace()[-1]: %.16f" % x[-1])

$ python test_fma.py
res * cnt + x0 = -44.9900015648454428
len(arange()): 7002  arange()[-1]:       -44.975044
linspace()[-1]: -44.9900016784667969
```
arange just produces silly results (puts out an extra element... adding
half of the resolution is typically mentioned as a solution on mailing
lists to get around arange()'s limitations -- I personally don't do this).

linspace() is a tad bit better than arange(), but still a little bit worse
than just computing the value straight out. It also produces a result that
is on par with fma(), so that is reassuring that it is about as accurate as
can be at the moment. This also matches up almost exactly to what I would
get if I used Decimal()s and then casted to float32's for final storage, so
that's not too bad as well.

So, does it make any sense to improve arange by utilizing fma() under the
hood? Also, any plans for making fma() available as a ufunc?

Notice that most of my examples required knowing the number of grid points
ahead of time. But what if I didn't know that? What if I just have the
bounds and the resolution? Then arange() is the natural fit, but as I
showed, its accuracy is lacking, and you have to do some sort of hack to do
a closed interval.

Thoughts? Comments?
Ben Root
(donning flame suit)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180206/fcda5f5d/attachment.html>

From ralf.gommers at gmail.com  Wed Feb  7 01:09:45 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 7 Feb 2018 06:09:45 +0000
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
Message-ID: <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>

On Wed, Feb 7, 2018 at 2:57 AM, Benjamin Root <ben.v.root at gmail.com> wrote:

> Note, the following is partly to document my explorations in computing
> lat/on grids in numpy lately, in case others come across these issues in
> the future. It is partly a plea for some development of numerically
> accurate functions for computing lat/lon grids from a combination of
> inputs: bounds, counts, and resolutions.
>

You're talking about accuracy several times, it would help to define what
your actual requirements are. And a quantification of what the accuracy
loss currently is. I wouldn't expect it to me more than a few ulp for
float64.


>
> I have been playing around with the decimal package a bit lately, and I
> discovered the concept of "fused multiply-add" operations for improved
> accuracy. I have come to realize that fma operations could be used to
> greatly improve the accuracy of linspace() and arange(). In particular, I
> have been needing improved results for computing latitude/longitude grids,
> which tend to be done in float32's to save memory (at least, this is true
> in data I come across).
>

If you care about saving memory *and* accuracy, wouldn't it make more sense
to do your computations in float64, and convert to float32 at the end? That
must be much more accurate than any new float32 algorithm you come up with.

Ralf



>
> Since fma is not available yet in python, please consider the following C
> snippet:
>
> ```
> $ cat test_fma.c
> #include<stdlib.h>
> #include<math.h>
> #include<stdio.h>
>
> int main(){
>     float res = 0.01;
>     float x0 = -115.0;
>     int cnt = 7001;
>     float x1 = res * cnt + x0;
>     float x2 = fma(res, cnt, x0);
>     printf("x1 %.16f  x2 %.16f\n", x1, x2);
>     float err1 = fabs(x1 - -44.990000000000000);
>     float err2 = fabs(x2 - -44.990000000000000);
>     printf("err1 %.5e  err2 %.5e\n", err1, err2);
>     return 0;
> }
> $ gcc test_fma.c -lm -o test_fma
> $ ./test_fma
> x1 -44.9899978637695312  x2 -44.9900016784667969
> err1 2.13623e-06  err2 1.67847e-06
> ```
>
> And if you do similar for double-precision, the fma still yields
> significant accuracy over double precision explicit multiply-add.
>
> Now, to the crux of my problem. It is next to impossible to generate a
> non-trivial numpy array of coordinates, even in double precision, without
> hitting significant numerical errors. Which has lead me down the path of
> using the decimal package (which doesn't play very nicely with numpy
> because of the lack of casting rules for it). Consider the following:
> ```
> $ cat test_fma.py
> from __future__ import print_function
> import numpy as np
> res = np.float32(0.01)
> cnt = 7001
> x0 = np.float32(-115.0)
> x1 = res * cnt + x0
> print("res * cnt + x0 = %.16f" % x1)
> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
> print("linspace()[-1]: %.16f" % x[-1])
>
> $ python test_fma.py
> res * cnt + x0 = -44.9900015648454428
> len(arange()): 7002  arange()[-1]:       -44.975044
> linspace()[-1]: -44.9900016784667969
> ```
> arange just produces silly results (puts out an extra element... adding
> half of the resolution is typically mentioned as a solution on mailing
> lists to get around arange()'s limitations -- I personally don't do this).
>
> linspace() is a tad bit better than arange(), but still a little bit worse
> than just computing the value straight out. It also produces a result that
> is on par with fma(), so that is reassuring that it is about as accurate as
> can be at the moment. This also matches up almost exactly to what I would
> get if I used Decimal()s and then casted to float32's for final storage, so
> that's not too bad as well.
>
> So, does it make any sense to improve arange by utilizing fma() under the
> hood? Also, any plans for making fma() available as a ufunc?
>
> Notice that most of my examples required knowing the number of grid points
> ahead of time. But what if I didn't know that? What if I just have the
> bounds and the resolution? Then arange() is the natural fit, but as I
> showed, its accuracy is lacking, and you have to do some sort of hack to do
> a closed interval.
>
> Thoughts? Comments?
> Ben Root
> (donning flame suit)
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180207/84791d65/attachment-0001.html>

From charlesr.harris at gmail.com  Wed Feb  7 16:26:37 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 7 Feb 2018 14:26:37 -0700
Subject: [Numpy-discussion] Hoop jumping, and other sports
Message-ID: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>

Hi All,

I was thinking about things to do to simplify the NumPy development
process. One thing that came to mind was our use of prefixes on commits,
BUG, TST, etc. Those prefixes were originally introduced by David
Cournapeau when he was managing releases in order help him track commits
that might need backports. I like the prefixes, but now that we are
organized by PRs, rather than commits, the only place we really need them,
for some meaning of "need", is in the commit titles, and maintainers can
change and edit those without problems. So I would like to propose that we
no longer be picky about having them in the commit summary line.
Furthermore, that got me thinking that there are probably other things we
could do to simplify the development process. So I'd like folks to weigh in
with other ideas for simplification or complaints about nit picky things
that have annoyed them.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180207/129320ec/attachment.html>

From dallan at bnl.gov  Wed Feb  7 17:02:10 2018
From: dallan at bnl.gov (Allan, Daniel)
Date: Wed, 7 Feb 2018 22:02:10 +0000
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
Message-ID: <505AC41CA3A34E47B70C7B5C794CBF071DAE2476@EXMB3.bnl.gov>

I think that makes sense. Incidentally, I had imitated numpy's abbreviated prefixes on various other projects until it was recently pointed out to me that abbreviations, especially more cryptic ones like "ENH", "REF", and "MNT", are a particularly tricky "hoop" to jump through for non-native speakers of English.

Daniel B. Allan, Ph.D
Associate Computational Scientist, Brookhaven National Lab
(631) 344-3281 (no voicemail set up)
________________________________
From: NumPy-Discussion [numpy-discussion-bounces+dallan=bnl.gov at python.org] on behalf of Charles R Harris [charlesr.harris at gmail.com]
Sent: Wednesday, February 07, 2018 4:26 PM
To: numpy-discussion
Subject: [Numpy-discussion] Hoop jumping, and other sports

Hi All,

I was thinking about things to do to simplify the NumPy development process. One thing that came to mind was our use of prefixes on commits, BUG, TST, etc. Those prefixes were originally introduced by David Cournapeau when he was managing releases in order help him track commits that might need backports. I like the prefixes, but now that we are organized by PRs, rather than commits, the only place we really need them, for some meaning of "need", is in the commit titles, and maintainers can change and edit those without problems. So I would like to propose that we no longer be picky about having them in the commit summary line. Furthermore, that got me thinking that there are probably other things we could do to simplify the development process. So I'd like folks to weigh in with other ideas for simplification or complaints about nit picky things that have annoyed them.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180207/d05f5eaa/attachment.html>

From andyfaff at gmail.com  Wed Feb  7 18:04:25 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Thu, 8 Feb 2018 10:04:25 +1100
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <505AC41CA3A34E47B70C7B5C794CBF071DAE2476@EXMB3.bnl.gov>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <505AC41CA3A34E47B70C7B5C794CBF071DAE2476@EXMB3.bnl.gov>
Message-ID: <CAAbtOZeOOCDWO9VHKifc7EW8S2CpkqYpq3rbCTjf08yfcHGqKw@mail.gmail.com>

Other factors hindering new contributors:

1) Being unfamiliar with git. e.g. knowing that you have to fork numpy
first, then clone from your fork, then create a feature branch. That's just
the """straightforward bit""". It's hell the first time you have to rebase
something/correct commit messages/squash, etc.

2) I have a mindblock on the correct way to use ReStructured Text in
docstrings. When do I use backticks/double backticks, etc. Getting
documentation changes to perfection is hard.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180208/42953e02/attachment.html>

From stefanv at berkeley.edu  Wed Feb  7 18:29:20 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Wed, 7 Feb 2018 15:29:20 -0800
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
Message-ID: <20180207232919.mzsvnszi3nt7rtuh@fastmail.com>

On Wed, 07 Feb 2018 14:26:37 -0700, Charles R Harris wrote:
> I was thinking about things to do to simplify the NumPy development
> process. One thing that came to mind was our use of prefixes on commits,
> BUG, TST, etc. Those prefixes were originally introduced by David
> Cournapeau when he was managing releases in order help him track commits
> that might need backports. I like the prefixes, but now that we are
> organized by PRs, rather than commits, the only place we really need them,
> for some meaning of "need", is in the commit titles, and maintainers can
> change and edit those without problems. So I would like to propose that we
> no longer be picky about having them in the commit summary line.
> Furthermore, that got me thinking that there are probably other things we
> could do to simplify the development process. So I'd like folks to weigh in
> with other ideas for simplification or complaints about nit picky things
> that have annoyed them.

For scikit-image, we've started a policy of pushing minor edits
(spelling corrections, sentence restructuring, etc.) directly to the PR
branch, instead of flagging those during a review (we also have a PEP8
bot that mentions PEP8 issues, which makes the human reviews less
nitpicky).

To avoid users having to rebase, we now squash all PRs before merge
(and, typically, remove the commit messages).  This also means we can
allow merges with master to exist in the PR history.  Of course, if it
is clear that a user took particular care to hand-craft each patch we
don't squash, but those cases seem to be in the minority.

Thank you for bringing this up, Chuck; on projects with very limited
developer hours (almost any open source project?) whatever we can do to
lower the contribution barrier helps a great deal.

Best regards
St?fan

From allanhaldane at gmail.com  Wed Feb  7 18:35:23 2018
From: allanhaldane at gmail.com (Allan Haldane)
Date: Wed, 7 Feb 2018 18:35:23 -0500
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
Message-ID: <5eea72e7-8e15-72a6-fdea-3e534f068d19@gmail.com>

On 02/07/2018 04:26 PM, Charles R Harris wrote:
> Hi All,
> 
> I was thinking about things to do to simplify the NumPy development
> process. One thing that came to mind was our use of prefixes on commits,
> BUG, TST, etc. Those prefixes were originally introduced by David
> Cournapeau when he was managing releases in order help him track commits
> that might need backports. I like the prefixes, but now that we are
> organized by PRs, rather than commits, the only place we really need
> them, for some meaning of "need", is in the commit titles, and
> maintainers can change and edit those without problems. So I would like
> to propose that we no longer be picky about having them in the commit
> summary line. Furthermore, that got me thinking that there are probably
> other things we could do to simplify the development process. So I'd
> like folks to weigh in with other ideas for simplification or complaints
> about nit picky things that have annoyed them.
> 
> Chuck

When I was first contributing, the main obstacle was not the nitpicks
but reading through all the contributor guidelines pages, as well as
learning github. I also remember finding it hard to find that
documentation in the first place.

It is at
https://docs.scipy.org/doc/numpy/dev/index.html
and a shorter summary at
https://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html

Maybe we should have a much more prominent link about how to contribute,
eg on the main "README.md" front page, or at the start of the user
guide, which links to a "really really" short contributing guide for
someone who does not use github, maybe a screenful or two only. Even the
short development workflow above has lots of info that usually isn't
needed and takes a long time to read through.

Allan

From deak.andris at gmail.com  Wed Feb  7 21:03:41 2018
From: deak.andris at gmail.com (Andras Deak)
Date: Thu, 8 Feb 2018 03:03:41 +0100
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <5eea72e7-8e15-72a6-fdea-3e534f068d19@gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <5eea72e7-8e15-72a6-fdea-3e534f068d19@gmail.com>
Message-ID: <CAMEWA4Moc2kBmKriBC+Y3KWUZY15ThZdgndg8Yyd7WiJT27Xzw@mail.gmail.com>

On Thu, Feb 8, 2018 at 12:35 AM, Allan Haldane <allanhaldane at gmail.com> wrote:
> On 02/07/2018 04:26 PM, Charles R Harris wrote:
>> Hi All,
>>
>> I was thinking about things to do to simplify the NumPy development
>> process. One thing that came to mind was our use of prefixes on commits,
>> BUG, TST, etc. Those prefixes were originally introduced by David
>> Cournapeau when he was managing releases in order help him track commits
>> that might need backports. I like the prefixes, but now that we are
>> organized by PRs, rather than commits, the only place we really need
>> them, for some meaning of "need", is in the commit titles, and
>> maintainers can change and edit those without problems. So I would like
>> to propose that we no longer be picky about having them in the commit
>> summary line. Furthermore, that got me thinking that there are probably
>> other things we could do to simplify the development process. So I'd
>> like folks to weigh in with other ideas for simplification or complaints
>> about nit picky things that have annoyed them.
>>
>> Chuck
>
> When I was first contributing, the main obstacle was not the nitpicks
> but reading through all the contributor guidelines pages, as well as
> learning github. I also remember finding it hard to find that
> documentation in the first place.
>
> It is at
> https://docs.scipy.org/doc/numpy/dev/index.html
> and a shorter summary at
> https://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html
>
> Maybe we should have a much more prominent link about how to contribute,
> eg on the main "README.md" front page, or at the start of the user
> guide, which links to a "really really" short contributing guide for
> someone who does not use github, maybe a screenful or two only. Even the
> short development workflow above has lots of info that usually isn't
> needed and takes a long time to read through.
>
> Allan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion

As a new (though so far superficial) contributor I can say that I
didn't find it difficult at all to locate the resources needed for
contributing. I found the gitwash link very easily and naturally, and
I didn't find it too long nor confusing (but I did have a fresh
understanding of git itself, so I can't reliably assess the contents
from a git newcomer's standpoint). The willingness to touch numpy
without a thorough understanding of its internals is much more of a
barrier at least in my case.

Andr?s

From p.j.a.cock at googlemail.com  Thu Feb  8 04:52:46 2018
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Thu, 8 Feb 2018 09:52:46 +0000
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CAAbtOZeOOCDWO9VHKifc7EW8S2CpkqYpq3rbCTjf08yfcHGqKw@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <505AC41CA3A34E47B70C7B5C794CBF071DAE2476@EXMB3.bnl.gov>
 <CAAbtOZeOOCDWO9VHKifc7EW8S2CpkqYpq3rbCTjf08yfcHGqKw@mail.gmail.com>
Message-ID: <CAKVJ-_7WzxZpeV3VDnYmKZDbYUh34dvmnmfsvoSYWWOD2bTvNA@mail.gmail.com>

On Wed, Feb 7, 2018 at 11:04 PM, Andrew Nelson <andyfaff at gmail.com> wrote:
> Other factors hindering new contributors:
>
> 1) Being unfamiliar with git. e.g. knowing that you have to fork numpy
> first, then clone from your fork, then create a feature branch. That's just
> the """straightforward bit""". It's hell the first time you have to rebase
> something/correct commit messages/squash, etc.

Is there anything specific to NumPy's use of git which is unsual
which needs particular warning for newcomers?

> 2) I have a mindblock on the correct way to use ReStructured Text in
> docstrings. When do I use backticks/double backticks, etc. Getting
> documentation changes to perfection is hard.

Me too. I still regularly copy-paste my docstrings from the Python
code into an RST editor to confirm the formatting, or do any
non-trivial editing.

I also found I'd often only discover invalid RST when trying to
build a project's documentation (using Sphinx or epydoc as
appropriate for the project at hand), so I wrong a flake8 plugin
to do basic reStructuredText validation:

https://pypi.python.org/pypi/flake8-rst-docstrings
https://github.com/peterjc/flake8-rst-docstrings

One could imagine checking NumPy's specific docstring
style with a more complicated flake8 plugin?

Peter

From ralf.gommers at gmail.com  Thu Feb  8 06:22:36 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 8 Feb 2018 11:22:36 +0000
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <20180207232919.mzsvnszi3nt7rtuh@fastmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <20180207232919.mzsvnszi3nt7rtuh@fastmail.com>
Message-ID: <CABL7CQixswVOLCdYPFb7NydZZw-LRSkgPxtw3Curfg0nw_zMFA@mail.gmail.com>

On Wed, Feb 7, 2018 at 11:29 PM, Stefan van der Walt <stefanv at berkeley.edu>
wrote:

> On Wed, 07 Feb 2018 14:26:37 -0700, Charles R Harris wrote:
> > I was thinking about things to do to simplify the NumPy development
> > process. One thing that came to mind was our use of prefixes on commits,
> > BUG, TST, etc. Those prefixes were originally introduced by David
> > Cournapeau when he was managing releases in order help him track commits
> > that might need backports. I like the prefixes, but now that we are
> > organized by PRs, rather than commits, the only place we really need
> them,
> > for some meaning of "need", is in the commit titles, and maintainers can
> > change and edit those without problems. So I would like to propose that
> we
> > no longer be picky about having them in the commit summary line.
>

+1 we got more picky over time, and that was never the intention. They're
still useful, but we shouldn't request rework for such a minor thing.

> Furthermore, that got me thinking that there are probably other things we
> > could do to simplify the development process. So I'd like folks to weigh
> in
> > with other ideas for simplification or complaints about nit picky things
> > that have annoyed them.
>
> For scikit-image, we've started a policy of pushing minor edits
> (spelling corrections, sentence restructuring, etc.) directly to the PR
> branch, instead of flagging those during a review (we also have a PEP8
> bot that mentions PEP8 issues, which makes the human reviews less
> nitpicky).
>

+1. Especially useful for docstring formatting, that doesn't get picked up
by a PEP8 bot

Ralf


> To avoid users having to rebase, we now squash all PRs before merge
> (and, typically, remove the commit messages).  This also means we can
> allow merges with master to exist in the PR history.  Of course, if it
> is clear that a user took particular care to hand-craft each patch we
> don't squash, but those cases seem to be in the minority.
>
> Thank you for bringing this up, Chuck; on projects with very limited
> developer hours (almost any open source project?) whatever we can do to
> lower the contribution barrier helps a great deal.
>
> Best regards
> St?fan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180208/e46b2369/attachment-0001.html>

From p.j.a.cock at googlemail.com  Thu Feb  8 06:32:45 2018
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Thu, 8 Feb 2018 11:32:45 +0000
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CABL7CQixswVOLCdYPFb7NydZZw-LRSkgPxtw3Curfg0nw_zMFA@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <20180207232919.mzsvnszi3nt7rtuh@fastmail.com>
 <CABL7CQixswVOLCdYPFb7NydZZw-LRSkgPxtw3Curfg0nw_zMFA@mail.gmail.com>
Message-ID: <CAKVJ-_6w4uZxK1u+pN-n=nuB=sWFTvMJTT0Nw+HsT8qucRovLA@mail.gmail.com>

On Thu, Feb 8, 2018 at 11:22 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> On Wed, Feb 7, 2018 at 11:29 PM, Stefan van der Walt <stefanv at berkeley.edu>
> wrote:
>>
>> For scikit-image, we've started a policy of pushing minor edits
>> (spelling corrections, sentence restructuring, etc.) directly to the PR
>> branch, instead of flagging those during a review (we also have a PEP8
>> bot that mentions PEP8 issues, which makes the human reviews less
>> nitpicky).
>
> +1. Especially useful for docstring formatting, that doesn't get picked up
> by a PEP8 bot
>
> Ralf

Were is the current NumPy PEP8 bot setup? I don't see flake8 or
similar under the TravisCI or AppVeyor setup - which is where I am
using a flake8 plugin to at least partially validate RST docstrings [*]
(and catch things automatically in pull requests).

Peter

[*] My plugin:

https://github.com/peterjc/flake8-rst-docstrings
https://pypi.python.org/pypi/flake8-rst-docstrings

From ralf.gommers at gmail.com  Thu Feb  8 10:37:08 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 8 Feb 2018 15:37:08 +0000
Subject: [Numpy-discussion] Hoop jumping, and other sports
In-Reply-To: <CAKVJ-_6w4uZxK1u+pN-n=nuB=sWFTvMJTT0Nw+HsT8qucRovLA@mail.gmail.com>
References: <CAB6mnx+3MMY844FHL0=Y2mBAFeiaQLe4SowgcGo+xqjVn1639Q@mail.gmail.com>
 <20180207232919.mzsvnszi3nt7rtuh@fastmail.com>
 <CABL7CQixswVOLCdYPFb7NydZZw-LRSkgPxtw3Curfg0nw_zMFA@mail.gmail.com>
 <CAKVJ-_6w4uZxK1u+pN-n=nuB=sWFTvMJTT0Nw+HsT8qucRovLA@mail.gmail.com>
Message-ID: <CABL7CQhcsvWtyYY0yW9cJS5enh=ZqUFM56=5rAjGmt9bwxXo7Q@mail.gmail.com>

On Thu, Feb 8, 2018 at 11:32 AM, Peter Cock <p.j.a.cock at googlemail.com>
wrote:

> On Thu, Feb 8, 2018 at 11:22 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> > On Wed, Feb 7, 2018 at 11:29 PM, Stefan van der Walt <
> stefanv at berkeley.edu>
> > wrote:
> >>
> >> For scikit-image, we've started a policy of pushing minor edits
> >> (spelling corrections, sentence restructuring, etc.) directly to the PR
> >> branch, instead of flagging those during a review (we also have a PEP8
> >> bot that mentions PEP8 issues, which makes the human reviews less
> >> nitpicky).
> >
> > +1. Especially useful for docstring formatting, that doesn't get picked
> up
> > by a PEP8 bot
> >
> > Ralf
>
> Were is the current NumPy PEP8 bot setup? I don't see flake8 or
> similar under the TravisCI or AppVeyor setup - which is where I am
> using a flake8 plugin to at least partially validate RST docstrings [*]
> (and catch things automatically in pull requests).
>

Good question. It's missing, I confused it with the scipy version:
https://github.com/scipy/scipy/blob/master/.travis.yml#L18. We should copy
that build matrix entry for numpy.

Ralf


>
> Peter
>
> [*] My plugin:
>
> https://github.com/peterjc/flake8-rst-docstrings
> https://pypi.python.org/pypi/flake8-rst-docstrings
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180208/fc5110aa/attachment.html>

From pi at berkeley.edu  Thu Feb  8 17:53:55 2018
From: pi at berkeley.edu (Paul Ivanov)
Date: Thu, 8 Feb 2018 14:53:55 -0800
Subject: [Numpy-discussion] SciPy 2018 - one week left for submissions
Message-ID: <CAHK2zoDs3GjKNKsdEwBTvK+vcrUF0xjXPbt3xHD_=cY1Z_Vw_g@mail.gmail.com>

SciPy 2018, the 17th annual Scientific Computing with Python conference,
will be held July 9-15, 2018 in Austin, Texas. The annual SciPyConference
brings together over 700 participants from industry, academia, and
government to showcase their latest projects, learn from skilled users and
developers, and collaborate on code development. The call for abstracts
for SciPy 2018 for talks, posters and tutorials is now open. The new
extended deadline for submissions is February 15, 2018.

Conference Website: https://scipy2018.scipy.org

Submission Website: https://easychair.org/conferences/?conf=scipy2018


*July 9-15, 2018 | Austin, Texas *



   - *Tutorials:* July 9-10, 2018
   - *Conference (Talks and Posters):* July 11-13, 2018
   - *Sprints: *July 14-15, 2018


In addition to the general track, this year will have specialized tracks
focused on:

- Data Visualization

- Reproducibility and Software Sustainability

*Mini Symposia*

?         Astronomy

?         Biology and Bioinformatics

?         Data Science

?         Earth, Ocean and Geo Science

?         Image Processing

?         Language Interoperability

?         Library Science and Digital Humanities

?         Machine Learning

?         Materials Science

?         Political and Social Sciences

There will also be a SciPy Tools Plenary Session each day with 2 to 5
minute updates on tools and libraries.


*Tutorials (July 9-10, 2018)*

Tutorials should be focused on covering a well-defined topic in a hands-on
manner. We are looking for awesome techniques or packages, helping new or
advanced Python programmers develop better or faster scientific
applications. We encourage submissions to be designed to allow at least 50%
of the time for hands-on exercises even if this means the subject matter
needs to be limited. Tutorials will be 4 hours in duration. In your
tutorial application, you can indicate what prerequisite skills and
knowledge will be needed for your tutorial, and the approximate expected
level of knowledge of your students (i.e., beginner, intermediate,
advanced). Instructors of accepted tutorials will receive a stipend.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180208/759e82b1/attachment.html>

From shoyer at gmail.com  Fri Feb  9 01:12:03 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Fri, 09 Feb 2018 06:12:03 +0000
Subject: [Numpy-discussion] NumPy should not silently promote numbers to
 strings
Message-ID: <CAEQ_TvfpLysca=UhZv1-gfJ+RoDb+BNFNcEm8B5ku8+TAG_MNw@mail.gmail.com>

This is one of my oldest NumPy pain-points:
>>> np.array([1, 2, 'three'])
array(['1', '2', 'three'],
      dtype='<U21')

This is almost never what I want. In many cases, I simply write
dtype=object, but for others (e.g., numpy.where), it's a minor annoyance to
explicitly cast inputs to the right type.

Autoconverting numbers into strings occasionally introduces real bugs
(e.g., where using `np.nan` as a sentinel value for NA when working with
strings, as in https://github.com/pydata/xarray/pull/1847), but mostly just
hides bugs until later. It's certainly very un-Pythonic.

The sane promotion rule would be `np.promote_types(str, float) -> object`,
not a size 32 string.

Is it way too late to fix this for NumPy, or is this something we could
change in a major release? It would certainly need at least a deprecation
cycle. This is easy enough to introduce accidentally that there are
undoubtedly many users whose code would break if we changed this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/ec1f9669/attachment.html>

From wieser.eric+numpy at gmail.com  Fri Feb  9 01:58:35 2018
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Fri, 09 Feb 2018 06:58:35 +0000
Subject: [Numpy-discussion] NumPy should not silently promote numbers to
 strings
In-Reply-To: <CAEQ_TvfpLysca=UhZv1-gfJ+RoDb+BNFNcEm8B5ku8+TAG_MNw@mail.gmail.com>
References: <CAEQ_TvfpLysca=UhZv1-gfJ+RoDb+BNFNcEm8B5ku8+TAG_MNw@mail.gmail.com>
Message-ID: <CAL1kJvBeN5B8mtosBCt2EXLAw+Bvsm8GzDkRwYEZSoPuY+ngZw@mail.gmail.com>

Presumably you would extend that to all (str, np.number), or even (str,
np.generic_)?
I suppose there?s the argument that with python-3-only support around the
corner, even (str, bytes) should go to object.

Right now, promote_types gives examples in the docs of int/string
conversions, so changing it might be tricky.
On the other hand, the docs also falsely claim that the conversion is
associative <https://github.com/numpy/numpy/pull/10554/files>, which your
proposed change would fix.
?

On Thu, 8 Feb 2018 at 22:12 Stephan Hoyer <shoyer at gmail.com> wrote:

> This is one of my oldest NumPy pain-points:
> >>> np.array([1, 2, 'three'])
> array(['1', '2', 'three'],
>       dtype='<U21')
>
> This is almost never what I want. In many cases, I simply write
> dtype=object, but for others (e.g., numpy.where), it's a minor annoyance to
> explicitly cast inputs to the right type.
>
> Autoconverting numbers into strings occasionally introduces real bugs
> (e.g., where using `np.nan` as a sentinel value for NA when working with
> strings, as in https://github.com/pydata/xarray/pull/1847), but mostly
> just hides bugs until later. It's certainly very un-Pythonic.
>
> The sane promotion rule would be `np.promote_types(str, float) -> object`,
> not a size 32 string.
>
> Is it way too late to fix this for NumPy, or is this something we could
> change in a major release? It would certainly need at least a deprecation
> cycle. This is easy enough to introduce accidentally that there are
> undoubtedly many users whose code would break if we changed this.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/65e0bb2a/attachment-0001.html>

From shoyer at gmail.com  Fri Feb  9 02:21:47 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Fri, 09 Feb 2018 07:21:47 +0000
Subject: [Numpy-discussion] NumPy should not silently promote numbers to
 strings
In-Reply-To: <CAL1kJvBeN5B8mtosBCt2EXLAw+Bvsm8GzDkRwYEZSoPuY+ngZw@mail.gmail.com>
References: <CAEQ_TvfpLysca=UhZv1-gfJ+RoDb+BNFNcEm8B5ku8+TAG_MNw@mail.gmail.com>
 <CAL1kJvBeN5B8mtosBCt2EXLAw+Bvsm8GzDkRwYEZSoPuY+ngZw@mail.gmail.com>
Message-ID: <CAEQ_TvchCoRbnDYGvSp81+kRT3oMKpJ53BEUDNNf1F+658xcBQ@mail.gmail.com>

On Thu, Feb 8, 2018 at 11:00 PM Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> Presumably you would extend that to all (str, np.number), or even (str,
> np.generic_)?
>
Yes, I'm currently doing (np.character, np.number) and (np.character,
np.bool_). But only in direct consultation with the diagram of NumPy's type
hierarchy :).

> I suppose there?s the argument that with python-3-only support around the
> corner, even (str, bytes) should go to object.
>
Yes, that's also pretty bad.

The current behavior (str, bytes) -> str relies on bytes being valid ASCII:
>>> np.array([b'\xFF', u'cd'])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
ordinal not in range(128)

It exactly matches Python 2's str/unicode behavior, but doesn't make sense
at all in a Python 3 world.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/c3956581/attachment.html>

From kirillbalunov at gmail.com  Fri Feb  9 05:30:49 2018
From: kirillbalunov at gmail.com (Kirill Balunov)
Date: Fri, 9 Feb 2018 13:30:49 +0300
Subject: [Numpy-discussion] dtype argument description for np.array
Message-ID: <CABwCVQAW2aDyJ=NZZp1ch3RF-DM-WrHj5Xfb6fjYZbpNbt4=cQ@mail.gmail.com>

Currently in docstring the description of dtype argument for np.array says
this:

dtype : data-type, optional
>     The desired data-type for the array.  If not given, then the type will
>     be determined as the minimum type required to hold the objects in the
>     sequence.  This argument can only be used to 'upcast' the array.  For
>     downcasting, use the .astype(t) method.
>

But I found this description somewhat misleading for integer types. Is this
generally true that "the type will
be determined as the minimum type required to hold the objects in the
sequence."?

As I always thought for integer arrays first `np.int_` is assumed and if it
is not enough the bigger dtype s are used, and finally falling to `object`
dtype. This question comes from this discussion:
https://github.com/numba/numba/issues/2729.

Also somewhat (un)related (but I was always curious), why `float` type was
chosen as a default for dtype?

>>> np.array([]).dtype
dtype('float64')
>>> np.ones([]).dtype
dtype('float64')

Why np.int_ was not chosen and will this change be a good idea (in general,
without taking into account backward compatibility :-))? There is some
discussion here: https://github.com/numpy/numpy/issues/10405.

-----------------

p.s.: For some constructors the signature looks as follows (in IPython
console):

>>> np.zeros?
Docstring:
zeros(shape, dtype=float, order='C')

Return a new array of given shape and type, filled with zeros.


>>> np.empty?
Docstring:
empty(shape, dtype=float, order='C')

Return a new array of given shape and type, without initializing entries.


But for `ones` the dtype is none instead of float and looks different:

>>> np.ones?
Signature: np.ones(shape, dtype=None, order='C')
Docstring:
Return a new array of given shape and type, filled with ones.

-----------------

With kind regards,
-gdg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/c7d984d1/attachment.html>

From kirillbalunov at gmail.com  Fri Feb  9 05:47:27 2018
From: kirillbalunov at gmail.com (Kirill Balunov)
Date: Fri, 9 Feb 2018 13:47:27 +0300
Subject: [Numpy-discussion] type promotion rules for integers
Message-ID: <CABwCVQBZfXnwQkCWOxmuwRKF1xzmxZqyEKMWrUFRQrPrKMwYig@mail.gmail.com>

 What considerations formed the basis for choosing the next type promotion
behavior in numpy:

In[2] : a = np.array([10], dtype=np.int64)
        b = np.array([10], dtype=np.uint64)
        (a+b).dtype
Out[2]: dtype('float64')

Why the `object` dtype was not chosen for the resulting dtype? Are
there any disadvantages in this case when choosing `object` instead of
`float64`?

With kind regards,

-gdg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/a8d55c1c/attachment.html>

From chris.barker at noaa.gov  Fri Feb  9 12:19:40 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 9 Feb 2018 11:19:40 -0600
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
Message-ID: <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>

On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:
>
>  It is partly a plea for some development of numerically accurate
>> functions for computing lat/lon grids from a combination of inputs: bounds,
>> counts, and resolutions.
>>
>
Can you be more specific about what problems you've run into -- I work with
lat-lon grids all the time, and have never had a problem.

float32 degrees gives you about 1 meter accuracy or better, so I can see
how losing a few digits might be an issue, though I would argue that you
maybe shouldn't use float32 if you are worried about anything close to 1m
accuracy... -- or shift to a relative coordinate system of some sort.

I have been playing around with the decimal package a bit lately,
>>
>
sigh. decimal is so often looked at a solution to a problem it isn't
designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
:-)

what you get from decimal is variable precision -- maybe a binary variable
precision lib is a better answer -- that would be a good thing to have easy
access to in numpy, but in this case, if you want better accuracy in a
computation that will end up in float32, just use float64.

and I discovered the concept of "fused multiply-add" operations for
>> improved accuracy. I have come to realize that fma operations could be used
>> to greatly improve the accuracy of linspace() and arange().
>>
>
arange() is problematic for non-integer use anyway, by its very definition
(getting the "end point" correct requires the right step, even without FP
error).

and would it really help with linspace? it's computing a delta with one
division in fp, then multiplying it by an integer (represented in fp --
why? why not keep that an integer till the multiply?).

In particular, I have been needing improved results for computing
>> latitude/longitude grids, which tend to be done in float32's to save memory
>> (at least, this is true in data I come across).
>>
>
> If you care about saving memory *and* accuracy, wouldn't it make more
> sense to do your computations in float64, and convert to float32 at the
> end?
>

that does seem to be the easy option :-)


> Now, to the crux of my problem. It is next to impossible to generate a
>> non-trivial numpy array of coordinates, even in double precision, without
>> hitting significant numerical errors.
>>
>
I'm confused, the example you posted doesn't have significant errors...


> Which has lead me down the path of using the decimal package (which
>> doesn't play very nicely with numpy because of the lack of casting rules
>> for it). Consider the following:
>> ```
>> $ cat test_fma.py
>> from __future__ import print_function
>> import numpy as np
>> res = np.float32(0.01)
>> cnt = 7001
>> x0 = np.float32(-115.0)
>> x1 = res * cnt + x0
>> print("res * cnt + x0 = %.16f" % x1)
>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>> print("linspace()[-1]: %.16f" % x[-1])
>>
>> $ python test_fma.py
>> res * cnt + x0 = -44.9900015648454428
>> len(arange()): 7002  arange()[-1]:       -44.975044
>> linspace()[-1]: -44.9900016784667969
>> ```
>> arange just produces silly results (puts out an extra element... adding
>> half of the resolution is typically mentioned as a solution on mailing
>> lists to get around arange()'s limitations -- I personally don't do this).
>>
>
The real solution is "don't do that" arange is not the right tool for the
job.

Then there is this:

res * cnt + x0 = -44.9900015648454428
linspace()[-1]: -44.9900016784667969

that's as good as you are ever going to get with 32 bit floats...

Though I just noticed something about your numbers -- there should be a
nice even base ten delta if you have 7001 gaps -- but linspace produces N
points, not N gaps -- so maybe you want:


In [*17*]: l = np.linspace(-115.0, -44.99, 7002)


In [*18*]: l[:5]

Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])


In [*19*]: l[-5:]

Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])


or, in float32 -- not as pretty:


In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)


In [*21*]: l[:5]

Out[*21*]:

array([-115.        , -114.98999786, -114.98000336, -114.97000122,

       -114.95999908], dtype=float32)


In [*22*]: l[-5:]

Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
-44.99000168], dtype=float32)


but still as good as you get with float32, and exactly the same result as
computing in float64 and converting:



In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)


In [*26*]: l[:5]

Out[*26*]:

array([-115.        , -114.98999786, -114.98000336, -114.97000122,

       -114.95999908], dtype=float32)


In [*27*]: l[-5:]

Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
-44.99000168], dtype=float32)



>> So, does it make any sense to improve arange by utilizing fma() under the
>> hood?
>>
>
no -- this is simply not the right use-case for arange() anyway.


> Also, any plans for making fma() available as a ufunc?
>>
>
could be nice -- especially if used internally.


> Notice that most of my examples required knowing the number of grid points
>> ahead of time. But what if I didn't know that? What if I just have the
>> bounds and the resolution? Then arange() is the natural fit, but as I
>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>> a closed interval.
>>
>
no -- it's not -- if you have the bounds and the resolution, you have an
over-specified problem. That is:

x_min + (n * delta_x) == x_max

If there is ANY error in either delta_x or x_max (or x_min), then you'll
get a missmatch. which is why arange is not the answer (you can make the
algorithm a bit more accurate, I suppose but there is still fp limited
precision -- if you can't exactly represent either delta_x or x_max, then
you CAN'T use the arange() definition and expect to work consistently.

The "right" way to do it is to compute N with: round((x_max - x_min) /
delta), and then use linspace:

linspace(x_min, x_max, N+1)

(note that it's too bad you need to do N+1 -- if I had to do it over again,
I'd use N as the number of "gaps" rather than the number of points --
that's more commonly what people want, if they care at all)

This way, you get a grid with the endpoints as exact as they can be, and
the deltas as close to each-other as they can be as well.

maybe you can do a better algorithm in linspace to save an ULP, but it's
hard to imagine when that would matter.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/6a6bc04f/attachment-0001.html>

From ben.v.root at gmail.com  Fri Feb  9 15:43:45 2018
From: ben.v.root at gmail.com (Benjamin Root)
Date: Fri, 9 Feb 2018 15:43:45 -0500
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
Message-ID: <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>

On Fri, Feb 9, 2018 at 12:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>>
>>  It is partly a plea for some development of numerically accurate
>>> functions for computing lat/lon grids from a combination of inputs: bounds,
>>> counts, and resolutions.
>>>
>>
> Can you be more specific about what problems you've run into -- I work
> with lat-lon grids all the time, and have never had a problem.
>
> float32 degrees gives you about 1 meter accuracy or better, so I can see
> how losing a few digits might be an issue, though I would argue that you
> maybe shouldn't use float32 if you are worried about anything close to 1m
> accuracy... -- or shift to a relative coordinate system of some sort.
>

The issue isn't so much the accuracy of the coordinates themselves. I am
only worried about 1km resolution (which is approximately 0.01 degrees at
mid-latitudes). My concern is with consistent *construction* of a
coordinate grid with even spacing. As it stands right now. If I provide a
corner coordinate, a resolution, and the number of pixels, the result is
not terrible (indeed, this is the approach used by gdal/rasterio). If I
have start/end coordinates and the number of pixels, the result is not bad,
either (use linspace). But, if I have start/end coordinates and a
resolution, then determining the number of pixels from that is actually
tricky to get right in the general case, especially with float32 and large
grids, and especially if the bounding box specified isn't exactly divisible
by the resolution.


>
> I have been playing around with the decimal package a bit lately,
>>>
>>
> sigh. decimal is so often looked at a solution to a problem it isn't
> designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
> :-)
>
> what you get from decimal is variable precision -- maybe a binary variable
> precision lib is a better answer -- that would be a good thing to have easy
> access to in numpy, but in this case, if you want better accuracy in a
> computation that will end up in float32, just use float64.
>

I am not concerned about computing distances or anything like that, I am
trying to properly construct my grid. I need consistent results regardless
of which way the grid is specified (start/end/count, start/res/count,
start/end/res). I have found that loading up the grid specs (using in a
config file or command-line) using the Decimal class allows me to exactly
and consistently represent the grid specification, and gets me most of the
way there. But the problems with arange() is frustrating, and I have to
have extra logic to go around that and over to linspace() instead.


>
> and I discovered the concept of "fused multiply-add" operations for
>>> improved accuracy. I have come to realize that fma operations could be used
>>> to greatly improve the accuracy of linspace() and arange().
>>>
>>
> arange() is problematic for non-integer use anyway, by its very definition
> (getting the "end point" correct requires the right step, even without FP
> error).
>
> and would it really help with linspace? it's computing a delta with one
> division in fp, then multiplying it by an integer (represented in fp --
> why? why not keep that an integer till the multiply?).
>

Sorry, that was a left-over from a previous draft of my email after I
discovered that linspace's accuracy was on par with fma(). And while
arange() has inherent problems, it can still be made better than it is now.
In fact, I haven't investigated this, but I did recently discover some unit
tests of mine started to fail after a numpy upgrade, and traced it back to
a reduction in the accuracy of a usage of arange() with float32s. So,
something got worse at some point, which means we could still get accuracy
back if we can figure out what changed.


>
> In particular, I have been needing improved results for computing
>>> latitude/longitude grids, which tend to be done in float32's to save memory
>>> (at least, this is true in data I come across).
>>>
>>
>> If you care about saving memory *and* accuracy, wouldn't it make more
>> sense to do your computations in float64, and convert to float32 at the
>> end?
>>
>
> that does seem to be the easy option :-)
>

Kinda missing the point, isn't it? Isn't that like saying "convert all your
data to float64s prior to calling np.mean()"? That's ridiculous. Instead,
we made np.mean() upcast the inner-loop operation, and even allow an option
to specify what the dtype that should be used for the aggregator.


>
>
>> Now, to the crux of my problem. It is next to impossible to generate a
>>> non-trivial numpy array of coordinates, even in double precision, without
>>> hitting significant numerical errors.
>>>
>>
> I'm confused, the example you posted doesn't have significant errors...
>

Hmm, "errors" was the wrong word. "Differences between methods" might be
more along the lines of what I was thinking. Remember, I am looking for
consistency.


>
>
>> Which has lead me down the path of using the decimal package (which
>>> doesn't play very nicely with numpy because of the lack of casting rules
>>> for it). Consider the following:
>>> ```
>>> $ cat test_fma.py
>>> from __future__ import print_function
>>> import numpy as np
>>> res = np.float32(0.01)
>>> cnt = 7001
>>> x0 = np.float32(-115.0)
>>> x1 = res * cnt + x0
>>> print("res * cnt + x0 = %.16f" % x1)
>>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>>> print("linspace()[-1]: %.16f" % x[-1])
>>>
>>> $ python test_fma.py
>>> res * cnt + x0 = -44.9900015648454428
>>> len(arange()): 7002  arange()[-1]:       -44.975044
>>> linspace()[-1]: -44.9900016784667969
>>> ```
>>> arange just produces silly results (puts out an extra element... adding
>>> half of the resolution is typically mentioned as a solution on mailing
>>> lists to get around arange()'s limitations -- I personally don't do this).
>>>
>>
> The real solution is "don't do that" arange is not the right tool for the
> job.
>

Well, it isn't the right tool because as far as I am concerned, it is
useless for anything but integers. Why not fix it to be more suitable for
floating point?


>
> Then there is this:
>
> res * cnt + x0 = -44.9900015648454428
> linspace()[-1]: -44.9900016784667969
>
> that's as good as you are ever going to get with 32 bit floats...
>

Consistency is the key thing. I am fine with one of those values, so long
as that value is what happens no matter which way I specify my grid.


>
> Though I just noticed something about your numbers -- there should be a
> nice even base ten delta if you have 7001 gaps -- but linspace produces N
> points, not N gaps -- so maybe you want:
>
>
> In [*17*]: l = np.linspace(-115.0, -44.99, 7002)
>
>
> In [*18*]: l[:5]
>
> Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])
>
>
> In [*19*]: l[-5:]
>
> Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])
>
>
> or, in float32 -- not as pretty:
>
>
> In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)
>
>
> In [*21*]: l[:5]
>
> Out[*21*]:
>
> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>
>        -114.95999908], dtype=float32)
>
>
> In [*22*]: l[-5:]
>
> Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
> -44.99000168], dtype=float32)
>
>
> but still as good as you get with float32, and exactly the same result as
> computing in float64 and converting:
>
>
>
> In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)
>
>
> In [*26*]: l[:5]
>
> Out[*26*]:
>
> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>
>        -114.95999908], dtype=float32)
>
>
> In [*27*]: l[-5:]
>
> Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
> -44.99000168], dtype=float32)
>

Argh! I got myself mixed up between specifying pixel corners versus pixel
centers. rasterio has been messing me up on this.


>
>
>>> So, does it make any sense to improve arange by utilizing fma() under
>>> the hood?
>>>
>>
> no -- this is simply not the right use-case for arange() anyway.
>

arange() has accuracy problems, so why not fix it?

>>> l4 = np.arange(-115, -44.99, 0.01, dtype=np.float32)
>>> np.median(np.diff(l4))
0.0099945068
>>> np.float32(0.01)
0.0099999998

There is something significantly wrong here if arange(), which takes a
resolution parameter, can't seem to produce a sequence with the proper
delta.



>
>
>> Also, any plans for making fma() available as a ufunc?
>>>
>>
> could be nice -- especially if used internally.
>
>
>> Notice that most of my examples required knowing the number of grid
>>> points ahead of time. But what if I didn't know that? What if I just have
>>> the bounds and the resolution? Then arange() is the natural fit, but as I
>>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>>> a closed interval.
>>>
>>
> no -- it's not -- if you have the bounds and the resolution, you have an
> over-specified problem. That is:
>
> x_min + (n * delta_x) == x_max
>
> If there is ANY error in either delta_x or x_max (or x_min), then you'll
> get a missmatch. which is why arange is not the answer (you can make the
> algorithm a bit more accurate, I suppose but there is still fp limited
> precision -- if you can't exactly represent either delta_x or x_max, then
> you CAN'T use the arange() definition and expect to work consistently.
>
> The "right" way to do it is to compute N with: round((x_max - x_min) /
> delta), and then use linspace:
>
> linspace(x_min, x_max, N+1)
>
> (note that it's too bad you need to do N+1 -- if I had to do it over
> again, I'd use N as the number of "gaps" rather than the number of points
> -- that's more commonly what people want, if they care at all)
>
> This way, you get a grid with the endpoints as exact as they can be, and
> the deltas as close to each-other as they can be as well.
>
> maybe you can do a better algorithm in linspace to save an ULP, but it's
> hard to imagine when that would matter.
>

Yes, it is overspecified. My problem is that different tools require
different specs (ahem... rasterio/gdal), and I have gird specs coming from
other sources. And I need to produce data onto the same grid so that tools
like xarray won't yell at me when I am trying to do an operation between
gridded data that should have the same coordinates, but are off slightly
because they were computed differently for whatever reason.

I guess I am crying out for some sort of tool that will help the community
stop making the same mistakes. A one-stop shop that'll allow us to specify
a grid in a few different ways and still produce the right thing, and even
do the inverse... provide a coordinate array and get grids specs in
whatever form we want. Maybe even have options for dealing with pixel
corner vs. pixel centers, too? There are additional fun problems such as
padding out coordinate arrays, which np.pad doesn't really do a great job
with.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/26928050/attachment-0001.html>

From harrigan.matthew at gmail.com  Fri Feb  9 16:16:44 2018
From: harrigan.matthew at gmail.com (Matthew Harrigan)
Date: Fri, 9 Feb 2018 16:16:44 -0500
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
Message-ID: <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>

I apologize if I'm missing something basic, but why are floats being
accumulated in the first place?  Can't arange and linspace operations with
floats be done internally similar to `start + np.arange(num_steps) *
step_size`?  I.e. always accumulate (really increment) integers to limit
errors.

On Fri, Feb 9, 2018 at 3:43 PM, Benjamin Root <ben.v.root at gmail.com> wrote:

>
>
> On Fri, Feb 9, 2018 at 12:19 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>>
>>>  It is partly a plea for some development of numerically accurate
>>>> functions for computing lat/lon grids from a combination of inputs: bounds,
>>>> counts, and resolutions.
>>>>
>>>
>> Can you be more specific about what problems you've run into -- I work
>> with lat-lon grids all the time, and have never had a problem.
>>
>> float32 degrees gives you about 1 meter accuracy or better, so I can see
>> how losing a few digits might be an issue, though I would argue that you
>> maybe shouldn't use float32 if you are worried about anything close to 1m
>> accuracy... -- or shift to a relative coordinate system of some sort.
>>
>
> The issue isn't so much the accuracy of the coordinates themselves. I am
> only worried about 1km resolution (which is approximately 0.01 degrees at
> mid-latitudes). My concern is with consistent *construction* of a
> coordinate grid with even spacing. As it stands right now. If I provide a
> corner coordinate, a resolution, and the number of pixels, the result is
> not terrible (indeed, this is the approach used by gdal/rasterio). If I
> have start/end coordinates and the number of pixels, the result is not bad,
> either (use linspace). But, if I have start/end coordinates and a
> resolution, then determining the number of pixels from that is actually
> tricky to get right in the general case, especially with float32 and large
> grids, and especially if the bounding box specified isn't exactly divisible
> by the resolution.
>
>
>>
>> I have been playing around with the decimal package a bit lately,
>>>>
>>>
>> sigh. decimal is so often looked at a solution to a problem it isn't
>> designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
>> :-)
>>
>> what you get from decimal is variable precision -- maybe a binary
>> variable precision lib is a better answer -- that would be a good thing to
>> have easy access to in numpy, but in this case, if you want better accuracy
>> in a computation that will end up in float32, just use float64.
>>
>
> I am not concerned about computing distances or anything like that, I am
> trying to properly construct my grid. I need consistent results regardless
> of which way the grid is specified (start/end/count, start/res/count,
> start/end/res). I have found that loading up the grid specs (using in a
> config file or command-line) using the Decimal class allows me to exactly
> and consistently represent the grid specification, and gets me most of the
> way there. But the problems with arange() is frustrating, and I have to
> have extra logic to go around that and over to linspace() instead.
>
>
>>
>> and I discovered the concept of "fused multiply-add" operations for
>>>> improved accuracy. I have come to realize that fma operations could be used
>>>> to greatly improve the accuracy of linspace() and arange().
>>>>
>>>
>> arange() is problematic for non-integer use anyway, by its very
>> definition (getting the "end point" correct requires the right step, even
>> without FP error).
>>
>> and would it really help with linspace? it's computing a delta with one
>> division in fp, then multiplying it by an integer (represented in fp --
>> why? why not keep that an integer till the multiply?).
>>
>
> Sorry, that was a left-over from a previous draft of my email after I
> discovered that linspace's accuracy was on par with fma(). And while
> arange() has inherent problems, it can still be made better than it is now.
> In fact, I haven't investigated this, but I did recently discover some unit
> tests of mine started to fail after a numpy upgrade, and traced it back to
> a reduction in the accuracy of a usage of arange() with float32s. So,
> something got worse at some point, which means we could still get accuracy
> back if we can figure out what changed.
>
>
>>
>> In particular, I have been needing improved results for computing
>>>> latitude/longitude grids, which tend to be done in float32's to save memory
>>>> (at least, this is true in data I come across).
>>>>
>>>
>>> If you care about saving memory *and* accuracy, wouldn't it make more
>>> sense to do your computations in float64, and convert to float32 at the
>>> end?
>>>
>>
>> that does seem to be the easy option :-)
>>
>
> Kinda missing the point, isn't it? Isn't that like saying "convert all
> your data to float64s prior to calling np.mean()"? That's ridiculous.
> Instead, we made np.mean() upcast the inner-loop operation, and even allow
> an option to specify what the dtype that should be used for the aggregator.
>
>
>>
>>
>>> Now, to the crux of my problem. It is next to impossible to generate a
>>>> non-trivial numpy array of coordinates, even in double precision, without
>>>> hitting significant numerical errors.
>>>>
>>>
>> I'm confused, the example you posted doesn't have significant errors...
>>
>
> Hmm, "errors" was the wrong word. "Differences between methods" might be
> more along the lines of what I was thinking. Remember, I am looking for
> consistency.
>
>
>>
>>
>>> Which has lead me down the path of using the decimal package (which
>>>> doesn't play very nicely with numpy because of the lack of casting rules
>>>> for it). Consider the following:
>>>> ```
>>>> $ cat test_fma.py
>>>> from __future__ import print_function
>>>> import numpy as np
>>>> res = np.float32(0.01)
>>>> cnt = 7001
>>>> x0 = np.float32(-115.0)
>>>> x1 = res * cnt + x0
>>>> print("res * cnt + x0 = %.16f" % x1)
>>>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>>>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>>>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>>>> print("linspace()[-1]: %.16f" % x[-1])
>>>>
>>>> $ python test_fma.py
>>>> res * cnt + x0 = -44.9900015648454428
>>>> len(arange()): 7002  arange()[-1]:       -44.975044
>>>> linspace()[-1]: -44.9900016784667969
>>>> ```
>>>> arange just produces silly results (puts out an extra element... adding
>>>> half of the resolution is typically mentioned as a solution on mailing
>>>> lists to get around arange()'s limitations -- I personally don't do this).
>>>>
>>>
>> The real solution is "don't do that" arange is not the right tool for the
>> job.
>>
>
> Well, it isn't the right tool because as far as I am concerned, it is
> useless for anything but integers. Why not fix it to be more suitable for
> floating point?
>
>
>>
>> Then there is this:
>>
>> res * cnt + x0 = -44.9900015648454428
>> linspace()[-1]: -44.9900016784667969
>>
>> that's as good as you are ever going to get with 32 bit floats...
>>
>
> Consistency is the key thing. I am fine with one of those values, so long
> as that value is what happens no matter which way I specify my grid.
>
>
>>
>> Though I just noticed something about your numbers -- there should be a
>> nice even base ten delta if you have 7001 gaps -- but linspace produces N
>> points, not N gaps -- so maybe you want:
>>
>>
>> In [*17*]: l = np.linspace(-115.0, -44.99, 7002)
>>
>>
>> In [*18*]: l[:5]
>>
>> Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])
>>
>>
>> In [*19*]: l[-5:]
>>
>> Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])
>>
>>
>> or, in float32 -- not as pretty:
>>
>>
>> In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)
>>
>>
>> In [*21*]: l[:5]
>>
>> Out[*21*]:
>>
>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>
>>        -114.95999908], dtype=float32)
>>
>>
>> In [*22*]: l[-5:]
>>
>> Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
>> -44.99000168], dtype=float32)
>>
>>
>> but still as good as you get with float32, and exactly the same result as
>> computing in float64 and converting:
>>
>>
>>
>> In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)
>>
>>
>> In [*26*]: l[:5]
>>
>> Out[*26*]:
>>
>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>
>>        -114.95999908], dtype=float32)
>>
>>
>> In [*27*]: l[-5:]
>>
>> Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
>> -44.99000168], dtype=float32)
>>
>
> Argh! I got myself mixed up between specifying pixel corners versus pixel
> centers. rasterio has been messing me up on this.
>
>
>>
>>
>>>> So, does it make any sense to improve arange by utilizing fma() under
>>>> the hood?
>>>>
>>>
>> no -- this is simply not the right use-case for arange() anyway.
>>
>
> arange() has accuracy problems, so why not fix it?
>
> >>> l4 = np.arange(-115, -44.99, 0.01, dtype=np.float32)
> >>> np.median(np.diff(l4))
> 0.0099945068
> >>> np.float32(0.01)
> 0.0099999998
>
> There is something significantly wrong here if arange(), which takes a
> resolution parameter, can't seem to produce a sequence with the proper
> delta.
>
>
>
>>
>>
>>> Also, any plans for making fma() available as a ufunc?
>>>>
>>>
>> could be nice -- especially if used internally.
>>
>>
>>> Notice that most of my examples required knowing the number of grid
>>>> points ahead of time. But what if I didn't know that? What if I just have
>>>> the bounds and the resolution? Then arange() is the natural fit, but as I
>>>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>>>> a closed interval.
>>>>
>>>
>> no -- it's not -- if you have the bounds and the resolution, you have an
>> over-specified problem. That is:
>>
>> x_min + (n * delta_x) == x_max
>>
>> If there is ANY error in either delta_x or x_max (or x_min), then you'll
>> get a missmatch. which is why arange is not the answer (you can make the
>> algorithm a bit more accurate, I suppose but there is still fp limited
>> precision -- if you can't exactly represent either delta_x or x_max, then
>> you CAN'T use the arange() definition and expect to work consistently.
>>
>> The "right" way to do it is to compute N with: round((x_max - x_min) /
>> delta), and then use linspace:
>>
>> linspace(x_min, x_max, N+1)
>>
>> (note that it's too bad you need to do N+1 -- if I had to do it over
>> again, I'd use N as the number of "gaps" rather than the number of points
>> -- that's more commonly what people want, if they care at all)
>>
>> This way, you get a grid with the endpoints as exact as they can be, and
>> the deltas as close to each-other as they can be as well.
>>
>> maybe you can do a better algorithm in linspace to save an ULP, but it's
>> hard to imagine when that would matter.
>>
>
> Yes, it is overspecified. My problem is that different tools require
> different specs (ahem... rasterio/gdal), and I have gird specs coming from
> other sources. And I need to produce data onto the same grid so that tools
> like xarray won't yell at me when I am trying to do an operation between
> gridded data that should have the same coordinates, but are off slightly
> because they were computed differently for whatever reason.
>
> I guess I am crying out for some sort of tool that will help the community
> stop making the same mistakes. A one-stop shop that'll allow us to specify
> a grid in a few different ways and still produce the right thing, and even
> do the inverse... provide a coordinate array and get grids specs in
> whatever form we want. Maybe even have options for dealing with pixel
> corner vs. pixel centers, too? There are additional fun problems such as
> padding out coordinate arrays, which np.pad doesn't really do a great job
> with.
>
> Cheers!
> Ben Root
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/5985eceb/attachment-0001.html>

From wieser.eric+numpy at gmail.com  Fri Feb  9 16:56:10 2018
From: wieser.eric+numpy at gmail.com (Eric Wieser)
Date: Fri, 09 Feb 2018 21:56:10 +0000
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
Message-ID: <CAL1kJvBspn0oV6UcsJV_67_fv75VNP1g-nhOZnm_zSNx8591CQ@mail.gmail.com>

Can?t arange and linspace operations with floats be done internally

Yes, and they probably should be - they?re done this way as a hack because
the api exposed for custom dtypes is here
<https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/include/numpy/ndarraytypes.h#L507-L511>,
(example implementation here
<https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/src/multiarray/arraytypes.c.src#L3711-L3721>)
- essentially, you give it the first two elements of the array, and ask it
to fill in the rest.
?

On Fri, 9 Feb 2018 at 13:17 Matthew Harrigan <harrigan.matthew at gmail.com>
wrote:

> I apologize if I'm missing something basic, but why are floats being
> accumulated in the first place?  Can't arange and linspace operations with
> floats be done internally similar to `start + np.arange(num_steps) *
> step_size`?  I.e. always accumulate (really increment) integers to limit
> errors.
>
> On Fri, Feb 9, 2018 at 3:43 PM, Benjamin Root <ben.v.root at gmail.com>
> wrote:
>
>>
>>
>> On Fri, Feb 9, 2018 at 12:19 PM, Chris Barker <chris.barker at noaa.gov>
>> wrote:
>>
>>> On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>>
>>>>  It is partly a plea for some development of numerically accurate
>>>>> functions for computing lat/lon grids from a combination of inputs: bounds,
>>>>> counts, and resolutions.
>>>>>
>>>>
>>> Can you be more specific about what problems you've run into -- I work
>>> with lat-lon grids all the time, and have never had a problem.
>>>
>>> float32 degrees gives you about 1 meter accuracy or better, so I can see
>>> how losing a few digits might be an issue, though I would argue that you
>>> maybe shouldn't use float32 if you are worried about anything close to 1m
>>> accuracy... -- or shift to a relative coordinate system of some sort.
>>>
>>
>> The issue isn't so much the accuracy of the coordinates themselves. I am
>> only worried about 1km resolution (which is approximately 0.01 degrees at
>> mid-latitudes). My concern is with consistent *construction* of a
>> coordinate grid with even spacing. As it stands right now. If I provide a
>> corner coordinate, a resolution, and the number of pixels, the result is
>> not terrible (indeed, this is the approach used by gdal/rasterio). If I
>> have start/end coordinates and the number of pixels, the result is not bad,
>> either (use linspace). But, if I have start/end coordinates and a
>> resolution, then determining the number of pixels from that is actually
>> tricky to get right in the general case, especially with float32 and large
>> grids, and especially if the bounding box specified isn't exactly divisible
>> by the resolution.
>>
>>
>>>
>>> I have been playing around with the decimal package a bit lately,
>>>>>
>>>>
>>> sigh. decimal is so often looked at a solution to a problem it isn't
>>> designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
>>> :-)
>>>
>>> what you get from decimal is variable precision -- maybe a binary
>>> variable precision lib is a better answer -- that would be a good thing to
>>> have easy access to in numpy, but in this case, if you want better accuracy
>>> in a computation that will end up in float32, just use float64.
>>>
>>
>> I am not concerned about computing distances or anything like that, I am
>> trying to properly construct my grid. I need consistent results regardless
>> of which way the grid is specified (start/end/count, start/res/count,
>> start/end/res). I have found that loading up the grid specs (using in a
>> config file or command-line) using the Decimal class allows me to exactly
>> and consistently represent the grid specification, and gets me most of the
>> way there. But the problems with arange() is frustrating, and I have to
>> have extra logic to go around that and over to linspace() instead.
>>
>>
>>>
>>> and I discovered the concept of "fused multiply-add" operations for
>>>>> improved accuracy. I have come to realize that fma operations could be used
>>>>> to greatly improve the accuracy of linspace() and arange().
>>>>>
>>>>
>>> arange() is problematic for non-integer use anyway, by its very
>>> definition (getting the "end point" correct requires the right step, even
>>> without FP error).
>>>
>>> and would it really help with linspace? it's computing a delta with one
>>> division in fp, then multiplying it by an integer (represented in fp --
>>> why? why not keep that an integer till the multiply?).
>>>
>>
>> Sorry, that was a left-over from a previous draft of my email after I
>> discovered that linspace's accuracy was on par with fma(). And while
>> arange() has inherent problems, it can still be made better than it is now.
>> In fact, I haven't investigated this, but I did recently discover some unit
>> tests of mine started to fail after a numpy upgrade, and traced it back to
>> a reduction in the accuracy of a usage of arange() with float32s. So,
>> something got worse at some point, which means we could still get accuracy
>> back if we can figure out what changed.
>>
>>
>>>
>>> In particular, I have been needing improved results for computing
>>>>> latitude/longitude grids, which tend to be done in float32's to save memory
>>>>> (at least, this is true in data I come across).
>>>>>
>>>>
>>>> If you care about saving memory *and* accuracy, wouldn't it make more
>>>> sense to do your computations in float64, and convert to float32 at the
>>>> end?
>>>>
>>>
>>> that does seem to be the easy option :-)
>>>
>>
>> Kinda missing the point, isn't it? Isn't that like saying "convert all
>> your data to float64s prior to calling np.mean()"? That's ridiculous.
>> Instead, we made np.mean() upcast the inner-loop operation, and even allow
>> an option to specify what the dtype that should be used for the aggregator.
>>
>>
>>>
>>>
>>>> Now, to the crux of my problem. It is next to impossible to generate a
>>>>> non-trivial numpy array of coordinates, even in double precision, without
>>>>> hitting significant numerical errors.
>>>>>
>>>>
>>> I'm confused, the example you posted doesn't have significant errors...
>>>
>>
>> Hmm, "errors" was the wrong word. "Differences between methods" might be
>> more along the lines of what I was thinking. Remember, I am looking for
>> consistency.
>>
>>
>>>
>>>
>>>> Which has lead me down the path of using the decimal package (which
>>>>> doesn't play very nicely with numpy because of the lack of casting rules
>>>>> for it). Consider the following:
>>>>> ```
>>>>> $ cat test_fma.py
>>>>> from __future__ import print_function
>>>>> import numpy as np
>>>>> res = np.float32(0.01)
>>>>> cnt = 7001
>>>>> x0 = np.float32(-115.0)
>>>>> x1 = res * cnt + x0
>>>>> print("res * cnt + x0 = %.16f" % x1)
>>>>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>>>>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>>>>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>>>>> print("linspace()[-1]: %.16f" % x[-1])
>>>>>
>>>>> $ python test_fma.py
>>>>> res * cnt + x0 = -44.9900015648454428
>>>>> len(arange()): 7002  arange()[-1]:       -44.975044
>>>>> linspace()[-1]: -44.9900016784667969
>>>>> ```
>>>>> arange just produces silly results (puts out an extra element...
>>>>> adding half of the resolution is typically mentioned as a solution on
>>>>> mailing lists to get around arange()'s limitations -- I personally don't do
>>>>> this).
>>>>>
>>>>
>>> The real solution is "don't do that" arange is not the right tool for
>>> the job.
>>>
>>
>> Well, it isn't the right tool because as far as I am concerned, it is
>> useless for anything but integers. Why not fix it to be more suitable for
>> floating point?
>>
>>
>>>
>>> Then there is this:
>>>
>>> res * cnt + x0 = -44.9900015648454428
>>> linspace()[-1]: -44.9900016784667969
>>>
>>> that's as good as you are ever going to get with 32 bit floats...
>>>
>>
>> Consistency is the key thing. I am fine with one of those values, so long
>> as that value is what happens no matter which way I specify my grid.
>>
>>
>>>
>>> Though I just noticed something about your numbers -- there should be a
>>> nice even base ten delta if you have 7001 gaps -- but linspace produces N
>>> points, not N gaps -- so maybe you want:
>>>
>>>
>>> In [*17*]: l = np.linspace(-115.0, -44.99, 7002)
>>>
>>>
>>> In [*18*]: l[:5]
>>>
>>> Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])
>>>
>>>
>>> In [*19*]: l[-5:]
>>>
>>> Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])
>>>
>>>
>>> or, in float32 -- not as pretty:
>>>
>>>
>>> In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)
>>>
>>>
>>> In [*21*]: l[:5]
>>>
>>> Out[*21*]:
>>>
>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>
>>>        -114.95999908], dtype=float32)
>>>
>>>
>>> In [*22*]: l[-5:]
>>>
>>> Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
>>> -44.99000168], dtype=float32)
>>>
>>>
>>> but still as good as you get with float32, and exactly the same result
>>> as computing in float64 and converting:
>>>
>>>
>>>
>>> In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)
>>>
>>>
>>> In [*26*]: l[:5]
>>>
>>> Out[*26*]:
>>>
>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>
>>>        -114.95999908], dtype=float32)
>>>
>>>
>>> In [*27*]: l[-5:]
>>>
>>> Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.        ,
>>> -44.99000168], dtype=float32)
>>>
>>
>> Argh! I got myself mixed up between specifying pixel corners versus pixel
>> centers. rasterio has been messing me up on this.
>>
>>
>>>
>>>
>>>>> So, does it make any sense to improve arange by utilizing fma() under
>>>>> the hood?
>>>>>
>>>>
>>> no -- this is simply not the right use-case for arange() anyway.
>>>
>>
>> arange() has accuracy problems, so why not fix it?
>>
>> >>> l4 = np.arange(-115, -44.99, 0.01, dtype=np.float32)
>> >>> np.median(np.diff(l4))
>> 0.0099945068
>> >>> np.float32(0.01)
>> 0.0099999998
>>
>> There is something significantly wrong here if arange(), which takes a
>> resolution parameter, can't seem to produce a sequence with the proper
>> delta.
>>
>>
>>
>>>
>>>
>>>> Also, any plans for making fma() available as a ufunc?
>>>>>
>>>>
>>> could be nice -- especially if used internally.
>>>
>>>
>>>> Notice that most of my examples required knowing the number of grid
>>>>> points ahead of time. But what if I didn't know that? What if I just have
>>>>> the bounds and the resolution? Then arange() is the natural fit, but as I
>>>>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>>>>> a closed interval.
>>>>>
>>>>
>>> no -- it's not -- if you have the bounds and the resolution, you have an
>>> over-specified problem. That is:
>>>
>>> x_min + (n * delta_x) == x_max
>>>
>>> If there is ANY error in either delta_x or x_max (or x_min), then you'll
>>> get a missmatch. which is why arange is not the answer (you can make the
>>> algorithm a bit more accurate, I suppose but there is still fp limited
>>> precision -- if you can't exactly represent either delta_x or x_max, then
>>> you CAN'T use the arange() definition and expect to work consistently.
>>>
>>> The "right" way to do it is to compute N with: round((x_max - x_min) /
>>> delta), and then use linspace:
>>>
>>> linspace(x_min, x_max, N+1)
>>>
>>> (note that it's too bad you need to do N+1 -- if I had to do it over
>>> again, I'd use N as the number of "gaps" rather than the number of points
>>> -- that's more commonly what people want, if they care at all)
>>>
>>> This way, you get a grid with the endpoints as exact as they can be, and
>>> the deltas as close to each-other as they can be as well.
>>>
>>> maybe you can do a better algorithm in linspace to save an ULP, but it's
>>> hard to imagine when that would matter.
>>>
>>
>> Yes, it is overspecified. My problem is that different tools require
>> different specs (ahem... rasterio/gdal), and I have gird specs coming from
>> other sources. And I need to produce data onto the same grid so that tools
>> like xarray won't yell at me when I am trying to do an operation between
>> gridded data that should have the same coordinates, but are off slightly
>> because they were computed differently for whatever reason.
>>
>> I guess I am crying out for some sort of tool that will help the
>> community stop making the same mistakes. A one-stop shop that'll allow us
>> to specify a grid in a few different ways and still produce the right
>> thing, and even do the inverse... provide a coordinate array and get grids
>> specs in whatever form we want. Maybe even have options for dealing with
>> pixel corner vs. pixel centers, too? There are additional fun problems such
>> as padding out coordinate arrays, which np.pad doesn't really do a great
>> job with.
>>
>> Cheers!
>> Ben Root
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/f5153d93/attachment-0001.html>

From ben.v.root at gmail.com  Fri Feb  9 17:17:44 2018
From: ben.v.root at gmail.com (Benjamin Root)
Date: Fri, 9 Feb 2018 17:17:44 -0500
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CAL1kJvBspn0oV6UcsJV_67_fv75VNP1g-nhOZnm_zSNx8591CQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
 <CAL1kJvBspn0oV6UcsJV_67_fv75VNP1g-nhOZnm_zSNx8591CQ@mail.gmail.com>
Message-ID: <CANNq6Fnu2FyvmQ8JZWq=Gs+vK3D5Psx77TAkGZZH73ftgKEUqQ@mail.gmail.com>

Interesting...

```
static void
@NAME at _fill(@type@ *buffer, npy_intp length, void *NPY_UNUSED(ignored))
{
npy_intp i;
@type@ start = buffer[0];
@type@ delta = buffer[1];
delta -= start;
for (i = 2; i < length; ++i) {
buffer[i] = start + i*delta;
}
}
```

So, the second value is computed using the delta arange was given, but then
tries to get the delta back, which incurs errors:
```
>>> a = np.float32(-115)
>>> delta = np.float32(0.01)
>>> b = a + delta
>>> new_delta = b - a
>>> "%.16f" % delta
'0.0099999997764826'
>>> "%.16f" % new_delta
'0.0100021362304688'
```

Also, right there is a good example of where the use of fma() could be of
value.

Cheers!
Ben Root


On Fri, Feb 9, 2018 at 4:56 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
wrote:

> Can?t arange and linspace operations with floats be done internally
>
> Yes, and they probably should be - they?re done this way as a hack because
> the api exposed for custom dtypes is here
> <https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/include/numpy/ndarraytypes.h#L507-L511>,
> (example implementation here
> <https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/src/multiarray/arraytypes.c.src#L3711-L3721>)
> - essentially, you give it the first two elements of the array, and ask it
> to fill in the rest.
> ?
>
> On Fri, 9 Feb 2018 at 13:17 Matthew Harrigan <harrigan.matthew at gmail.com>
> wrote:
>
>> I apologize if I'm missing something basic, but why are floats being
>> accumulated in the first place?  Can't arange and linspace operations with
>> floats be done internally similar to `start + np.arange(num_steps) *
>> step_size`?  I.e. always accumulate (really increment) integers to limit
>> errors.
>>
>> On Fri, Feb 9, 2018 at 3:43 PM, Benjamin Root <ben.v.root at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Fri, Feb 9, 2018 at 12:19 PM, Chris Barker <chris.barker at noaa.gov>
>>> wrote:
>>>
>>>> On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>>>
>>>>>  It is partly a plea for some development of numerically accurate
>>>>>> functions for computing lat/lon grids from a combination of inputs: bounds,
>>>>>> counts, and resolutions.
>>>>>>
>>>>>
>>>> Can you be more specific about what problems you've run into -- I work
>>>> with lat-lon grids all the time, and have never had a problem.
>>>>
>>>> float32 degrees gives you about 1 meter accuracy or better, so I can
>>>> see how losing a few digits might be an issue, though I would argue that
>>>> you maybe shouldn't use float32 if you are worried about anything close to
>>>> 1m accuracy... -- or shift to a relative coordinate system of some sort.
>>>>
>>>
>>> The issue isn't so much the accuracy of the coordinates themselves. I am
>>> only worried about 1km resolution (which is approximately 0.01 degrees at
>>> mid-latitudes). My concern is with consistent *construction* of a
>>> coordinate grid with even spacing. As it stands right now. If I provide a
>>> corner coordinate, a resolution, and the number of pixels, the result is
>>> not terrible (indeed, this is the approach used by gdal/rasterio). If I
>>> have start/end coordinates and the number of pixels, the result is not bad,
>>> either (use linspace). But, if I have start/end coordinates and a
>>> resolution, then determining the number of pixels from that is actually
>>> tricky to get right in the general case, especially with float32 and large
>>> grids, and especially if the bounding box specified isn't exactly divisible
>>> by the resolution.
>>>
>>>
>>>>
>>>> I have been playing around with the decimal package a bit lately,
>>>>>>
>>>>>
>>>> sigh. decimal is so often looked at a solution to a problem it isn't
>>>> designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
>>>> :-)
>>>>
>>>> what you get from decimal is variable precision -- maybe a binary
>>>> variable precision lib is a better answer -- that would be a good thing to
>>>> have easy access to in numpy, but in this case, if you want better accuracy
>>>> in a computation that will end up in float32, just use float64.
>>>>
>>>
>>> I am not concerned about computing distances or anything like that, I am
>>> trying to properly construct my grid. I need consistent results regardless
>>> of which way the grid is specified (start/end/count, start/res/count,
>>> start/end/res). I have found that loading up the grid specs (using in a
>>> config file or command-line) using the Decimal class allows me to exactly
>>> and consistently represent the grid specification, and gets me most of the
>>> way there. But the problems with arange() is frustrating, and I have to
>>> have extra logic to go around that and over to linspace() instead.
>>>
>>>
>>>>
>>>> and I discovered the concept of "fused multiply-add" operations for
>>>>>> improved accuracy. I have come to realize that fma operations could be used
>>>>>> to greatly improve the accuracy of linspace() and arange().
>>>>>>
>>>>>
>>>> arange() is problematic for non-integer use anyway, by its very
>>>> definition (getting the "end point" correct requires the right step, even
>>>> without FP error).
>>>>
>>>> and would it really help with linspace? it's computing a delta with one
>>>> division in fp, then multiplying it by an integer (represented in fp --
>>>> why? why not keep that an integer till the multiply?).
>>>>
>>>
>>> Sorry, that was a left-over from a previous draft of my email after I
>>> discovered that linspace's accuracy was on par with fma(). And while
>>> arange() has inherent problems, it can still be made better than it is now.
>>> In fact, I haven't investigated this, but I did recently discover some unit
>>> tests of mine started to fail after a numpy upgrade, and traced it back to
>>> a reduction in the accuracy of a usage of arange() with float32s. So,
>>> something got worse at some point, which means we could still get accuracy
>>> back if we can figure out what changed.
>>>
>>>
>>>>
>>>> In particular, I have been needing improved results for computing
>>>>>> latitude/longitude grids, which tend to be done in float32's to save memory
>>>>>> (at least, this is true in data I come across).
>>>>>>
>>>>>
>>>>> If you care about saving memory *and* accuracy, wouldn't it make more
>>>>> sense to do your computations in float64, and convert to float32 at the
>>>>> end?
>>>>>
>>>>
>>>> that does seem to be the easy option :-)
>>>>
>>>
>>> Kinda missing the point, isn't it? Isn't that like saying "convert all
>>> your data to float64s prior to calling np.mean()"? That's ridiculous.
>>> Instead, we made np.mean() upcast the inner-loop operation, and even allow
>>> an option to specify what the dtype that should be used for the aggregator.
>>>
>>>
>>>>
>>>>
>>>>> Now, to the crux of my problem. It is next to impossible to generate a
>>>>>> non-trivial numpy array of coordinates, even in double precision, without
>>>>>> hitting significant numerical errors.
>>>>>>
>>>>>
>>>> I'm confused, the example you posted doesn't have significant errors...
>>>>
>>>
>>> Hmm, "errors" was the wrong word. "Differences between methods" might be
>>> more along the lines of what I was thinking. Remember, I am looking for
>>> consistency.
>>>
>>>
>>>>
>>>>
>>>>> Which has lead me down the path of using the decimal package (which
>>>>>> doesn't play very nicely with numpy because of the lack of casting rules
>>>>>> for it). Consider the following:
>>>>>> ```
>>>>>> $ cat test_fma.py
>>>>>> from __future__ import print_function
>>>>>> import numpy as np
>>>>>> res = np.float32(0.01)
>>>>>> cnt = 7001
>>>>>> x0 = np.float32(-115.0)
>>>>>> x1 = res * cnt + x0
>>>>>> print("res * cnt + x0 = %.16f" % x1)
>>>>>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>>>>>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>>>>>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>>>>>> print("linspace()[-1]: %.16f" % x[-1])
>>>>>>
>>>>>> $ python test_fma.py
>>>>>> res * cnt + x0 = -44.9900015648454428
>>>>>> len(arange()): 7002  arange()[-1]:       -44.975044
>>>>>> linspace()[-1]: -44.9900016784667969
>>>>>> ```
>>>>>> arange just produces silly results (puts out an extra element...
>>>>>> adding half of the resolution is typically mentioned as a solution on
>>>>>> mailing lists to get around arange()'s limitations -- I personally don't do
>>>>>> this).
>>>>>>
>>>>>
>>>> The real solution is "don't do that" arange is not the right tool for
>>>> the job.
>>>>
>>>
>>> Well, it isn't the right tool because as far as I am concerned, it is
>>> useless for anything but integers. Why not fix it to be more suitable for
>>> floating point?
>>>
>>>
>>>>
>>>> Then there is this:
>>>>
>>>> res * cnt + x0 = -44.9900015648454428
>>>> linspace()[-1]: -44.9900016784667969
>>>>
>>>> that's as good as you are ever going to get with 32 bit floats...
>>>>
>>>
>>> Consistency is the key thing. I am fine with one of those values, so
>>> long as that value is what happens no matter which way I specify my grid.
>>>
>>>
>>>>
>>>> Though I just noticed something about your numbers -- there should be a
>>>> nice even base ten delta if you have 7001 gaps -- but linspace produces N
>>>> points, not N gaps -- so maybe you want:
>>>>
>>>>
>>>> In [*17*]: l = np.linspace(-115.0, -44.99, 7002)
>>>>
>>>>
>>>> In [*18*]: l[:5]
>>>>
>>>> Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])
>>>>
>>>>
>>>> In [*19*]: l[-5:]
>>>>
>>>> Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])
>>>>
>>>>
>>>> or, in float32 -- not as pretty:
>>>>
>>>>
>>>> In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)
>>>>
>>>>
>>>> In [*21*]: l[:5]
>>>>
>>>> Out[*21*]:
>>>>
>>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>>
>>>>        -114.95999908], dtype=float32)
>>>>
>>>>
>>>> In [*22*]: l[-5:]
>>>>
>>>> Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.
>>>>   , -44.99000168], dtype=float32)
>>>>
>>>>
>>>> but still as good as you get with float32, and exactly the same result
>>>> as computing in float64 and converting:
>>>>
>>>>
>>>>
>>>> In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)
>>>>
>>>>
>>>> In [*26*]: l[:5]
>>>>
>>>> Out[*26*]:
>>>>
>>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>>
>>>>        -114.95999908], dtype=float32)
>>>>
>>>>
>>>> In [*27*]: l[-5:]
>>>>
>>>> Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.
>>>>   , -44.99000168], dtype=float32)
>>>>
>>>
>>> Argh! I got myself mixed up between specifying pixel corners versus
>>> pixel centers. rasterio has been messing me up on this.
>>>
>>>
>>>>
>>>>
>>>>>> So, does it make any sense to improve arange by utilizing fma() under
>>>>>> the hood?
>>>>>>
>>>>>
>>>> no -- this is simply not the right use-case for arange() anyway.
>>>>
>>>
>>> arange() has accuracy problems, so why not fix it?
>>>
>>> >>> l4 = np.arange(-115, -44.99, 0.01, dtype=np.float32)
>>> >>> np.median(np.diff(l4))
>>> 0.0099945068
>>> >>> np.float32(0.01)
>>> 0.0099999998
>>>
>>> There is something significantly wrong here if arange(), which takes a
>>> resolution parameter, can't seem to produce a sequence with the proper
>>> delta.
>>>
>>>
>>>
>>>>
>>>>
>>>>> Also, any plans for making fma() available as a ufunc?
>>>>>>
>>>>>
>>>> could be nice -- especially if used internally.
>>>>
>>>>
>>>>> Notice that most of my examples required knowing the number of grid
>>>>>> points ahead of time. But what if I didn't know that? What if I just have
>>>>>> the bounds and the resolution? Then arange() is the natural fit, but as I
>>>>>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>>>>>> a closed interval.
>>>>>>
>>>>>
>>>> no -- it's not -- if you have the bounds and the resolution, you have
>>>> an over-specified problem. That is:
>>>>
>>>> x_min + (n * delta_x) == x_max
>>>>
>>>> If there is ANY error in either delta_x or x_max (or x_min), then
>>>> you'll get a missmatch. which is why arange is not the answer (you can make
>>>> the algorithm a bit more accurate, I suppose but there is still fp limited
>>>> precision -- if you can't exactly represent either delta_x or x_max, then
>>>> you CAN'T use the arange() definition and expect to work consistently.
>>>>
>>>> The "right" way to do it is to compute N with: round((x_max - x_min) /
>>>> delta), and then use linspace:
>>>>
>>>> linspace(x_min, x_max, N+1)
>>>>
>>>> (note that it's too bad you need to do N+1 -- if I had to do it over
>>>> again, I'd use N as the number of "gaps" rather than the number of points
>>>> -- that's more commonly what people want, if they care at all)
>>>>
>>>> This way, you get a grid with the endpoints as exact as they can be,
>>>> and the deltas as close to each-other as they can be as well.
>>>>
>>>> maybe you can do a better algorithm in linspace to save an ULP, but
>>>> it's hard to imagine when that would matter.
>>>>
>>>
>>> Yes, it is overspecified. My problem is that different tools require
>>> different specs (ahem... rasterio/gdal), and I have gird specs coming from
>>> other sources. And I need to produce data onto the same grid so that tools
>>> like xarray won't yell at me when I am trying to do an operation between
>>> gridded data that should have the same coordinates, but are off slightly
>>> because they were computed differently for whatever reason.
>>>
>>> I guess I am crying out for some sort of tool that will help the
>>> community stop making the same mistakes. A one-stop shop that'll allow us
>>> to specify a grid in a few different ways and still produce the right
>>> thing, and even do the inverse... provide a coordinate array and get grids
>>> specs in whatever form we want. Maybe even have options for dealing with
>>> pixel corner vs. pixel centers, too? There are additional fun problems such
>>> as padding out coordinate arrays, which np.pad doesn't really do a great
>>> job with.
>>>
>>> Cheers!
>>> Ben Root
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180209/a5a70414/attachment-0001.html>

From matthias.geier at gmail.com  Sat Feb 10 05:39:33 2018
From: matthias.geier at gmail.com (Matthias Geier)
Date: Sat, 10 Feb 2018 11:39:33 +0100
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CANNq6Fnu2FyvmQ8JZWq=Gs+vK3D5Psx77TAkGZZH73ftgKEUqQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
 <CAL1kJvBspn0oV6UcsJV_67_fv75VNP1g-nhOZnm_zSNx8591CQ@mail.gmail.com>
 <CANNq6Fnu2FyvmQ8JZWq=Gs+vK3D5Psx77TAkGZZH73ftgKEUqQ@mail.gmail.com>
Message-ID: <CAFesC-eDnFe0Tq4veCRqY1XVdvXXgdP7CdV5WfUZ7wYuUBx2Jw@mail.gmail.com>

I just want to add a few links related to the topic:

https://mail.python.org/pipermail/numpy-discussion/2007-September/029129.html
https://quantumwise.com/forum/index.php?topic=110.0#.VIVgyIctjRZ
https://mail.python.org/pipermail/numpy-discussion/2012-February/060238.html
http://nbviewer.jupyter.org/github/mgeier/python-audio/blob/master/misc/arange.ipynb

cheers,
Matthias

On Fri, Feb 9, 2018 at 11:17 PM, Benjamin Root <ben.v.root at gmail.com> wrote:

> Interesting...
>
> ```
> static void
> @NAME at _fill(@type@ *buffer, npy_intp length, void *NPY_UNUSED(ignored))
> {
> npy_intp i;
> @type@ start = buffer[0];
> @type@ delta = buffer[1];
> delta -= start;
> for (i = 2; i < length; ++i) {
> buffer[i] = start + i*delta;
> }
> }
> ```
>
> So, the second value is computed using the delta arange was given, but
> then tries to get the delta back, which incurs errors:
> ```
> >>> a = np.float32(-115)
> >>> delta = np.float32(0.01)
> >>> b = a + delta
> >>> new_delta = b - a
> >>> "%.16f" % delta
> '0.0099999997764826'
> >>> "%.16f" % new_delta
> '0.0100021362304688'
> ```
>
> Also, right there is a good example of where the use of fma() could be of
> value.
>
> Cheers!
> Ben Root
>
>
> On Fri, Feb 9, 2018 at 4:56 PM, Eric Wieser <wieser.eric+numpy at gmail.com>
> wrote:
>
>> Can?t arange and linspace operations with floats be done internally
>>
>> Yes, and they probably should be - they?re done this way as a hack
>> because the api exposed for custom dtypes is here
>> <https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/include/numpy/ndarraytypes.h#L507-L511>,
>> (example implementation here
>> <https://github.com/numpy/numpy/blob/81e15e812574d956fcc304c3982e2b59aa18aafb/numpy/core/src/multiarray/arraytypes.c.src#L3711-L3721>)
>> - essentially, you give it the first two elements of the array, and ask it
>> to fill in the rest.
>> ?
>>
>> On Fri, 9 Feb 2018 at 13:17 Matthew Harrigan <harrigan.matthew at gmail.com>
>> wrote:
>>
>>> I apologize if I'm missing something basic, but why are floats being
>>> accumulated in the first place?  Can't arange and linspace operations with
>>> floats be done internally similar to `start + np.arange(num_steps) *
>>> step_size`?  I.e. always accumulate (really increment) integers to limit
>>> errors.
>>>
>>> On Fri, Feb 9, 2018 at 3:43 PM, Benjamin Root <ben.v.root at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Feb 9, 2018 at 12:19 PM, Chris Barker <chris.barker at noaa.gov>
>>>> wrote:
>>>>
>>>>> On Wed, Feb 7, 2018 at 12:09 AM, Ralf Gommers <ralf.gommers at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>  It is partly a plea for some development of numerically accurate
>>>>>>> functions for computing lat/lon grids from a combination of inputs: bounds,
>>>>>>> counts, and resolutions.
>>>>>>>
>>>>>>
>>>>> Can you be more specific about what problems you've run into -- I work
>>>>> with lat-lon grids all the time, and have never had a problem.
>>>>>
>>>>> float32 degrees gives you about 1 meter accuracy or better, so I can
>>>>> see how losing a few digits might be an issue, though I would argue that
>>>>> you maybe shouldn't use float32 if you are worried about anything close to
>>>>> 1m accuracy... -- or shift to a relative coordinate system of some sort.
>>>>>
>>>>
>>>> The issue isn't so much the accuracy of the coordinates themselves. I
>>>> am only worried about 1km resolution (which is approximately 0.01 degrees
>>>> at mid-latitudes). My concern is with consistent *construction* of a
>>>> coordinate grid with even spacing. As it stands right now. If I provide a
>>>> corner coordinate, a resolution, and the number of pixels, the result is
>>>> not terrible (indeed, this is the approach used by gdal/rasterio). If I
>>>> have start/end coordinates and the number of pixels, the result is not bad,
>>>> either (use linspace). But, if I have start/end coordinates and a
>>>> resolution, then determining the number of pixels from that is actually
>>>> tricky to get right in the general case, especially with float32 and large
>>>> grids, and especially if the bounding box specified isn't exactly divisible
>>>> by the resolution.
>>>>
>>>>
>>>>>
>>>>> I have been playing around with the decimal package a bit lately,
>>>>>>>
>>>>>>
>>>>> sigh. decimal is so often looked at a solution to a problem it isn't
>>>>> designed for. lat-lon is natively Sexagesimal -- maybe we need that dtype
>>>>> :-)
>>>>>
>>>>> what you get from decimal is variable precision -- maybe a binary
>>>>> variable precision lib is a better answer -- that would be a good thing to
>>>>> have easy access to in numpy, but in this case, if you want better accuracy
>>>>> in a computation that will end up in float32, just use float64.
>>>>>
>>>>
>>>> I am not concerned about computing distances or anything like that, I
>>>> am trying to properly construct my grid. I need consistent results
>>>> regardless of which way the grid is specified (start/end/count,
>>>> start/res/count, start/end/res). I have found that loading up the grid
>>>> specs (using in a config file or command-line) using the Decimal class
>>>> allows me to exactly and consistently represent the grid specification, and
>>>> gets me most of the way there. But the problems with arange() is
>>>> frustrating, and I have to have extra logic to go around that and over to
>>>> linspace() instead.
>>>>
>>>>
>>>>>
>>>>> and I discovered the concept of "fused multiply-add" operations for
>>>>>>> improved accuracy. I have come to realize that fma operations could be used
>>>>>>> to greatly improve the accuracy of linspace() and arange().
>>>>>>>
>>>>>>
>>>>> arange() is problematic for non-integer use anyway, by its very
>>>>> definition (getting the "end point" correct requires the right step, even
>>>>> without FP error).
>>>>>
>>>>> and would it really help with linspace? it's computing a delta with
>>>>> one division in fp, then multiplying it by an integer (represented in fp --
>>>>> why? why not keep that an integer till the multiply?).
>>>>>
>>>>
>>>> Sorry, that was a left-over from a previous draft of my email after I
>>>> discovered that linspace's accuracy was on par with fma(). And while
>>>> arange() has inherent problems, it can still be made better than it is now.
>>>> In fact, I haven't investigated this, but I did recently discover some unit
>>>> tests of mine started to fail after a numpy upgrade, and traced it back to
>>>> a reduction in the accuracy of a usage of arange() with float32s. So,
>>>> something got worse at some point, which means we could still get accuracy
>>>> back if we can figure out what changed.
>>>>
>>>>
>>>>>
>>>>> In particular, I have been needing improved results for computing
>>>>>>> latitude/longitude grids, which tend to be done in float32's to save memory
>>>>>>> (at least, this is true in data I come across).
>>>>>>>
>>>>>>
>>>>>> If you care about saving memory *and* accuracy, wouldn't it make more
>>>>>> sense to do your computations in float64, and convert to float32 at the
>>>>>> end?
>>>>>>
>>>>>
>>>>> that does seem to be the easy option :-)
>>>>>
>>>>
>>>> Kinda missing the point, isn't it? Isn't that like saying "convert all
>>>> your data to float64s prior to calling np.mean()"? That's ridiculous.
>>>> Instead, we made np.mean() upcast the inner-loop operation, and even allow
>>>> an option to specify what the dtype that should be used for the aggregator.
>>>>
>>>>
>>>>>
>>>>>
>>>>>> Now, to the crux of my problem. It is next to impossible to generate
>>>>>>> a non-trivial numpy array of coordinates, even in double precision, without
>>>>>>> hitting significant numerical errors.
>>>>>>>
>>>>>>
>>>>> I'm confused, the example you posted doesn't have significant errors...
>>>>>
>>>>
>>>> Hmm, "errors" was the wrong word. "Differences between methods" might
>>>> be more along the lines of what I was thinking. Remember, I am looking for
>>>> consistency.
>>>>
>>>>
>>>>>
>>>>>
>>>>>> Which has lead me down the path of using the decimal package (which
>>>>>>> doesn't play very nicely with numpy because of the lack of casting rules
>>>>>>> for it). Consider the following:
>>>>>>> ```
>>>>>>> $ cat test_fma.py
>>>>>>> from __future__ import print_function
>>>>>>> import numpy as np
>>>>>>> res = np.float32(0.01)
>>>>>>> cnt = 7001
>>>>>>> x0 = np.float32(-115.0)
>>>>>>> x1 = res * cnt + x0
>>>>>>> print("res * cnt + x0 = %.16f" % x1)
>>>>>>> x = np.arange(-115.0, -44.99 + (res / 2), 0.01, dtype='float32')
>>>>>>> print("len(arange()): %d  arange()[-1]: %16f" % (len(x), x[-1]))
>>>>>>> x = np.linspace(-115.0, -44.99, cnt, dtype='float32')
>>>>>>> print("linspace()[-1]: %.16f" % x[-1])
>>>>>>>
>>>>>>> $ python test_fma.py
>>>>>>> res * cnt + x0 = -44.9900015648454428
>>>>>>> len(arange()): 7002  arange()[-1]:       -44.975044
>>>>>>> linspace()[-1]: -44.9900016784667969
>>>>>>> ```
>>>>>>> arange just produces silly results (puts out an extra element...
>>>>>>> adding half of the resolution is typically mentioned as a solution on
>>>>>>> mailing lists to get around arange()'s limitations -- I personally don't do
>>>>>>> this).
>>>>>>>
>>>>>>
>>>>> The real solution is "don't do that" arange is not the right tool for
>>>>> the job.
>>>>>
>>>>
>>>> Well, it isn't the right tool because as far as I am concerned, it is
>>>> useless for anything but integers. Why not fix it to be more suitable for
>>>> floating point?
>>>>
>>>>
>>>>>
>>>>> Then there is this:
>>>>>
>>>>> res * cnt + x0 = -44.9900015648454428
>>>>> linspace()[-1]: -44.9900016784667969
>>>>>
>>>>> that's as good as you are ever going to get with 32 bit floats...
>>>>>
>>>>
>>>> Consistency is the key thing. I am fine with one of those values, so
>>>> long as that value is what happens no matter which way I specify my grid.
>>>>
>>>>
>>>>>
>>>>> Though I just noticed something about your numbers -- there should be
>>>>> a nice even base ten delta if you have 7001 gaps -- but linspace produces N
>>>>> points, not N gaps -- so maybe you want:
>>>>>
>>>>>
>>>>> In [*17*]: l = np.linspace(-115.0, -44.99, 7002)
>>>>>
>>>>>
>>>>> In [*18*]: l[:5]
>>>>>
>>>>> Out[*18*]: array([-115.  , -114.99, -114.98, -114.97, -114.96])
>>>>>
>>>>>
>>>>> In [*19*]: l[-5:]
>>>>>
>>>>> Out[*19*]: array([-45.03, -45.02, -45.01, -45.  , -44.99])
>>>>>
>>>>>
>>>>> or, in float32 -- not as pretty:
>>>>>
>>>>>
>>>>> In [*20*]: l = np.linspace(-115.0, -44.99, 7002, dtype=np.float32)
>>>>>
>>>>>
>>>>> In [*21*]: l[:5]
>>>>>
>>>>> Out[*21*]:
>>>>>
>>>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>>>
>>>>>        -114.95999908], dtype=float32)
>>>>>
>>>>>
>>>>> In [*22*]: l[-5:]
>>>>>
>>>>> Out[*22*]: array([-45.02999878, -45.02000046, -45.00999832, -45.
>>>>>   , -44.99000168], dtype=float32)
>>>>>
>>>>>
>>>>> but still as good as you get with float32, and exactly the same result
>>>>> as computing in float64 and converting:
>>>>>
>>>>>
>>>>>
>>>>> In [*25*]: l = np.linspace(-115.0, -44.99, 7002).astype(np.float32)
>>>>>
>>>>>
>>>>> In [*26*]: l[:5]
>>>>>
>>>>> Out[*26*]:
>>>>>
>>>>> array([-115.        , -114.98999786, -114.98000336, -114.97000122,
>>>>>
>>>>>        -114.95999908], dtype=float32)
>>>>>
>>>>>
>>>>> In [*27*]: l[-5:]
>>>>>
>>>>> Out[*27*]: array([-45.02999878, -45.02000046, -45.00999832, -45.
>>>>>   , -44.99000168], dtype=float32)
>>>>>
>>>>
>>>> Argh! I got myself mixed up between specifying pixel corners versus
>>>> pixel centers. rasterio has been messing me up on this.
>>>>
>>>>
>>>>>
>>>>>
>>>>>>> So, does it make any sense to improve arange by utilizing fma()
>>>>>>> under the hood?
>>>>>>>
>>>>>>
>>>>> no -- this is simply not the right use-case for arange() anyway.
>>>>>
>>>>
>>>> arange() has accuracy problems, so why not fix it?
>>>>
>>>> >>> l4 = np.arange(-115, -44.99, 0.01, dtype=np.float32)
>>>> >>> np.median(np.diff(l4))
>>>> 0.0099945068
>>>> >>> np.float32(0.01)
>>>> 0.0099999998
>>>>
>>>> There is something significantly wrong here if arange(), which takes a
>>>> resolution parameter, can't seem to produce a sequence with the proper
>>>> delta.
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>> Also, any plans for making fma() available as a ufunc?
>>>>>>>
>>>>>>
>>>>> could be nice -- especially if used internally.
>>>>>
>>>>>
>>>>>> Notice that most of my examples required knowing the number of grid
>>>>>>> points ahead of time. But what if I didn't know that? What if I just have
>>>>>>> the bounds and the resolution? Then arange() is the natural fit, but as I
>>>>>>> showed, its accuracy is lacking, and you have to do some sort of hack to do
>>>>>>> a closed interval.
>>>>>>>
>>>>>>
>>>>> no -- it's not -- if you have the bounds and the resolution, you have
>>>>> an over-specified problem. That is:
>>>>>
>>>>> x_min + (n * delta_x) == x_max
>>>>>
>>>>> If there is ANY error in either delta_x or x_max (or x_min), then
>>>>> you'll get a missmatch. which is why arange is not the answer (you can make
>>>>> the algorithm a bit more accurate, I suppose but there is still fp limited
>>>>> precision -- if you can't exactly represent either delta_x or x_max, then
>>>>> you CAN'T use the arange() definition and expect to work consistently.
>>>>>
>>>>> The "right" way to do it is to compute N with: round((x_max - x_min) /
>>>>> delta), and then use linspace:
>>>>>
>>>>> linspace(x_min, x_max, N+1)
>>>>>
>>>>> (note that it's too bad you need to do N+1 -- if I had to do it over
>>>>> again, I'd use N as the number of "gaps" rather than the number of points
>>>>> -- that's more commonly what people want, if they care at all)
>>>>>
>>>>> This way, you get a grid with the endpoints as exact as they can be,
>>>>> and the deltas as close to each-other as they can be as well.
>>>>>
>>>>> maybe you can do a better algorithm in linspace to save an ULP, but
>>>>> it's hard to imagine when that would matter.
>>>>>
>>>>
>>>> Yes, it is overspecified. My problem is that different tools require
>>>> different specs (ahem... rasterio/gdal), and I have gird specs coming from
>>>> other sources. And I need to produce data onto the same grid so that tools
>>>> like xarray won't yell at me when I am trying to do an operation between
>>>> gridded data that should have the same coordinates, but are off slightly
>>>> because they were computed differently for whatever reason.
>>>>
>>>> I guess I am crying out for some sort of tool that will help the
>>>> community stop making the same mistakes. A one-stop shop that'll allow us
>>>> to specify a grid in a few different ways and still produce the right
>>>> thing, and even do the inverse... provide a coordinate array and get grids
>>>> specs in whatever form we want. Maybe even have options for dealing with
>>>> pixel corner vs. pixel centers, too? There are additional fun problems such
>>>> as padding out coordinate arrays, which np.pad doesn't really do a great
>>>> job with.
>>>>
>>>> Cheers!
>>>> Ben Root
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at python.org
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180210/84b4e709/attachment-0001.html>

From nilsc.becker at gmail.com  Sun Feb 11 17:19:11 2018
From: nilsc.becker at gmail.com (Nils Becker)
Date: Sun, 11 Feb 2018 23:19:11 +0100
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
Message-ID: <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>

A slightly longer comment although I have to admit that it misses the
discussion here partially. Still I hope it provides some background to your
question.

Generating equidistantly spaced grids is simply not always possible. The
reason is that the absolute spacing of the possible floating point numbers
depends on their magnitude [1]. The true, equidistant grid may not be
representable in the precision that is used. Consequently, the spacing
between consecutive points in a grid created with linspace can vary:

x = np.linspace(-50000.0, 50000.0, 1000001) # dx = 0.1
np.min(np.diff(x))
> 0.099999999991268851
np.max(np.diff(x))
> 0.10000000000582077

This does not necessarily mean that the calculation of the grid points is
off by more than one ulp. It simply shows that the absolute value of one
ulp changes over the extent of the grid.
It follows that even if the grid is calculated with higher precision (e.g.
fma operations) or even arbitrary precision, we will hit a fundamental
boundary: the finite precision floating point numbers closest to the "true"
grid simply are not equidistantly spaced.
My estimate is that the grid spacing is at most off by ~N ulp. This should
be far too small to have any consequences in calculations based on the grid
points themselves.
However, if you write code that reconstructs the grid spacing from
consecutive grid points, e.g. for scaling the value of an integral, you may
notice it. I did when writing a unit test for Fourier transforms where I
compared to analytical results. Take home message: use dx = (x[-1] -
x[0])/(N-1) instead of dx = x[1] - x[0].

If you - for some reason - want the same grid spacing everywhere you may
choose an appropriate new spacing. What generally seems to work (but not
always) is one that fits the floating point spacing at the largest number
in the grid. In other words add and subtract the largest number in your
grid to the grid spacing:

dx = 0.1
N = 1000001
x0 = -50000.0

# "round" dx, assuming that the maximal magnitude of the grid is ~N*dx
dx = (dx + N * dx) - (N * dx)
> 0.10000000000582077
x = x0 + np.arange(N) * dx
np.max(np.diff(x)) - np.min(np.diff(x))
> 0.0

Curiosly, either by design or accident, arange() seems to do something
similar as was mentioned by Eric. It creates a new grid spacing by adding
and subtracting the starting point of the grid. This often has similar
effect as adding and subtracting N*dx (e.g. if the grid is symmetric around
0.0). Consequently, arange() seems to trade keeping the grid spacing
constant for a larger error in the grid size and consequently in the end
point.

The above discussion somewhat ignored additional errors in the calculation
of the grid points. If every grid point is the result of one multiplication
and one addition, they should stay within one ulp (for x[i] != 0.0). If the
grid is calculated by accumulation they may be larger.

If you are concerned about how arange() or linspace() calculates the grid,
I would recommend to simply use

x = x0 + np.arange(N) * dx

which will yield an accurate representation of the grid you most probably
want (within 1 ulp for x[i] != 0.0). If you do not have x0, N, and dx, try
to infer them from other information on the grid.

To sum this all up: Even if your grid calculation is free of round-off
error, the resulting grid using finite precision floating point numbers
will only be exactly equidistant for appropriate grid spacing dx and
starting values x0. On the other hand neither this effect nor round-off in
the grid calculation should ever play a signification role in your
calculations as the resulting errors are too small.

Some additional comments:

1. Comparison to calculations with decimal can be difficult as not all
simple decimal step sizes are exactly representable as finite floating
point numbers. E.g., in single precision 0.1000000014901161193847656250 is
the number closest to 0.1. To make a good comparison you would have to
compare the output of arange() to a grid calculated to infinite precision
which has the single precision representation of 0.1 as grid spacing and
not its decimal value.

2. Calculating the difference in single precision in your C code as below
yields

    float actual = -44.990000000000000;
    float err1 = fabs(x1 - actual);
    float err2 = fabs(x2 - actual);
$ ./test_fma
x1 -44.9899978637695312  x2 -44.9900016784667969
err1 3.81470e-06  err2 0.00000e+00

If you calculate the ulp error you notice that the non-FMA solution is off
by exactly one ulp, while the FMA solution yields the single precision
float closest to the actual solution (error = 0 ulp). Losing one ulp should
not matter for any application. If it does, use a data type with higher
precision, e.g., double.

3. I wrote Python script that compares grids generated in different ways
with a grid calculated with mpmath in high precision [2]. Maybe it is
useful for further insights.

Cheers
Nils

PS: I realize that I went a little off-topic here and apologize for that.
However, a while ago I spent some time trying to figure this issue out and
thought that this may be an occasion to write some of it down.

[1] http://www.exploringbinary.com/the-spacing-of-binary-flo
ating-point-numbers/
[2] https://ncbecker.de/linear_grid.py
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180211/3584d288/attachment.html>

From eugen.wintersberger at gmail.com  Mon Feb 12 06:13:36 2018
From: eugen.wintersberger at gmail.com (Eugen Wintersberger)
Date: Mon, 12 Feb 2018 12:13:36 +0100
Subject: [Numpy-discussion] Using the C-API iterator with object arrays
Message-ID: <1518434016.25323.16.camel@gmail.com>

Hi there, 
I have a question concerning the numpy iterator C-API. I want to create
a numpy 
string array using NPY_OBJECT as a datatype for creating the array (the
reason I am going for this 
approach is that I do not know the length of the individual strings at
the time I construct 
the array, I only know its shape). The original strings are stored 
in a std::vector<char*> instance. The approach I took was something like
this


    std::vector<char*> buffer = ....;
    NpyIter *iter = NpyIter_New(numpy_array,
                                NPY_ITER_READWRITE | NPY_ITER_C_INDEX | NPY_ITER_REFS_OK,
                                NPY_CORDER , NPY_NO_CASTING,nullptr);
    if(iter==NULL)
    {
      return;
    }
    NpyIter_IterNextFunc *iternext = NpyIter_GetIterNext(iter,nullptr);
    if(iternext == NULL)
    {
      std::cerr<<"Could not instantiate next iterator function"<<std::endl;
      return;
    }
    PyObject **dataptr = (PyObject**)NpyIter_GetDataPtrArray(iter);
    for(auto string: buffer)
    {
      dataptr[0] = PyString_FromSting(string); // this string construction seem to work
      iternext(iter);
    }
    NpyIter_Deallocate(iter);


This code snippet is a bit stripped down with all the safety checks
removed to make 
things more readable. 
However, the array I get back still contains only a None instance. Does
anyone have an idea 
what I am doing wrong here?
Thanks in advance.

best regards
   Eugen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/ad11527d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 213 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/ad11527d/attachment-0001.sig>

From jaime.frio at gmail.com  Mon Feb 12 07:56:21 2018
From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=)
Date: Mon, 12 Feb 2018 12:56:21 +0000
Subject: [Numpy-discussion] Using the C-API iterator with object arrays
In-Reply-To: <1518434016.25323.16.camel@gmail.com>
References: <1518434016.25323.16.camel@gmail.com>
Message-ID: <CAPOWHWmkmM+2+XW6wvctVC1+sXdUkBxG3rL0hc426x7ntyrSaw@mail.gmail.com>

On Mon, Feb 12, 2018 at 12:13 PM Eugen Wintersberger <
eugen.wintersberger at gmail.com> wrote:

> Hi there,
> I have a question concerning the numpy iterator C-API. I want to create a
> numpy
> string array using NPY_OBJECT as a datatype for creating the array (the
> reason I am going for this
> approach is that I do not know the length of the individual strings at the
> time I construct
> the array, I only know its shape). The original strings are stored
> in a std::vector<char*> instance. The approach I took was something like
> this
>
>     std::vector<char*> buffer = ....;
>     NpyIter *iter = NpyIter_New(numpy_array,
>                                 NPY_ITER_READWRITE | NPY_ITER_C_INDEX | NPY_ITER_REFS_OK,
>                                 NPY_CORDER , NPY_NO_CASTING,nullptr);
>     if(iter==NULL)
>     {
>       return;
>     }
>     NpyIter_IterNextFunc *iternext = NpyIter_GetIterNext(iter,nullptr);
>     if(iternext == NULL)
>     {
>       std::cerr<<"Could not instantiate next iterator function"<<std::endl;
>       return;
>     }
>     PyObject **dataptr = (PyObject**)NpyIter_GetDataPtrArray(iter);
>     for(auto string: buffer)
>     {
>       dataptr[0] = PyString_FromSting(string); // this string construction seem to work
>       iternext(iter);
>     }
>     NpyIter_Deallocate(iter);
>
>
> This code snippet is a bit stripped down with all the safety checks
> removed to make
> things more readable.
> However, the array I get back still contains only a None instance. Does
> anyone have an idea
> what I am doing wrong here?
>

I think you have the indirections wrong in dataptr?

NpyIter_GetDataPtrArray returns a char**, that hold the address of the
variable where the iterator holds the address of the first byte of the
current item being iterated.

When you write to dataptr[0] you are not writing to the array, but to where
the iterator holds the address of the first byte of the current item.

So you would have to write to dataptr[0][0], or **dataptr, to actually
affect the contents of the array. Of course, your dataptr being a PyObject**,
the compiler would probably complaint about such an assignment.

I think that if you define dataptr as a PyObject*** (yay, three star
programming <http://wiki.c2.com/?ThreeStarProgrammer>!) and then assign to
**dataptr, everything will work fine. If that doesn't work, maybe try to
make dataptr a char**, and then assign to (PyObject *)(**dataptr) = ...

Jaime



> Thanks in advance.
>
> best regards
>    Eugen
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>


-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/bc4d1120/attachment.html>

From ewm at redtetrahedron.org  Mon Feb 12 08:22:36 2018
From: ewm at redtetrahedron.org (Eric Moore)
Date: Mon, 12 Feb 2018 08:22:36 -0500
Subject: [Numpy-discussion] Using the C-API iterator with object arrays
In-Reply-To: <1518434016.25323.16.camel@gmail.com>
References: <1518434016.25323.16.camel@gmail.com>
Message-ID: <CAGeA38kOtLevaNOLmEOO3rJtY-ML2A1mZBbrpbqLWxk_yoz0WQ@mail.gmail.com>

On Mon, Feb 12, 2018 at 6:13 AM, Eugen Wintersberger <
eugen.wintersberger at gmail.com> wrote:

> Hi there,
> I have a question concerning the numpy iterator C-API. I want to create a
> numpy
> string array using NPY_OBJECT as a datatype for creating the array (the
> reason I am going for this
> approach is that I do not know the length of the individual strings at the
> time I construct
> the array, I only know its shape). The original strings are stored
> in a std::vector<char*> instance. The approach I took was something like
> this
>
>     std::vector<char*> buffer = ....;
>     NpyIter *iter = NpyIter_New(numpy_array,
>                                 NPY_ITER_READWRITE | NPY_ITER_C_INDEX | NPY_ITER_REFS_OK,
>                                 NPY_CORDER , NPY_NO_CASTING,nullptr);
>     if(iter==NULL)
>     {
>       return;
>     }
>     NpyIter_IterNextFunc *iternext = NpyIter_GetIterNext(iter,nullptr);
>     if(iternext == NULL)
>     {
>       std::cerr<<"Could not instantiate next iterator function"<<std::endl;
>       return;
>     }
>     PyObject **dataptr = (PyObject**)NpyIter_GetDataPtrArray(iter);
>     for(auto string: buffer)
>     {
>       dataptr[0] = PyString_FromSting(string); // this string construction seem to work
>       iternext(iter);
>     }
>     NpyIter_Deallocate(iter);
>
>
> This code snippet is a bit stripped down with all the safety checks
> removed to make
> things more readable.
> However, the array I get back still contains only a None instance. Does
> anyone have an idea
> what I am doing wrong here?
> Thanks in advance.
>
> best regards
>    Eugen
>
>
Honestly, given that you're iterating over a single 1D array you've just
constructed, I don't think I would even bother trying to use the iterator.
For this case, a simple loop will be both concise and clear.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/2f37f9b1/attachment.html>

From chris.barker at noaa.gov  Mon Feb 12 12:33:23 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 12 Feb 2018 09:33:23 -0800
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
Message-ID: <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>

I think it's all been said, but a few comments:

On Sun, Feb 11, 2018 at 2:19 PM, Nils Becker <nilsc.becker at gmail.com> wrote:

> Generating equidistantly spaced grids is simply not always possible.
>

exactly -- and linspace gives pretty much teh best possible result,
guaranteeing tha tthe start an end points are exact, and the spacing is
within an ULP or two (maybe we could make that within 1 ULP always, but not
sure that's worth it).


> The reason is that the absolute spacing of the possible floating point
> numbers depends on their magnitude [1].
>

Also that the exact spacing may not be exactly representable in FP -- so
you have to have at least one space that's a bit off to get the end points
right (or have the endpoints not exact).


> If you - for some reason - want the same grid spacing everywhere you may
> choose an appropriate new spacing.
>

well, yeah, but usually you are trying to fit to some other constraint. I'm
still confused as to where these couple of ULPs actually cause problems,
unless you are doing in appropriate FP comparisons elsewhere.

Curiously, either by design or accident, arange() seems to do something
> similar as was mentioned by Eric. It creates a new grid spacing by adding
> and subtracting the starting point of the grid. This often has similar
> effect as adding and subtracting N*dx (e.g. if the grid is symmetric around
> 0.0). Consequently, arange() seems to trade keeping the grid spacing
> constant for a larger error in the grid size and consequently in the end
> point.
>

interesting -- but it actually makes sense -- that is the definition of
arange(), borrowed from range(), which was designed for integers, and, in
fact, pretty much mirroered the classic C index for loop:


for (int i=0; i<N; i++) {
...


or in python:

i = start
while i < stop:
    i += step

The problem here is that termination criteria -- i < stop -- that is the
definition of the function, and works just fine for integers (where it came
from), but with FP, even with no error accumulation, stop may not be
exactly representable, so you could end up with a value for your last item
that is about (stop-step), or you could end up with a value that is a
couple ULPs less than step -- essentially including the end point when you
weren't supposed to.

The truth is, making a floating point range() was simply a bad idea to
begin with -- it's not the way to define a range of numbers in floating
point. Whiuch is why the docs now say "When using a non-integer step, such
as 0.1, the results will often not
be consistent.  It is better to use ``linspace`` for these cases."

Ben wants a way to define grids in a consistent way -- make sense. And yes,
sometimes, the original source you are trying to match (like GDAL) provides
a starting point and step. But with FP, that is simply problematic. If:

start + step*num_steps != stop

exactly in floating point, then you'll need to do the math one way or
another to get what you want -- and I'm not sure anyone but the user knows
what they want -- do you want step to be as exact as possible, or do you
want stop to be as exact as possible?

All that being said -- if arange() could be made a tiny bit more accurate
with fma or other numerical technique, why not? it won't solve the problem,
but if someone writes and tests the code (and it does not require compiler
or hardware features that aren't supported everywhere numpy compiles), then
sure. (Same for linspace, though I'm not sure it's possible)

There is one other option: a new function (or option) that makes a grid
from a specification of: start, step, num_points. If that is really a
common use case (that is, you don't care exactly what the end-point is),
then it might be handy to have it as a utility.

We could also have an arange-like function that, rather than < stop, would
do "close to" stop. Someone that understands FP better than I might be able
to compute what the expected error might be, and find the closest end point
within that error. But I think that's a bad specification -- (stop - start)
/ step may be nowhere near an integer -- then what is the function supposed
to do??


BTW: I kind of wish that linspace specified the number of steps, rather
than the number of points, that is (num+points - 1) that would save a
little bit of logical thinking. So if something new is made, let's keep
that in mind.



> 1. Comparison to calculations with decimal can be difficult as not all
> simple decimal step sizes are exactly representable as
>
finite floating point numbers.
>

yeah, this is what I mean by inappropriate use of Decimal -- decimal is not
inherently "more accurate" than fp -- is just can represent _decimal_
numbers exactly, which we are all use to -- we want  1 / 10 to be exact,
but dont mind that 1 / 3 isn't.

Decimal also provided variable precision -- so it can be handy for that. I
kinda wish Python had an arbitrary precision binary floating point built
in...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/1e2ab11a/attachment.html>

From charlesr.harris at gmail.com  Mon Feb 12 13:42:27 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 12 Feb 2018 11:42:27 -0700
Subject: [Numpy-discussion] @xoviat
Message-ID: <CAB6mnx+zH95_+uNtv7PyARUvTBXFHBrBhdF+1Nov4LijGqWKnA@mail.gmail.com>

Anyone know what happened to xoviat? He seems to have been disappeared.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180212/f597084c/attachment.html>

From ralf.gommers at gmail.com  Mon Feb 12 20:32:44 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 13 Feb 2018 02:32:44 +0100
Subject: [Numpy-discussion] @xoviat
In-Reply-To: <CAB6mnx+zH95_+uNtv7PyARUvTBXFHBrBhdF+1Nov4LijGqWKnA@mail.gmail.com>
References: <CAB6mnx+zH95_+uNtv7PyARUvTBXFHBrBhdF+1Nov4LijGqWKnA@mail.gmail.com>
Message-ID: <CABL7CQjsartYaQgXoNBzcyF4F9b_QdSwVEqW-wxb-yn6-M-DBQ@mail.gmail.com>

On Mon, Feb 12, 2018 at 7:42 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> Anyone know what happened to xoviat? He seems to have been disappeared.
>

His account was flagged by GitHub after scipy issue gh-8373. Flagging means
your profile, and all your issue comments, PRs, etc. are all made private.
It is quite bad if they don't come back; GitHub shouldn't break
conversations like that by removing comments on a different repo made well
prior to an incident. We'll be contacting GitHub about this.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180213/344695e2/attachment-0001.html>

From tcaswell at gmail.com  Wed Feb 14 15:47:58 2018
From: tcaswell at gmail.com (Thomas Caswell)
Date: Wed, 14 Feb 2018 20:47:58 +0000
Subject: [Numpy-discussion] NumPy should not silently promote numbers to
 strings
In-Reply-To: <CAEQ_TvchCoRbnDYGvSp81+kRT3oMKpJ53BEUDNNf1F+658xcBQ@mail.gmail.com>
References: <CAEQ_TvfpLysca=UhZv1-gfJ+RoDb+BNFNcEm8B5ku8+TAG_MNw@mail.gmail.com>
 <CAL1kJvBeN5B8mtosBCt2EXLAw+Bvsm8GzDkRwYEZSoPuY+ngZw@mail.gmail.com>
 <CAEQ_TvchCoRbnDYGvSp81+kRT3oMKpJ53BEUDNNf1F+658xcBQ@mail.gmail.com>
Message-ID: <CAA48SF_9W+C7vm=MFrEXHcLh+FXnTW4a-bN=bwq2UY6GSwCJQQ@mail.gmail.com>

This has recently been a major point point for Matplotlib for the
implementation of string-categoricals as well.

Having numpy go to object or fail on `np.asarray([1, 2, 'foo'])` would make
things much easier for us.

Tom

On Fri, Feb 9, 2018 at 2:22 AM Stephan Hoyer <shoyer at gmail.com> wrote:

> On Thu, Feb 8, 2018 at 11:00 PM Eric Wieser <wieser.eric+numpy at gmail.com>
> wrote:
>
>> Presumably you would extend that to all (str, np.number), or even (str,
>> np.generic_)?
>>
> Yes, I'm currently doing (np.character, np.number) and (np.character,
> np.bool_). But only in direct consultation with the diagram of NumPy's
> type hierarchy :).
>
>> I suppose there?s the argument that with python-3-only support around the
>> corner, even (str, bytes) should go to object.
>>
> Yes, that's also pretty bad.
>
> The current behavior (str, bytes) -> str relies on bytes being valid ASCII:
> >>> np.array([b'\xFF', u'cd'])
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
> ordinal not in range(128)
>
> It exactly matches Python 2's str/unicode behavior, but doesn't make sense
> at all in a Python 3 world.
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180214/e71e213d/attachment.html>

From dgasmith at vt.edu  Fri Feb 16 13:03:41 2018
From: dgasmith at vt.edu (Daniel Smith)
Date: Fri, 16 Feb 2018 13:03:41 -0500
Subject: [Numpy-discussion] Permissable NumPy logo usage
Message-ID: <CBCF1A03-51BA-44C5-BFFA-CE755622CCDD@vt.edu>

Hello everyone,
I have a project which combines NumPy and a quantum chemistry program (Psi4, psicode.org <http://psicode.org/>) for education and rapid prototyping. One of the authors has proposed a new logo which is a tweaked form of the the NumPy and Psi4 logos combined. I was curious if we were violating any NumPy copyright or community taboos in either using the NumPy name or a modified version of the NumPy logo. Any advice or direction would be most welcome.

Current logo:
https://github.com/psi4/psi4numpy/blob/master/media/psi4banner_numpy_interactive.png <https://github.com/psi4/psi4numpy/blob/master/media/psi4banner_numpy_interactive.png>

Proposed logo:
https://github.com/loriab/psi4numpy/blob/0866d0fb67f2c9629e2ba37bc4a091e20695a09f/media/psi4numpybanner_eqn.png <https://github.com/loriab/psi4numpy/blob/0866d0fb67f2c9629e2ba37bc4a091e20695a09f/media/psi4numpybanner_eqn.png>

Project link:
https://github.com/psi4/psi4numpy <https://github.com/psi4/psi4numpy>

ChemRxiv:
https://chemrxiv.org/articles/Psi4NumPy_An_Interactive_Quantum_Chemistry_Programming_Environment_for_Reference_Implementations_and_Rapid_Development/5746059 <https://chemrxiv.org/articles/Psi4NumPy_An_Interactive_Quantum_Chemistry_Programming_Environment_for_Reference_Implementations_and_Rapid_Development/5746059>

Cheers,
-Daniel

?
Daniel G. A. Smith
Software Scientist
The Molecular Sciences Software Institute (MolSSI <http://molssi.org/>)
@dgas_smith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180216/1bdb12d5/attachment.html>

From shoyer at gmail.com  Fri Feb 16 15:28:33 2018
From: shoyer at gmail.com (Stephan Hoyer)
Date: Fri, 16 Feb 2018 20:28:33 +0000
Subject: [Numpy-discussion] Permissable NumPy logo usage
In-Reply-To: <CBCF1A03-51BA-44C5-BFFA-CE755622CCDD@vt.edu>
References: <CBCF1A03-51BA-44C5-BFFA-CE755622CCDD@vt.edu>
Message-ID: <CAEQ_TvcPEEwk6coA-C3OgpyMmqD=wbUWK2FpWexEtSEfLma43Q@mail.gmail.com>

I don't know the history of the NumPy logo, or who officially owns the
rights to NumPy's branding at this point. In principle, that might be
NumFOCUS, but the logo far predates NumFOCUS and NumFOCUS's fiscal
sponsorship of NumPy. Looking at the Git history, it looks like David
Cournapeau added it to NumPy's repo back in 2009:
https://github.com/numpy/numpy/commit/c5b2f31aeafa32c705f87f5801a952e394063a3d

Just speaking for myself, I think this use for NumPy's logo and name is
appropriate and within community norms. I don't think anyone would be
confuse your project with NumPy or assume any sort of official endorsement.

On Fri, Feb 16, 2018 at 10:52 AM Daniel Smith <dgasmith at vt.edu> wrote:

> Hello everyone,
> I have a project which combines NumPy and a quantum chemistry program
> (Psi4, psicode.org) for education and rapid prototyping. One of the
> authors has proposed a new logo which is a tweaked form of the the NumPy
> and Psi4 logos combined. I was curious if we were violating any NumPy
> copyright or community taboos in either using the NumPy name or a modified
> version of the NumPy logo. Any advice or direction would be most welcome.
>
> Current logo:
>
> https://github.com/psi4/psi4numpy/blob/master/media/psi4banner_numpy_interactive.png
>
> Proposed logo:
>
> https://github.com/loriab/psi4numpy/blob/0866d0fb67f2c9629e2ba37bc4a091e20695a09f/media/psi4numpybanner_eqn.png
>
> Project link:
> https://github.com/psi4/psi4numpy
>
> ChemRxiv:
>
> https://chemrxiv.org/articles/Psi4NumPy_An_Interactive_Quantum_Chemistry_Programming_Environment_for_Reference_Implementations_and_Rapid_Development/5746059
>
> Cheers,
> -Daniel
>
> ?
> Daniel G. A. Smith
> Software Scientist
> The Molecular Sciences Software Institute (MolSSI <http://molssi.org/>)
> @dgas_smith
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180216/d19b27e1/attachment.html>

From ralf.gommers at gmail.com  Fri Feb 16 15:34:34 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 16 Feb 2018 12:34:34 -0800
Subject: [Numpy-discussion] Permissable NumPy logo usage
In-Reply-To: <CAEQ_TvcPEEwk6coA-C3OgpyMmqD=wbUWK2FpWexEtSEfLma43Q@mail.gmail.com>
References: <CBCF1A03-51BA-44C5-BFFA-CE755622CCDD@vt.edu>
 <CAEQ_TvcPEEwk6coA-C3OgpyMmqD=wbUWK2FpWexEtSEfLma43Q@mail.gmail.com>
Message-ID: <CABL7CQjEAE1CHrFjF7vTX=cd_25uUj8kppjASVp9agdSppGk_Q@mail.gmail.com>

On Fri, Feb 16, 2018 at 12:28 PM, Stephan Hoyer <shoyer at gmail.com> wrote:

> I don't know the history of the NumPy logo, or who officially owns the
> rights to NumPy's branding at this point. In principle, that might be
> NumFOCUS, but the logo far predates NumFOCUS and NumFOCUS's fiscal
> sponsorship of NumPy. Looking at the Git history, it looks like David
> Cournapeau added it to NumPy's repo back in 2009:
> https://github.com/numpy/numpy/commit/c5b2f31aeafa32c705f87f5801a952
> e394063a3d
>

IIRC the NumPy name is trademarked, and Travis gave NumFOCUS the ownership
of that. That should also cover the logo I'd think.


>
>
> Just speaking for myself, I think this use for NumPy's logo and name is
> appropriate and within community norms. I don't think anyone would be
> confuse your project with NumPy or assume any sort of official endorsement.
>

I agree.

Ralf



>
> On Fri, Feb 16, 2018 at 10:52 AM Daniel Smith <dgasmith at vt.edu> wrote:
>
>> Hello everyone,
>> I have a project which combines NumPy and a quantum chemistry program
>> (Psi4, psicode.org) for education and rapid prototyping. One of the
>> authors has proposed a new logo which is a tweaked form of the the NumPy
>> and Psi4 logos combined. I was curious if we were violating any NumPy
>> copyright or community taboos in either using the NumPy name or a modified
>> version of the NumPy logo. Any advice or direction would be most welcome.
>>
>> Current logo:
>> https://github.com/psi4/psi4numpy/blob/master/media/
>> psi4banner_numpy_interactive.png
>>
>> Proposed logo:
>> https://github.com/loriab/psi4numpy/blob/0866d0fb67f2c9629e2ba37bc4a091
>> e20695a09f/media/psi4numpybanner_eqn.png
>>
>> Project link:
>> https://github.com/psi4/psi4numpy
>>
>> ChemRxiv:
>> https://chemrxiv.org/articles/Psi4NumPy_An_Interactive_
>> Quantum_Chemistry_Programming_Environment_for_Reference_
>> Implementations_and_Rapid_Development/5746059
>>
>> Cheers,
>> -Daniel
>>
>> ?
>> Daniel G. A. Smith
>> Software Scientist
>> The Molecular Sciences Software Institute (MolSSI <http://molssi.org/>)
>> @dgas_smith
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180216/e6345f7e/attachment-0001.html>

From eugen.wintersberger at gmail.com  Sat Feb 17 09:11:37 2018
From: eugen.wintersberger at gmail.com (Eugen Wintersberger)
Date: Sat, 17 Feb 2018 15:11:37 +0100
Subject: [Numpy-discussion] Using the C-API iterator with object arrays
In-Reply-To: <CAPOWHWmkmM+2+XW6wvctVC1+sXdUkBxG3rL0hc426x7ntyrSaw@mail.gmail.com>
References: <1518434016.25323.16.camel@gmail.com>
 <CAPOWHWmkmM+2+XW6wvctVC1+sXdUkBxG3rL0hc426x7ntyrSaw@mail.gmail.com>
Message-ID: <1518876697.3713.1.camel@gmail.com>

Hi Jaime?
On Mon, 2018-02-12 at 12:56 +0000, Jaime Fern?ndez del R?o wrote:
> On Mon, Feb 12, 2018 at 12:13 PM Eugen Wintersberger <eugen.wintersbe
> rger at gmail.com> wrote:
> 
> > 
> > ??
> > ??
> > 
> > 
> > Hi there,?
> > 
> > I have a question concerning the numpy iterator C-API. I want to
> > create a numpy?
> > 
> > string array using NPY_OBJECT as a datatype for creating the array
> > (the reason I am going for this?
> > 
> > approach is that I do not know the length of the individual strings
> > at the time I construct?
> > 
> > the array, I only know its shape). The original strings are stored?
> > 
> 
> I think you have the indirections wrong in dataptr?

You are absolutely right. Now everything works just as it should.
Thanks a lot. 
> NpyIter_GetDataPtrArray?returns a?char**, that hold the address of
> the variable where the iterator holds the address of the first byte
> of the current item being iterated.
> When you write to?dataptr[0]?you are not writing to the array, but to
> where the iterator holds the address of the first byte of the current
> item.
> So you would have to write to?dataptr[0][0], or?**dataptr, to
> actually affect the contents of the array. Of course,
> your?dataptr?being a?PyObject**, the compiler would probably
> complaint about such an assignment.
> I think that if you define?dataptr?as a?PyObject***?(yay,?three star
> programming!) and then assign to?**dataptr, everything will work
> fine. If that doesn't work, maybe try to make?dataptr?a?char**, and
> then assign to?(PyObject *)(**dataptr) = ...?
> Jaime
> ?
> > Thanks in advance.
> > 
> > 
> > 
> > best regards
> > 
> > ?? Eugen
> > 
> > 
> > _______________________________________________
> > 
> > NumPy-Discussion mailing list
> > 
> > NumPy-Discussion at python.org
> > 
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> > 
> > 
> 
> 
> --?
> (\__/)
> ( O.o)
> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus
> planes de dominaci?n mundial.
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180217/6469ad49/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 269 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180217/6469ad49/attachment.sig>

From charlesr.harris at gmail.com  Tue Feb 20 20:21:32 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 20 Feb 2018 18:21:32 -0700
Subject: [Numpy-discussion] NumPy 1.14.1 released
Message-ID: <CAB6mnxL+vL7GQJptvMiBj8NNtdVBTwnozYpo1KAVYdJ6=4H1BQ@mail.gmail.com>

Hi All,

On behalf of the NumPy team, I am pleased to announce NumPy 1.14.1. This is
a bugfix release for some problems reported following the 1.14.0 release.
The major problems fixed are the following.

   - Problems with the new array printing, particularly the printing of
   complex values, Please report any additional problems that may turn up.

   - Problems with ``np.einsum`` due to the new ``optimized=True`` default.
   Some fixes for optimization have been applied and ``optimize=False`` is now
   the default.

   - The sort order in ``np.unique`` when ``axis=<some-number>`` will now
   always be lexicographic in the subarray elements. In previous NumPy
   versions there was an optimization that could result in sorting the
   subarrays as unsigned byte strings.

   - The change in 1.14.0 that multi-field indexing of structured arrays
   returns a view instead of a copy has been reverted but remains on track for
   NumPy 1.15. Affected users should read the 1.14.1 Numpy User Guide section
   "basics/structured arrays/accessing multiple fields" for advice on how to
   manage this transition.

This release supports Python 2.7 and 3.4 - 3.6. Wheels for the release are
available on PyPI. Source tarballs, zipfiles, release notes, and the
changelog are available on github
<https://github.com/numpy/numpy/releases/tag/v1.14.1>.

*Contributors*

A total of 14 people contributed to this release.  People with a "+" by
their names contributed a patch for the first time.

* Allan Haldane
* Charles Harris
* Daniel Smith
* Dennis Weyland +
* Eric Larson
* Eric Wieser
* Jarrod Millman
* Kenichi Maehashi +
* Marten van Kerkwijk
* Mathieu Lamarre
* Sebastian Berg
* Simon Conseil
* Simon Gibbons
* xoviat

Cheers,

Charles Harris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180220/b6b560e6/attachment.html>

From sebastian at sipsolutions.net  Wed Feb 21 05:39:42 2018
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 21 Feb 2018 11:39:42 +0100
Subject: [Numpy-discussion] NumPy 1.14.1 released
In-Reply-To: <CAB6mnxL+vL7GQJptvMiBj8NNtdVBTwnozYpo1KAVYdJ6=4H1BQ@mail.gmail.com>
References: <CAB6mnxL+vL7GQJptvMiBj8NNtdVBTwnozYpo1KAVYdJ6=4H1BQ@mail.gmail.com>
Message-ID: <1519209582.26564.0.camel@sipsolutions.net>

Great news, as always, thanks for your relentless effort Chuck!

- Sebastian


On Tue, 2018-02-20 at 18:21 -0700, Charles R Harris wrote:
> Hi All,
> 
> On behalf of the NumPy team, I am pleased to announce NumPy
> 1.14.1. This is a bugfix release for some problems reported following
> the 1.14.0 release. The major problems fixed are the following.
> Problems with the new array printing, particularly the printing of
> complex values, Please report any additional problems that may turn
> up.
> 
> Problems with ``np.einsum`` due to the new ``optimized=True``
> default. Some fixes for optimization have been applied and
> ``optimize=False`` is now the default.
> 
> The sort order in ``np.unique`` when ``axis=<some-number>`` will now
> always be lexicographic in the subarray elements. In previous NumPy
> versions there was an optimization that could result in sorting the
> subarrays as unsigned byte strings.
> 
> The change in 1.14.0 that multi-field indexing of structured arrays
> returns a view instead of a copy has been reverted but remains on
> track for NumPy 1.15. Affected users should read the 1.14.1 Numpy
> User Guide section "basics/structured arrays/accessing multiple
> fields" for advice on how to manage this transition.
> This release supports Python 2.7 and 3.4 - 3.6. Wheels for the
> release are available on PyPI. Source tarballs, zipfiles, release
> notes, and the changelog are available on github.
> 
> Contributors
> 
> A total of 14 people contributed to this release.  People with a "+"
> by their names contributed a patch for the first time.
> 
> * Allan Haldane
> * Charles Harris
> * Daniel Smith
> * Dennis Weyland +
> * Eric Larson
> * Eric Wieser
> * Jarrod Millman
> * Kenichi Maehashi +
> * Marten van Kerkwijk
> * Mathieu Lamarre
> * Sebastian Berg
> * Simon Conseil
> * Simon Gibbons
> * xoviat
> 
> Cheers,
> 
> Charles Harris
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180221/b6da5e0b/attachment.sig>

From kevin.k.sheppard at gmail.com  Thu Feb 22 06:03:06 2018
From: kevin.k.sheppard at gmail.com (Kevin Sheppard)
Date: Thu, 22 Feb 2018 11:03:06 +0000
Subject: [Numpy-discussion] Prototype CorePRNG implementation feedback
In-Reply-To: <mailman.34.1516467602.29251.numpy-discussion@python.org>
References: <mailman.34.1516467602.29251.numpy-discussion@python.org>
Message-ID: <CAE8Ss2E30kjayc85MHcGuU+HekZF9-oSw+st5FRWwOLZjoQR9w@mail.gmail.com>

I have implemented a working prototype of a pluggable RandomState.  The
repo is located at https://github.com/bashtage/core-prng.

The main design has a small PRNG (currently only SplitMix64 and
Xoroshiro128 are implemented) that does not have any public functions
available to generate random numbers.  These objects only expose methods to
get and set state and to jump the PRNG.

The random number generator is exposed via an opaque structure wrapped in a
PyCapsule.  This structure has two void pointers, one to the state and one
to the function that generates the next uint 64.  I think in the long-run
it might make sense to expose a few additional interfaces, such as the next
random double (makes sense for dSFMT) or the next random uint32 (makes
sense for MT19937).  For now, I have deliberately kept it simple.

The user-facing object is called `RandomGenerator` and it can be
initialized using the syntax `RandomGenerator` or
`RandomGenerator(Xoroshiro128())`.  The object exposes user-facing methods
(currently only `random_integer` and `random_double`).

A basic demo:

from core_prng.generator import RandomGenerator
# Splitmix 64 for now
RandomGenerator().random_integer()

from core_prng.xoroshiro128 import Xoroshiro128
RandomGenerator(Xoroshiro128()).random_integer()

A few questions that have come up:

   1. Should a passed core PRNG be initialized or not. The syntax
RandomGenerator(Xoroshiro128)
   looks nicer than RandomGenerator(Xoroshiro128())
   2. Which low-level methods should be part of the core PRNG?  These would
   all be scalar generators that would be consumed by RandomGenerator and
   exposed through void *. I could imagine some subset of the following:
      - uint64
      - uint32
      - double
      - float
      - uint53 (applicable to PRNGs that use doubles as default type)
   3. Since RandomState is taken, a new name is needed for the user-facing
   object.  RandomGenerator is a placeholder but ideally, something meaningful
   could be chosen.
   4. I can't see how locking can be usefully implemented in the core PRNGs
   without a large penalty. This means that these are not threadsafe.  This
   isn't a massive problem but they do access the state (get or set) although
   with GIL.
   5. Should state be a property -- this isn't Java...


Other feedback is of course also welcome.

Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/c8fee1cc/attachment.html>

From ndbecker2 at gmail.com  Thu Feb 22 06:43:27 2018
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 22 Feb 2018 11:43:27 +0000
Subject: [Numpy-discussion] Prototype CorePRNG implementation feedback
In-Reply-To: <CAE8Ss2E30kjayc85MHcGuU+HekZF9-oSw+st5FRWwOLZjoQR9w@mail.gmail.com>
References: <mailman.34.1516467602.29251.numpy-discussion@python.org>
 <CAE8Ss2E30kjayc85MHcGuU+HekZF9-oSw+st5FRWwOLZjoQR9w@mail.gmail.com>
Message-ID: <CAG3t+pH+Kd4icrVru_1CJKbSgBd_tSi=7c_mqZUM8VF8CQRROA@mail.gmail.com>

What is the syntax to construct an initialized generator?
RandomGenerator(Xoroshiro128(my_seed))?

On Thu, Feb 22, 2018 at 6:06 AM Kevin Sheppard <kevin.k.sheppard at gmail.com>
wrote:

> I have implemented a working prototype of a pluggable RandomState.  The
> repo is located at https://github.com/bashtage/core-prng.
>
> The main design has a small PRNG (currently only SplitMix64 and
> Xoroshiro128 are implemented) that does not have any public functions
> available to generate random numbers.  These objects only expose methods to
> get and set state and to jump the PRNG.
>
> The random number generator is exposed via an opaque structure wrapped in
> a PyCapsule.  This structure has two void pointers, one to the state and
> one to the function that generates the next uint 64.  I think in the
> long-run it might make sense to expose a few additional interfaces, such as
> the next random double (makes sense for dSFMT) or the next random uint32
> (makes sense for MT19937).  For now, I have deliberately kept it simple.
>
> The user-facing object is called `RandomGenerator` and it can be
> initialized using the syntax `RandomGenerator` or
> `RandomGenerator(Xoroshiro128())`.  The object exposes user-facing methods
> (currently only `random_integer` and `random_double`).
>
> A basic demo:
>
> from core_prng.generator import RandomGenerator
> # Splitmix 64 for now
> RandomGenerator().random_integer()
>
> from core_prng.xoroshiro128 import Xoroshiro128
> RandomGenerator(Xoroshiro128()).random_integer()
>
> A few questions that have come up:
>
>    1. Should a passed core PRNG be initialized or not. The syntax RandomGenerator(Xoroshiro128)
>    looks nicer than RandomGenerator(Xoroshiro128())
>    2. Which low-level methods should be part of the core PRNG?  These
>    would all be scalar generators that would be consumed by RandomGenerator
>    and exposed through void *. I could imagine some subset of the
>    following:
>       - uint64
>       - uint32
>       - double
>       - float
>       - uint53 (applicable to PRNGs that use doubles as default type)
>    3. Since RandomState is taken, a new name is needed for the
>    user-facing object.  RandomGenerator is a placeholder but ideally,
>    something meaningful could be chosen.
>    4. I can't see how locking can be usefully implemented in the core
>    PRNGs without a large penalty. This means that these are not threadsafe.
>    This isn't a massive problem but they do access the state (get or set)
>    although with GIL.
>    5. Should state be a property -- this isn't Java...
>
>
> Other feedback is of course also welcome.
>
> Kevin
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/a51af2fd/attachment.html>

From chris.barker at noaa.gov  Thu Feb 22 14:02:29 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 22 Feb 2018 11:02:29 -0800
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
 <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>
Message-ID: <CALGmxEL5C27Zy4ihMOxRNYbx2Sgb=Gr_xfeBkEbsrftQ6rTLOw@mail.gmail.com>

@Ben: Have you found a solution to your problem? Are there thinks we could
do in numpy to make it better?

-CHB


On Mon, Feb 12, 2018 at 9:33 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> I think it's all been said, but a few comments:
>
> On Sun, Feb 11, 2018 at 2:19 PM, Nils Becker <nilsc.becker at gmail.com>
> wrote:
>
>> Generating equidistantly spaced grids is simply not always possible.
>>
>
> exactly -- and linspace gives pretty much teh best possible result,
> guaranteeing tha tthe start an end points are exact, and the spacing is
> within an ULP or two (maybe we could make that within 1 ULP always, but not
> sure that's worth it).
>
>
>> The reason is that the absolute spacing of the possible floating point
>> numbers depends on their magnitude [1].
>>
>
> Also that the exact spacing may not be exactly representable in FP -- so
> you have to have at least one space that's a bit off to get the end points
> right (or have the endpoints not exact).
>
>
>> If you - for some reason - want the same grid spacing everywhere you may
>> choose an appropriate new spacing.
>>
>
> well, yeah, but usually you are trying to fit to some other constraint.
> I'm still confused as to where these couple of ULPs actually cause
> problems, unless you are doing in appropriate FP comparisons elsewhere.
>
> Curiously, either by design or accident, arange() seems to do something
>> similar as was mentioned by Eric. It creates a new grid spacing by adding
>> and subtracting the starting point of the grid. This often has similar
>> effect as adding and subtracting N*dx (e.g. if the grid is symmetric around
>> 0.0). Consequently, arange() seems to trade keeping the grid spacing
>> constant for a larger error in the grid size and consequently in the end
>> point.
>>
>
> interesting -- but it actually makes sense -- that is the definition of
> arange(), borrowed from range(), which was designed for integers, and, in
> fact, pretty much mirroered the classic C index for loop:
>
>
> for (int i=0; i<N; i++) {
> ...
>
>
> or in python:
>
> i = start
> while i < stop:
>     i += step
>
> The problem here is that termination criteria -- i < stop -- that is the
> definition of the function, and works just fine for integers (where it came
> from), but with FP, even with no error accumulation, stop may not be
> exactly representable, so you could end up with a value for your last item
> that is about (stop-step), or you could end up with a value that is a
> couple ULPs less than step -- essentially including the end point when you
> weren't supposed to.
>
> The truth is, making a floating point range() was simply a bad idea to
> begin with -- it's not the way to define a range of numbers in floating
> point. Whiuch is why the docs now say "When using a non-integer step, such
> as 0.1, the results will often not
> be consistent.  It is better to use ``linspace`` for these cases."
>
> Ben wants a way to define grids in a consistent way -- make sense. And
> yes, sometimes, the original source you are trying to match (like GDAL)
> provides a starting point and step. But with FP, that is simply
> problematic. If:
>
> start + step*num_steps != stop
>
> exactly in floating point, then you'll need to do the math one way or
> another to get what you want -- and I'm not sure anyone but the user knows
> what they want -- do you want step to be as exact as possible, or do you
> want stop to be as exact as possible?
>
> All that being said -- if arange() could be made a tiny bit more accurate
> with fma or other numerical technique, why not? it won't solve the
> problem, but if someone writes and tests the code (and it does not require
> compiler or hardware features that aren't supported everywhere numpy
> compiles), then sure. (Same for linspace, though I'm not sure it's possible)
>
> There is one other option: a new function (or option) that makes a grid
> from a specification of: start, step, num_points. If that is really a
> common use case (that is, you don't care exactly what the end-point is),
> then it might be handy to have it as a utility.
>
> We could also have an arange-like function that, rather than < stop, would
> do "close to" stop. Someone that understands FP better than I might be able
> to compute what the expected error might be, and find the closest end point
> within that error. But I think that's a bad specification -- (stop - start)
> / step may be nowhere near an integer -- then what is the function supposed
> to do??
>
>
> BTW: I kind of wish that linspace specified the number of steps, rather
> than the number of points, that is (num+points - 1) that would save a
> little bit of logical thinking. So if something new is made, let's keep
> that in mind.
>
>
>
>> 1. Comparison to calculations with decimal can be difficult as not all
>> simple decimal step sizes are exactly representable as
>>
> finite floating point numbers.
>>
>
> yeah, this is what I mean by inappropriate use of Decimal -- decimal is
> not inherently "more accurate" than fp -- is just can represent _decimal_
> numbers exactly, which we are all use to -- we want  1 / 10 to be exact,
> but dont mind that 1 / 3 isn't.
>
> Decimal also provided variable precision -- so it can be handy for that. I
> kinda wish Python had an arbitrary precision binary floating point built
> in...
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/033fc6b2/attachment.html>

From charlesr.harris at gmail.com  Thu Feb 22 14:10:15 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 22 Feb 2018 12:10:15 -0700
Subject: [Numpy-discussion] 1.15 release manager
Message-ID: <CAB6mnxKnikfJzK8bX58E6wAG0OHUNROx93TPc77O=v99Ajr5Yg@mail.gmail.com>

Hi All,

Would any of the other NumPy developers be interested in making the 1.15
release? It isn't difficult, and I don't mind doing it, but it would be a
good thing if more people became familiar with the process.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/28873c5b/attachment.html>

From ben.v.root at gmail.com  Thu Feb 22 14:33:28 2018
From: ben.v.root at gmail.com (Benjamin Root)
Date: Thu, 22 Feb 2018 14:33:28 -0500
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CALGmxEL5C27Zy4ihMOxRNYbx2Sgb=Gr_xfeBkEbsrftQ6rTLOw@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
 <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>
 <CALGmxEL5C27Zy4ihMOxRNYbx2Sgb=Gr_xfeBkEbsrftQ6rTLOw@mail.gmail.com>
Message-ID: <CANNq6FnytvhWYgSOELJXas+GYrMjtX_ZNqifH4GP6bJcr_gw1A@mail.gmail.com>

Sorry, I have been distracted with xarray improvements the past couple of
weeks.

Some thoughts on what has been discussed:

First, you are right...Decimal is not the right module for this. I think
instead I should use the 'fractions' module for loading grid spec
information from strings (command-line, configs, etc). The tricky part is
getting the yaml reader to use it instead of converting to a float under
the hood.

Second, what has been pointed out about the implementation of arange
actually helps to explain some oddities I have encountered. In some
situations, I have found that it was better for me to produce the reversed
sequence, and then reverse that array back and use it.

Third, it would be nice to do what we can to improve arange()'s results.
Would we be ok with a PR that uses fma() if it is available, but then falls
back on a regular multiply and add if it isn't available, or are we going
to need to implement it ourselves for consistency?

Lastly, there definitely needs to be a better tool for grid making. The
problem appears easy at first, but it is fraught with many pitfalls and
subtle issues. It is easy to say, "always use linspace()", but if the user
doesn't have the number of pixels, they will need to calculate that using
--- gasp! -- floating point numbers, which could result in the wrong
answer. Or maybe their first/last positions were determined by some other
calculation, and so the resulting grid does not have the expected spacing.
Another problem that I run into is starting from two different sized grids
and padding them both to be the same spec -- and getting that to match what
would come about if I had generated the grid from scratch.

Getting these things right is hard. I am not even certain that my existing
code for doing this even right. But, what I do know is that until we build
such a tool, users will continue to incorrectly use arange() and
linspace(), and waste time trying to re-invent the wheel badly, assuming
they even notice their mistakes in the first place! So, should such a tool
go into numpy, given how fundamental it is to generate a sequence of
floating point numbers, or should we try to put it into a package like
rasterio or xarray?

Cheers!
Ben Root



On Thu, Feb 22, 2018 at 2:02 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> @Ben: Have you found a solution to your problem? Are there thinks we could
> do in numpy to make it better?
>
> -CHB
>
>
> On Mon, Feb 12, 2018 at 9:33 AM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> I think it's all been said, but a few comments:
>>
>> On Sun, Feb 11, 2018 at 2:19 PM, Nils Becker <nilsc.becker at gmail.com>
>> wrote:
>>
>>> Generating equidistantly spaced grids is simply not always possible.
>>>
>>
>> exactly -- and linspace gives pretty much teh best possible result,
>> guaranteeing tha tthe start an end points are exact, and the spacing is
>> within an ULP or two (maybe we could make that within 1 ULP always, but not
>> sure that's worth it).
>>
>>
>>> The reason is that the absolute spacing of the possible floating point
>>> numbers depends on their magnitude [1].
>>>
>>
>> Also that the exact spacing may not be exactly representable in FP -- so
>> you have to have at least one space that's a bit off to get the end points
>> right (or have the endpoints not exact).
>>
>>
>>> If you - for some reason - want the same grid spacing everywhere you may
>>> choose an appropriate new spacing.
>>>
>>
>> well, yeah, but usually you are trying to fit to some other constraint.
>> I'm still confused as to where these couple of ULPs actually cause
>> problems, unless you are doing in appropriate FP comparisons elsewhere.
>>
>> Curiously, either by design or accident, arange() seems to do something
>>> similar as was mentioned by Eric. It creates a new grid spacing by adding
>>> and subtracting the starting point of the grid. This often has similar
>>> effect as adding and subtracting N*dx (e.g. if the grid is symmetric around
>>> 0.0). Consequently, arange() seems to trade keeping the grid spacing
>>> constant for a larger error in the grid size and consequently in the end
>>> point.
>>>
>>
>> interesting -- but it actually makes sense -- that is the definition of
>> arange(), borrowed from range(), which was designed for integers, and, in
>> fact, pretty much mirroered the classic C index for loop:
>>
>>
>> for (int i=0; i<N; i++) {
>> ...
>>
>>
>> or in python:
>>
>> i = start
>> while i < stop:
>>     i += step
>>
>> The problem here is that termination criteria -- i < stop -- that is the
>> definition of the function, and works just fine for integers (where it came
>> from), but with FP, even with no error accumulation, stop may not be
>> exactly representable, so you could end up with a value for your last item
>> that is about (stop-step), or you could end up with a value that is a
>> couple ULPs less than step -- essentially including the end point when you
>> weren't supposed to.
>>
>> The truth is, making a floating point range() was simply a bad idea to
>> begin with -- it's not the way to define a range of numbers in floating
>> point. Whiuch is why the docs now say "When using a non-integer step, such
>> as 0.1, the results will often not
>> be consistent.  It is better to use ``linspace`` for these cases."
>>
>> Ben wants a way to define grids in a consistent way -- make sense. And
>> yes, sometimes, the original source you are trying to match (like GDAL)
>> provides a starting point and step. But with FP, that is simply
>> problematic. If:
>>
>> start + step*num_steps != stop
>>
>> exactly in floating point, then you'll need to do the math one way or
>> another to get what you want -- and I'm not sure anyone but the user knows
>> what they want -- do you want step to be as exact as possible, or do you
>> want stop to be as exact as possible?
>>
>> All that being said -- if arange() could be made a tiny bit more
>> accurate with fma or other numerical technique, why not? it won't solve
>> the problem, but if someone writes and tests the code (and it does not
>> require compiler or hardware features that aren't supported everywhere
>> numpy compiles), then sure. (Same for linspace, though I'm not sure it's
>> possible)
>>
>> There is one other option: a new function (or option) that makes a grid
>> from a specification of: start, step, num_points. If that is really a
>> common use case (that is, you don't care exactly what the end-point is),
>> then it might be handy to have it as a utility.
>>
>> We could also have an arange-like function that, rather than < stop,
>> would do "close to" stop. Someone that understands FP better than I might
>> be able to compute what the expected error might be, and find the closest
>> end point within that error. But I think that's a bad specification --
>> (stop - start) / step may be nowhere near an integer -- then what is the
>> function supposed to do??
>>
>>
>> BTW: I kind of wish that linspace specified the number of steps, rather
>> than the number of points, that is (num+points - 1) that would save a
>> little bit of logical thinking. So if something new is made, let's keep
>> that in mind.
>>
>>
>>
>>> 1. Comparison to calculations with decimal can be difficult as not all
>>> simple decimal step sizes are exactly representable as
>>>
>> finite floating point numbers.
>>>
>>
>> yeah, this is what I mean by inappropriate use of Decimal -- decimal is
>> not inherently "more accurate" than fp -- is just can represent _decimal_
>> numbers exactly, which we are all use to -- we want  1 / 10 to be exact,
>> but dont mind that 1 / 3 isn't.
>>
>> Decimal also provided variable precision -- so it can be handy for that.
>> I kinda wish Python had an arbitrary precision binary floating point built
>> in...
>>
>> -CHB
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/7cdf3601/attachment-0001.html>

From sebastian at sipsolutions.net  Thu Feb 22 14:57:17 2018
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 22 Feb 2018 20:57:17 +0100
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CANNq6FnytvhWYgSOELJXas+GYrMjtX_ZNqifH4GP6bJcr_gw1A@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
 <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>
 <CALGmxEL5C27Zy4ihMOxRNYbx2Sgb=Gr_xfeBkEbsrftQ6rTLOw@mail.gmail.com>
 <CANNq6FnytvhWYgSOELJXas+GYrMjtX_ZNqifH4GP6bJcr_gw1A@mail.gmail.com>
Message-ID: <1519329437.30436.12.camel@sipsolutions.net>

On Thu, 2018-02-22 at 14:33 -0500, Benjamin Root wrote:
> Sorry, I have been distracted with xarray improvements the past
> couple of weeks.
> 
> Some thoughts on what has been discussed:
> 
> First, you are right...Decimal is not the right module for this. I
> think instead I should use the 'fractions' module for loading grid
> spec information from strings (command-line, configs, etc). The
> tricky part is getting the yaml reader to use it instead of
> converting to a float under the hood.
> 
> Second, what has been pointed out about the implementation of arange
> actually helps to explain some oddities I have encountered. In some
> situations, I have found that it was better for me to produce the
> reversed sequence, and then reverse that array back and use it.
> 
> Third, it would be nice to do what we can to improve arange()'s
> results. Would we be ok with a PR that uses fma() if it is available,
> but then falls back on a regular multiply and add if it isn't
> available, or are we going to need to implement it ourselves for
> consistency?
> 

I am not sure I like the idea:
  1. It sounds like it might break code
  2. It sounds *not* like a fix, but rather a
     "make it slightly less bad, but it is still awful"

Using fma inside linspace might make linspace a bit more exact
possible, and would be a good thing, though I am not sure we have a
policy yet for something that is only used sometimes, nor am I sure it
actually helps.

It also would be nice to add stable summation to numpy in general (in
whatever way), which maybe is half related but on nobody's specific
todo list.

> 
> Lastly, there definitely needs to be a better tool for grid making.
> The problem appears easy at first, but it is fraught with many
> pitfalls and subtle issues. It is easy to say, "always use
> linspace()", but if the user doesn't have the number of pixels, they
> will need to calculate that using --- gasp! -- floating point
> numbers, which could result in the wrong answer. Or maybe their
> first/last positions were determined by some other calculation, and
> so the resulting grid does not have the expected spacing. Another
> problem that I run into is starting from two different sized grids
> and padding them both to be the same spec -- and getting that to
> match what would come about if I had generated the grid from scratch.
> 

Maybe you are right, but right now I have no clue what that tool would
do :). If we should add it to numpy likely depends on what exactly it
does and how complex it is.
I once wanted to add a "step" argument to linspace, but didn't in the
end, largely because it basically enforced in a very convoluted way
that the step fit exactly to a number of steps (up to floating point
precision) and body was quite sure it was a good idea, since it would
just be useful for a little convenience when you do not want to
calculate the steps.

Best,

Sebastian


> 
> Getting these things right is hard. I am not even certain that my
> existing code for doing this even right. But, what I do know is that
> until we build such a tool, users will continue to incorrectly use
> arange() and linspace(), and waste time trying to re-invent the wheel
> badly, assuming they even notice their mistakes in the first place!
> So, should such a tool go into numpy, given how fundamental it is to
> generate a sequence of floating point numbers, or should we try to
> put it into a package like rasterio or xarray?
> 
> Cheers!
> Ben Root
> 
> 
> 
> On Thu, Feb 22, 2018 at 2:02 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> > @Ben: Have you found a solution to your problem? Are there thinks
> > we could do in numpy to make it better?
> > 
> > -CHB
> > 
> > 
> > On Mon, Feb 12, 2018 at 9:33 AM, Chris Barker <chris.barker at noaa.go
> > v> wrote:
> > > I think it's all been said, but a few comments:
> > > 
> > > On Sun, Feb 11, 2018 at 2:19 PM, Nils Becker <nilsc.becker at gmail.
> > > com> wrote:
> > > > Generating equidistantly spaced grids is simply not always
> > > > possible.
> > > > 
> > > 
> > > exactly -- and linspace gives pretty much teh best possible
> > > result, guaranteeing tha tthe start an end points are exact, and
> > > the spacing is within an ULP or two (maybe we could make that
> > > within 1 ULP always, but not sure that's worth it).
> > >  
> > > > The reason is that the absolute spacing of the possible
> > > > floating point numbers depends on their magnitude [1].
> > > > 
> > > 
> > > Also that the exact spacing may not be exactly representable in
> > > FP -- so you have to have at least one space that's a bit off to
> > > get the end points right (or have the endpoints not exact).
> > >  
> > > > If you - for some reason - want the same grid spacing
> > > > everywhere you may choose an appropriate new spacing.
> > > > 
> > > 
> > > well, yeah, but usually you are trying to fit to some other
> > > constraint. I'm still confused as to where these couple of ULPs
> > > actually cause problems, unless you are doing in appropriate FP
> > > comparisons elsewhere.
> > > 
> > > > Curiously, either by design or accident, arange() seems to do
> > > > something similar as was mentioned by Eric. It creates a new
> > > > grid spacing by adding and subtracting the starting point of
> > > > the grid. This often has similar effect as adding and
> > > > subtracting N*dx (e.g. if the grid is symmetric around 0.0).
> > > > Consequently, arange() seems to trade keeping the grid spacing
> > > > constant for a larger error in the grid size and consequently
> > > > in the end point.
> > > > 
> > > 
> > > interesting -- but it actually makes sense -- that is the
> > > definition of arange(), borrowed from range(), which was designed
> > > for integers, and, in fact, pretty much mirroered the classic C
> > > index for loop:
> > > 
> > > 
> > > for (int i=0; i<N; i++) {
> > > ...
> > > 
> > > 
> > > or in python:
> > > 
> > > i = start
> > > while i < stop:
> > >     i += step
> > > 
> > > The problem here is that termination criteria -- i < stop -- that
> > > is the definition of the function, and works just fine for
> > > integers (where it came from), but with FP, even with no error
> > > accumulation, stop may not be exactly representable, so you could
> > > end up with a value for your last item that is about (stop-step), 
> > > or you could end up with a value that is a couple ULPs less than
> > > step -- essentially including the end point when you weren't
> > > supposed to.
> > > 
> > > The truth is, making a floating point range() was simply a bad
> > > idea to begin with -- it's not the way to define a range of
> > > numbers in floating point. Whiuch is why the docs now say "When
> > > using a non-integer step, such as 0.1, the results will often not
> > > be consistent.  It is better to use ``linspace`` for these
> > > cases."
> > > 
> > > Ben wants a way to define grids in a consistent way -- make
> > > sense. And yes, sometimes, the original source you are trying to
> > > match (like GDAL) provides a starting point and step. But with
> > > FP, that is simply problematic. If:
> > > 
> > > start + step*num_steps != stop
> > > 
> > > exactly in floating point, then you'll need to do the math one
> > > way or another to get what you want -- and I'm not sure anyone
> > > but the user knows what they want -- do you want step to be as
> > > exact as possible, or do you want stop to be as exact as
> > > possible?
> > > 
> > > All that being said -- if arange() could be made a tiny bit more
> > > accurate with fma or other numerical technique, why not? it won't
> > > solve the problem, but if someone writes and tests the code (and
> > > it does not require compiler or hardware features that aren't
> > > supported everywhere numpy compiles), then sure. (Same for
> > > linspace, though I'm not sure it's possible)
> > > 
> > > There is one other option: a new function (or option) that makes
> > > a grid from a specification of: start, step, num_points. If that
> > > is really a common use case (that is, you don't care exactly what
> > > the end-point is), then it might be handy to have it as a
> > > utility.
> > > 
> > > We could also have an arange-like function that, rather than <
> > > stop, would do "close to" stop. Someone that understands FP
> > > better than I might be able to compute what the expected error
> > > might be, and find the closest end point within that error. But I
> > > think that's a bad specification -- (stop - start) / step may be
> > > nowhere near an integer -- then what is the function supposed to
> > > do??
> > > 
> > > 
> > > BTW: I kind of wish that linspace specified the number of steps,
> > > rather than the number of points, that is (num+points - 1) that
> > > would save a little bit of logical thinking. So if something new
> > > is made, let's keep that in mind.
> > > 
> > >  
> > > > 1. Comparison to calculations with decimal can be difficult as
> > > > not all simple decimal step sizes are exactly representable as 
> > > > 
> > > > finite floating point numbers.
> > > > 
> > > 
> > > yeah, this is what I mean by inappropriate use of Decimal --
> > > decimal is not inherently "more accurate" than fp -- is just can
> > > represent _decimal_ numbers exactly, which we are all use to --
> > > we want  1 / 10 to be exact, but dont mind that 1 / 3 isn't.
> > > 
> > > Decimal also provided variable precision -- so it can be handy
> > > for that. I kinda wish Python had an arbitrary precision binary
> > > floating point built in...
> > > 
> > > -CHB
> > > 
> > > -- 
> > > 
> > > Christopher Barker, Ph.D.
> > > Oceanographer
> > > 
> > > Emergency Response Division
> > > NOAA/NOS/OR&R            (206) 526-6959   voice
> > > 7600 Sand Point Way NE   (206) 526-6329   fax
> > > Seattle, WA  98115       (206) 526-6317   main reception
> > > 
> > > Chris.Barker at noaa.gov
> > 
> > 
> > 
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/3bfe80a1/attachment.sig>

From chris.barker at noaa.gov  Thu Feb 22 16:25:02 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 22 Feb 2018 13:25:02 -0800
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <1519329437.30436.12.camel@sipsolutions.net>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CAC_ACBW+qz411TpVQC=Ba-VV_5JFQbv4qic6B274YUNHHhR8uw@mail.gmail.com>
 <CALGmxEJ4V09syp-5SGqvbKm6CgA68eCMbQvCVv84yM8w1DTk1A@mail.gmail.com>
 <CALGmxEL5C27Zy4ihMOxRNYbx2Sgb=Gr_xfeBkEbsrftQ6rTLOw@mail.gmail.com>
 <CANNq6FnytvhWYgSOELJXas+GYrMjtX_ZNqifH4GP6bJcr_gw1A@mail.gmail.com>
 <1519329437.30436.12.camel@sipsolutions.net>
Message-ID: <CALGmxEJ-whOC=k3Zm2w+P+dwMQtM14Ojvva+msW0wnZQVF1CvA@mail.gmail.com>

On Thu, Feb 22, 2018 at 11:57 AM, Sebastian Berg <sebastian at sipsolutions.net
> wrote:

> > First, you are right...Decimal is not the right module for this. I
> > think instead I should use the 'fractions' module for loading grid
> > spec information from strings (command-line, configs, etc). The
> > tricky part is getting the yaml reader to use it instead of
> > converting to a float under the hood.
>

I'm not sure fractions is any better (Or necessary, anyway) in the end, you
need floats, so the inherent limitations of floats aren't the problem. In
your original use-case, you wanted a 32 bit float grid in the end, so doing
the calculations is 64 bit float and then downcasting is as good as you're
going to get, and easy and fast. And I suppose you could use 128 bit float
if you want to get to 64 bit in the end -- not as easy, and python itself
doesn't have it.

>  The
> tricky part is getting the yaml reader to use it instead of
> converting to a float under the hood.

64 bit floats support about 15 decimal digits -- are you string-based
sources providing more than that?? if not, then the 64 bit float version is
as good as it's going to get.

> Second, what has been pointed out about the implementation of arange
> > actually helps to explain some oddities I have encountered. In some
> > situations, I have found that it was better for me to produce the
> > reversed sequence, and then reverse that array back and use it.
>

interesting -- though I'd still say "don't use arange" is the "correct"
answer.


> > Third, it would be nice to do what we can to improve arange()'s
> > results. Would we be ok with a PR that uses fma() if it is available,
> > but then falls back on a regular multiply and add if it isn't
> > available, or are we going to need to implement it ourselves for
> > consistency?
>

I would think calling fma() if supported would be fine -- if there is an
easy macro to check if it's there. I don't know if numpy has a policy about
this sort of thing, but I'm pretty sure everywhere else, the final details
of computation fal back to the hardware/compiler/library (i.e. Intel used
extended precision fp, other platforms don't, etc) so I can't see that
having a slightly more accurate computation in arange on some platforms and
not others would cause a problem. If any of the tests are checking to that
level of accuracy, they should be fixed :-)

  2. It sounds *not* like a fix, but rather a
>      "make it slightly less bad, but it is still awful"
>

exactly -- better accuracy is a good thing, but it's not the source of the
problem here -- the source of the problem is inherent to FP, and/or poorly
specified goal. having arrange or linspace lose a couple ULPs fewer isn't
going to change anything.


> Using fma inside linspace might make linspace a bit more exact
> possible, and would be a good thing, though I am not sure we have a
> policy yet for something that is only used sometimes,


see above -- nor do I, but it seems like a fine idea to me.


> It also would be nice to add stable summation to numpy in general (in
> whatever way), which maybe is half related but on nobody's specific
> todo list.


I recall a conversation on this list a (long) while back about compensated
summation (Kahan summation) -- I guess nothign ever came of it?

> Lastly, there definitely needs to be a better tool for grid making.
> > The problem appears easy at first, but it is fraught with many
> > pitfalls and subtle issues. It is easy to say, "always use
> > linspace()", but if the user doesn't have the number of pixels, they
> > will need to calculate that using --- gasp! -- floating point
> > numbers, which could result in the wrong answer.


agreed -- this tends to be an inherently over-specified problem:

min_value
max_value
spacing
number_of_grid_spaces

That is four values, and only three independent ones.

arange() looks like it uses: min_value, max_value, spacing -- but it
doesn't really (see previous discussion) so not the right tool for anything.

linspace() uses:  min_value, max_value, (number_of_grid_spaces + 1), which
is about as good as you can get (except for that annoying 1).

But what if you are given min_value, spacing, number_of_grid_spaces?

Maybe we need a function for that?? (which I think would simply be:

np.arange(number_of_grid_spaces + 1) * spacing

Which is why we probably don't need a function :-) (note that that's only
error of one multiplication per grid point)

Or maybe a way to take all four values, and return a "best fit" grid. The
problem with that is that it's over specified, and and it may not be only
fp error that makes it not fit. What should a code do???

So Ben:

What is the problem you are trying to solve?  -- I'm still confused. What
information do you have to define the grid? Maybe all we need are docs for
how to best compute a grid with given specifications? And point to them in
the arange() and linspace() docstrings.

-CHB

I once wanted to add a "step" argument to linspace, but didn't in the
> end, largely because it basically enforced in a very convoluted way
> that the step fit exactly to a number of steps (up to floating point
> precision) and body was quite sure it was a good idea, since it would
> just be useful for a little convenience when you do not want to
> calculate the steps.
>

exactly!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/f146040c/attachment-0001.html>

From kevin.k.sheppard at gmail.com  Thu Feb 22 19:28:44 2018
From: kevin.k.sheppard at gmail.com (Kevin Sheppard)
Date: Fri, 23 Feb 2018 00:28:44 +0000
Subject: [Numpy-discussion] Prototype CorePRNG implementation feedback
 (Neal Becker)
In-Reply-To: <mailman.31.1519318803.23937.numpy-discussion@python.org>
References: <mailman.31.1519318803.23937.numpy-discussion@python.org>
Message-ID: <CAE8Ss2EX5WyP95KA5GwM7Jn2JKfNhinkynWEHfq_WbADNXbDMg@mail.gmail.com>

>
>
> What is the syntax to construct an initialized generator?
> RandomGenerator(Xoroshiro128(my_seed))?
>
>
Not 100% certain on this.  There was talk in the earlier thread that seed
should be killed, although I wasn't clear what mathod would be
preferrable to get easily reproducible random numbers.

Another option would be

RandomGenerator(Xoroshiro128, seed=my_seed)

Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180223/c993ab9c/attachment.html>

From robert.kern at gmail.com  Thu Feb 22 20:09:47 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 22 Feb 2018 17:09:47 -0800
Subject: [Numpy-discussion] Prototype CorePRNG implementation feedback
 (Neal Becker)
In-Reply-To: <CAE8Ss2EX5WyP95KA5GwM7Jn2JKfNhinkynWEHfq_WbADNXbDMg@mail.gmail.com>
References: <mailman.31.1519318803.23937.numpy-discussion@python.org>
 <CAE8Ss2EX5WyP95KA5GwM7Jn2JKfNhinkynWEHfq_WbADNXbDMg@mail.gmail.com>
Message-ID: <CAF6FJisy0ZpN7by5rCmpdrVNP+zMHYT_KAKM3DR90GnaNY2VsA@mail.gmail.com>

On Feb 22, 2018 16:30, "Kevin Sheppard" <kevin.k.sheppard at gmail.com> wrote:


> What is the syntax to construct an initialized generator?
> RandomGenerator(Xoroshiro128(my_seed))?
>
>
Not 100% certain on this.  There was talk in the earlier thread that seed
should be killed,


No, just the np.random.seed() function alias for the hidden global PRNG.
Core PRNGs should be initializable with a seed number like this, and
instances of these objects should also have a seed() method to reset the
state.

I think all of the core PRNGs MUST be seedable by being given a uint32
integer, at minimum. They MAY be seedable by other inputs (uint64, Python
long, array of ints) depending on the algorithm.

although I wasn't clear what mathod would be
preferrable to get easily reproducible random numbers.

Another option would be

RandomGenerator(Xoroshiro128, seed=my_seed)


I recommend the first option. With things like settable streams, the
diversity of initialization options between algorithms is best implemented
by a diversity of initializers than trying to force everything into one
complicated initializer.

While I think the first option is the best way to implement things, we can
make some syntactic sugar to make things nicer. For example,
`Xoroshiro128(my_seed).generator` where `.generator` is implemented as a
property that returns `RandomGenerator(self)`.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180222/2694d8cf/attachment.html>

From chris.barker at noaa.gov  Fri Feb 23 13:01:05 2018
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 23 Feb 2018 10:01:05 -0800
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
Message-ID: <CALGmxEKZXYRnPtqGnchUje8Nq5m72-dg3sqGWoe9bfxFkNVUPQ@mail.gmail.com>

On Fri, Feb 9, 2018 at 1:16 PM, Matthew Harrigan <harrigan.matthew at gmail.com
> wrote:

> I apologize if I'm missing something basic, but why are floats being
> accumulated in the first place?  Can't arange and linspace operations with
> floats be done internally similar to `start + np.arange(num_steps) *
> step_size`?  I.e. always accumulate (really increment) integers to limit
> errors.
>

I haven't looked at the arange() code, but linspace does does not
accumulate floats -- which is why it's already almost as good as it can be.
As regards to a fused-multiply-add, it does have to do a single
multiply_add operation for each value (as per your example code), so we may
be able to save a ULP there.

The problem with arange() is that the definition is poorly specified:

start + (step_num * step) while value < stop.

Even without fp issues, it's weird if (stop - start) / step is not an
integer. -- the "final" step will not be the same as the rest.

Say you want a "grid" with fully integer values. if the step is just right,
all is easy:

In [*72*]: np.arange(0, 11, 2)

Out[*72*]: array([ 0,  2,  4,  6,  8, 10])

(this is assuming you want 10 as the end point.

but then:

In [*73*]: np.arange(0, 11, 3)

Out[*73*]: array([0, 3, 6, 9])

but I wanted 10 as an end point. so:

In [*74*]: np.arange(0, 13, 3)

Out[*74*]: array([ 0,  3,  6,  9, 12])

hmm, that's not right either. Of course it's not -- you can't get 10 as an
end point, 'cause it's not a multiple of the step. With integers, you CAN
require that the end point be a multiple of the step, but with fp, you
can't required that it be EXACTLY a multiple, because either the end point
or the step may not be exactly representable, even if you do the math with
no loss of precision. And now you are stuck with the user figuring out for
themselves whether the closest fp representation of the end point is
slightly larger or smaller than the real value, so the < check will work.
NOT good.

This is why arange is simply not the tool to use.

Making a grid, you usually want to specify the end points and the number of
steps which is almost what linspace does. Or, _maybe_ you want to specify
the step and the number of steps, and accept that the end point may not be
exactly what you "expect". There is no built-in function for this in numpy.
maybe there should be, but it's pretty easy to write, as you show above.

Anyone that understands FP better than I do:

In the above code, you are multiplying the step by an integer -- is there
any precision loss when you do that??

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180223/ad863370/attachment.html>

From nilsc.becker at gmail.com  Sun Feb 25 06:34:48 2018
From: nilsc.becker at gmail.com (Nils Becker)
Date: Sun, 25 Feb 2018 12:34:48 +0100
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CALGmxEKZXYRnPtqGnchUje8Nq5m72-dg3sqGWoe9bfxFkNVUPQ@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
 <CALGmxEKZXYRnPtqGnchUje8Nq5m72-dg3sqGWoe9bfxFkNVUPQ@mail.gmail.com>
Message-ID: <CAC_ACBXxS=O2oA4NAtU66Vy7vXMnvL1YRu6dCG6KwQhL+P6+Rw@mail.gmail.com>

Hey,


> Anyone that understands FP better than I do:
>
> In the above code, you are multiplying the step by an integer -- is there
> any precision loss when you do that??
>
>
Elementary operations (add, sub, mul, div) are demanded to be correctly
rounded (cr) by IEEE, i.e. accurate within +/- 0.5 ulp. Consequently, a cr
multiplication followed by a cr addition will be accurate within +/-1 ulp.
This is also true if the first multiplicand is an integer.
Using FMA will reduce this to +/- 0.5 ulp. This increase in accuracy of the
grid calculation should not be relevant - but it also does not hurt.

Still I would suggest adding the FMA operation to numpy, e.g. np.fma(a, b,
c). There are several places in numpy that could benefit from the increased
accuracy, e.g. evaluation of polynomials using Horner's method. In cases
like this due to iteration and consequent error propagation the accuracy
benefit of using FMA can be far larger. There may also be a performance
benefit on platforms that implement FMA in hardware (although I am not sure
about that).

Cheers
Nils
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180225/4f938d4a/attachment.html>

From arnaldorusso at gmail.com  Tue Feb 27 09:26:27 2018
From: arnaldorusso at gmail.com (Arnaldo Russo)
Date: Tue, 27 Feb 2018 11:26:27 -0300
Subject: [Numpy-discussion] improving arange()? introducing fma()?
In-Reply-To: <CAC_ACBXxS=O2oA4NAtU66Vy7vXMnvL1YRu6dCG6KwQhL+P6+Rw@mail.gmail.com>
References: <CANNq6F=wRp1DqUzfZjZCJo2nD9uYBHaM3+TJDdW9juFnk8HTEQ@mail.gmail.com>
 <CABL7CQiua3RNx+aS7Ao=9w0nDrHRmKj853iWOWF7HDVn_VK5+Q@mail.gmail.com>
 <CALGmxE+ciz7opdCtqYe2kZVhj8Y1N0o0ZTvt-ZKDEBaMdvFqmQ@mail.gmail.com>
 <CANNq6FntDKvQO7nt-T2Bb0dCet5wH8KTzE47M-sGS_MnjQoTYg@mail.gmail.com>
 <CAOfRF=gHETaOb6TwY5UG_PWTu4V7yoP7hxSAux0XSM+xUiRPJQ@mail.gmail.com>
 <CALGmxEKZXYRnPtqGnchUje8Nq5m72-dg3sqGWoe9bfxFkNVUPQ@mail.gmail.com>
 <CAC_ACBXxS=O2oA4NAtU66Vy7vXMnvL1YRu6dCG6KwQhL+P6+Rw@mail.gmail.com>
Message-ID: <CAKUv9OMzSdXXK6RgqhX+nOSi=pdS4x2SqdVShEj8+SZKC0=d=Q@mail.gmail.com>

Hi there,

Is this the implementation of fma?
https://github.com/nschloe/pyfma

Cheers,
Arnaldo

       .
       |\
     _/]_\_
~~~~~"~~~~~^~~~~~~~~~~~~~~~~~~~~~

Arnaldo D'Amaral Pereira Granja Russo
c i c l o t u x . o r g

2018-02-25 8:34 GMT-03:00 Nils Becker <nilsc.becker at gmail.com>:

> Hey,
>
>
>> Anyone that understands FP better than I do:
>>
>> In the above code, you are multiplying the step by an integer -- is there
>> any precision loss when you do that??
>>
>>
> Elementary operations (add, sub, mul, div) are demanded to be correctly
> rounded (cr) by IEEE, i.e. accurate within +/- 0.5 ulp. Consequently, a cr
> multiplication followed by a cr addition will be accurate within +/-1 ulp.
> This is also true if the first multiplicand is an integer.
> Using FMA will reduce this to +/- 0.5 ulp. This increase in accuracy of
> the grid calculation should not be relevant - but it also does not hurt.
>
> Still I would suggest adding the FMA operation to numpy, e.g. np.fma(a, b,
> c). There are several places in numpy that could benefit from the increased
> accuracy, e.g. evaluation of polynomials using Horner's method. In cases
> like this due to iteration and consequent error propagation the accuracy
> benefit of using FMA can be far larger. There may also be a performance
> benefit on platforms that implement FMA in hardware (although I am not sure
> about that).
>
> Cheers
> Nils
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180227/dddb93d5/attachment.html>

From xoviat at gmail.com  Tue Feb 27 17:11:33 2018
From: xoviat at gmail.com (xoviat)
Date: Tue, 27 Feb 2018 16:11:33 -0600
Subject: [Numpy-discussion] PyPy nditer context managers
Message-ID: <5a95d794.cdebca0a.b0c38.0a75@mx.google.com>

The Pull Request related nditer has been dormant for a while, and there seems to be no consensus on moving forward at this point. Charles, are you opposed to requiring the .close() call on nditer and/or adding context managers? If so, then we might as well let mattip know that the Pull Request is not moving forward.

If you aren?t opposed, perhaps we should clarify what the expectations for the Pull Request are and a path to having it merged.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180227/1932575a/attachment.html>