From othalan at othalan.net  Mon May  2 05:51:37 2016
From: othalan at othalan.net (David Morris)
Date: Mon, 2 May 2016 16:51:37 +0700
Subject: [Numpy-discussion] Unable to pickle numpy array on iOS
Message-ID: <CAHk-ryBbYRzsqNn+Ljq_e8qFZzvj2A_4RZYjYdLrUT8RXeTHNA@mail.gmail.com>

I have an application running on iOS where I pickle a numpy array in order
to save it for later use.  However, I receive the following error:

pickle.dumps(arr)
...
_pickle.PicklingError: Can't pickle <built-in function _reconstruct>:
import of module 'multiarray' failed

On a desktop system (OSX), there is no problem dumping the array.

I am using NumPy v1.9.3

Any ideas on why this might be happening?

Thank you,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160502/bb329497/attachment.html>

From honi at brandeis.edu  Tue May  3 11:43:18 2016
From: honi at brandeis.edu (Honi Sanders)
Date: Tue, 3 May 2016 11:43:18 -0400
Subject: [Numpy-discussion] Cross-correlation PR stuck in limbo
Message-ID: <8D8A0029-A28D-4E4E-A0D5-437FDC85AA73@brandeis.edu>

Hello all,
I have completed a pull request to add a ?maxlag? functionality to numpy.correlate.  See here: https://github.com/numpy/numpy/pull/5978 <https://github.com/numpy/numpy/pull/5978>.  This pull request has passed all tests and has been ready to be merged for around six months.  Several people have commented requesting for it to be included on stackoverflow, the listserve, and github.  Can someone please let me know what needs to be done or can it be merged?

Here is some background:
What was troubling me is that numpy.correlate does not have a maxlag feature. This means that even if I only want to see correlations between two time series with lags between -100 and +100 ms, for example, it will still calculate the correlation for every lag between -20000 and +20000 ms (which is the length of the time series). This (theoretically) gives a 200x performance hit! 

I have introduced this question as a numpy issue <https://github.com/numpy/numpy/issues/5954>, a scipy issue <https://github.com/scipy/scipy/issues/4940> and on the scipy-dev list <http://mail.scipy.org/pipermail/scipy-dev/2015-June/020757.html>. It seems the best place to start is with numpy.correlate, so that is what I am requesting. 

Previous discussion of this functionality can be found at another discussion on numpy correlate (and convolution) <http://numpy-discussion.10968.n7.nabble.com/another-discussion-on-numpy-correlate-and-convolution-td32925.html>. Other issues related to correlate functions include ENH: Fold fftconvolve into convolve/correlate functions as a parameter #2651 <https://github.com/scipy/scipy/issues/2651>, Use FFT in np.correlate/convolve? (Trac #1260) #1858 <https://github.com/numpy/numpy/issues/1858>, and normalized cross-correlation (Trac #1714) #2310 <https://github.com/numpy/numpy/issues/2310>.



The new implementation allows new types of the ?mode? argument, to include an int value, which defines the maximum lag for which cross-correlation should be calculated, or a tuple, which defines the minlag, maxlag, and lagstep to be used in the same format as the arguments to numpy.arange.


Please let me know what should be done to move this pull request forward.

Honi



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160503/efd7e0b5/attachment.html>

From pierre.haessig at crans.org  Wed May  4 08:07:43 2016
From: pierre.haessig at crans.org (Pierre Haessig)
Date: Wed, 4 May 2016 14:07:43 +0200
Subject: [Numpy-discussion] Cross-correlation PR stuck in limbo
In-Reply-To: <8D8A0029-A28D-4E4E-A0D5-437FDC85AA73@brandeis.edu>
References: <8D8A0029-A28D-4E4E-A0D5-437FDC85AA73@brandeis.edu>
Message-ID: <5729E60F.1040102@crans.org>

Hi,

I don't know how to push the PR forward, but all I can say is that this
maxlag feature would be a major improvement for using Numpy in time
series analysis! Immediate benefits downstream for Matplotlib and
statsmodel.

Thanks Honi for having taken the time to implement this!

best,
Pierre



From jeffreback at gmail.com  Wed May  4 20:30:00 2016
From: jeffreback at gmail.com (Jeff Reback)
Date: Wed, 4 May 2016 20:30:00 -0400
Subject: [Numpy-discussion] ANN: v0.18.1 pandas Released
Message-ID: <CAHMnJKgkd-y2m-ejqAiFZg=-xXWspTTv4S37cLP9QYDP+PUqvA@mail.gmail.com>

This is a minor bug-fix release from 0.18.0 and includes a large number of
bug fixes along several new features, enhancements, and performance
improvements. We recommend that all users upgrade to this version.

This was a release of 6 weeks with 210 commits by 60 authors  encompassing
142 issues and 164 pull-requests.


*What is it:*

*pandas* is a Python package providing fast, flexible, and expressive
data structures
designed to make working with ?relational? or ?labeled? data both easy and
intuitive. It aims to be the fundamental high-level building block
for doing practical, real world data analysis in Python. Additionally, it
has the broader goal of becoming the most powerful and flexible open source
data analysis / manipulation tool available in any language.


*Highlights*:

   - .groupby(...) has been enhanced to provide convenient syntax when working
   with .rolling(..), .expanding(..) and .resample(..) per group, see here
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html#whatsnew-0181-deferred-ops>
   - pd.to_datetime() has gained the ability to assemble dates from a
   DataFrame, see here
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html#whatsnew-0181-enhancements-assembling>
   - Method chaining improvements, see here
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html#whatsnew-0181-enhancements-method-chain>
   - Custom business hour offset, see here
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html#whatsnew-0181-enhancements-custombusinesshour>
   - Many bug fixes in the handling of sparse, see here
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html#whatsnew-0181-sparse>
   - Expanded the Tutorials section
   <http://pandas.pydata.org/pandas-docs/version/0.18.1/tutorials.html#tutorial-modern>
with
   a feature on modern pandas, courtesy of @TomAugsburger
   <https://twitter.com/TomAugspurger>.

See the Whatsnew
<http://pandas.pydata.org/pandas-docs/version/0.18.1/whatsnew.html> for
much more information, and the full Documentation
<http://pandas.pydata.org/pandas-docs/stable/> link.


*How to get it:*

Source tarballs, windows wheels, and macosx wheels are available on PyPI
<https://pypi.python.org/pypi/pandas>. Windows wheels are courtesy of Christoph
Gohlke, and are built on Numpy 1.10. Macosx wheels are courtesy of Matthew
Brett.

Installation via conda is: conda install pandas
currently its available via the conda-forge channel: conda install pandas
-c conda-forge
It will be available on the main channel shortly.

Please report any issues on our issue tracker
<https://github.com/pydata/pandas/issues>:

Jeff Reback


*Thanks to all of the contributors*


* - Andrew Fiore-Gartland- Bastiaan- Beno?t Vinot- Brandon Rhodes- DaCoEx-
Drew Fustin- Ernesto Freitas- Filip Ter- Gregory Livschitz- G?bor Lipt?k-
Hassan Kibirige- Iblis Lin- Israel Saeta P?rez- Jason Wolosonovich- Jeff
Reback- Joe Jevnik- Joris Van den Bossche- Joshua Storck- Ka Wo Chen- Kerby
Shedden- Kieran O'Mahony- Leif Walsh- Mahmoud Lababidi- Maoyuan Liu- Mark
Roth- Matt Wittmann- MaxU- Maximilian Roos- Michael Droettboom- Nick
Eubank- Nicolas Bonnotte- OXPHOS- Pauli Virtanen- Peter Waller- Pietro
Battiston- Prabhjot Singh- Robin Wilson- Roger Thomas- Sebastian Bank-
Stephen Hoover- Tim Hopper- Tom Augspurger- WANG Aiyong- Wes Turner-
Winand- Xbar- Yan Facai- adneu- ajenkins-cargometrics- behzad nouri-
chinskiy- gfyoung- jeps-journal- jonaslb- kotrfa- nileracecrew-
onesandzeroes- rs2- sinhrks- tsdlovell*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160504/d238e17c/attachment.html>

From oysteijo at gmail.com  Thu May  5 05:38:51 2016
From: oysteijo at gmail.com (=?UTF-8?Q?=C3=98ystein_Sch=C3=B8nning=2DJohansen?=)
Date: Thu, 5 May 2016 11:38:51 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned data.
Message-ID: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>

Hi!

I've written a little code of numpy code that does a neural network
feedforward calculation:

    def feedforward(self,x):
        for activation, w, b in zip( self.activations, self.weights,
self.biases ):
            x = activation( np.dot(w, x) + b)

This works fine when my activation functions are in Python, however I've
wrapped the activation functions from a C implementation that requires the
array to be memory aligned. (due to simd instructions in the C
implementation.) So I need the operation np.dot( w, x) + b to return a
ndarray where the data pointer is aligned. How can I do that? Is it
possible at all?

(BTW: the function works  correctly about 20% of the time I run it, and
else it segfaults on the simd instruction in the the C function)

Thanks,
-?ystein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160505/383ba64b/attachment.html>

From faltet at gmail.com  Thu May  5 07:55:36 2016
From: faltet at gmail.com (Francesc Alted)
Date: Thu, 5 May 2016 13:55:36 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
In-Reply-To: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
Message-ID: <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>

2016-05-05 11:38 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com>:

> Hi!
>
> I've written a little code of numpy code that does a neural network
> feedforward calculation:
>
>     def feedforward(self,x):
>         for activation, w, b in zip( self.activations, self.weights,
> self.biases ):
>             x = activation( np.dot(w, x) + b)
>
> This works fine when my activation functions are in Python, however I've
> wrapped the activation functions from a C implementation that requires the
> array to be memory aligned. (due to simd instructions in the C
> implementation.) So I need the operation np.dot( w, x) + b to return a
> ndarray where the data pointer is aligned. How can I do that? Is it
> possible at all?
>

Yes.  np.dot() does accept an `out` parameter where you can pass your
aligned array.  The way for testing if numpy is returning you an aligned
array is easy:

In [15]: x = np.arange(6).reshape(2,3)

In [16]: x.ctypes.data % 16
Out[16]: 0

but:

In [17]: x.ctypes.data % 32
Out[17]: 16

so, in this case NumPy returned a 16-byte aligned array which should be
enough for 128 bit SIMD (SSE family).  This kind of alignment is pretty
common in modern computers.  If you need 256 bit (32-byte) alignment then
you will need to build your container manually.  See here for an example:
http://stackoverflow.com/questions/9895787/memory-alignment-for-fast-fft-in-python-using-shared-arrrays

Francesc


>
> (BTW: the function works  correctly about 20% of the time I run it, and
> else it segfaults on the simd instruction in the the C function)
>
> Thanks,
> -?ystein
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160505/9529fbff/attachment.html>

From oysteijo at gmail.com  Thu May  5 16:10:32 2016
From: oysteijo at gmail.com (=?UTF-8?Q?=C3=98ystein_Sch=C3=B8nning=2DJohansen?=)
Date: Thu, 5 May 2016 22:10:32 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
In-Reply-To: <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
 <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
Message-ID: <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>

Thanks for your answer, Francesc. Knowing that there is no numpy solution
saves the work of searching for this. I've not tried the solution described
at SO, but it looks like a real performance killer. I'll rather try to
override malloc with glibs malloc_hooks or LD_PRELOAD tricks. Do you think
that will do it? I'll try it and report back.

Thanks,
-?ystein

On Thu, May 5, 2016 at 1:55 PM, Francesc Alted <faltet at gmail.com> wrote:

> 2016-05-05 11:38 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com>
> :
>
>> Hi!
>>
>> I've written a little code of numpy code that does a neural network
>> feedforward calculation:
>>
>>     def feedforward(self,x):
>>         for activation, w, b in zip( self.activations, self.weights,
>> self.biases ):
>>             x = activation( np.dot(w, x) + b)
>>
>> This works fine when my activation functions are in Python, however I've
>> wrapped the activation functions from a C implementation that requires the
>> array to be memory aligned. (due to simd instructions in the C
>> implementation.) So I need the operation np.dot( w, x) + b to return a
>> ndarray where the data pointer is aligned. How can I do that? Is it
>> possible at all?
>>
>
> Yes.  np.dot() does accept an `out` parameter where you can pass your
> aligned array.  The way for testing if numpy is returning you an aligned
> array is easy:
>
> In [15]: x = np.arange(6).reshape(2,3)
>
> In [16]: x.ctypes.data % 16
> Out[16]: 0
>
> but:
>
> In [17]: x.ctypes.data % 32
> Out[17]: 16
>
> so, in this case NumPy returned a 16-byte aligned array which should be
> enough for 128 bit SIMD (SSE family).  This kind of alignment is pretty
> common in modern computers.  If you need 256 bit (32-byte) alignment then
> you will need to build your container manually.  See here for an example:
> http://stackoverflow.com/questions/9895787/memory-alignment-for-fast-fft-in-python-using-shared-arrrays
>
> Francesc
>
>
>>
>> (BTW: the function works  correctly about 20% of the time I run it, and
>> else it segfaults on the simd instruction in the the C function)
>>
>> Thanks,
>> -?ystein
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
> Francesc Alted
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160505/b3161226/attachment.html>

From charlesr.harris at gmail.com  Thu May  5 16:32:46 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 5 May 2016 14:32:46 -0600
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
In-Reply-To: <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
 <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
 <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
Message-ID: <CAB6mnxJPCgmsagcANW_aKXeon6NTJEeT8HO2AEKo21Bnyx+G9w@mail.gmail.com>

On Thu, May 5, 2016 at 2:10 PM, ?ystein Sch?nning-Johansen <
oysteijo at gmail.com> wrote:

> Thanks for your answer, Francesc. Knowing that there is no numpy solution
> saves the work of searching for this. I've not tried the solution described
> at SO, but it looks like a real performance killer. I'll rather try to
> override malloc with glibs malloc_hooks or LD_PRELOAD tricks. Do you think
> that will do it? I'll try it and report back.
>
> Thanks,
> -?ystein
>

Might take a look at how numpy handles this in
`numpy/core/src/umath/simd.inc.src`.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160505/fab48303/attachment.html>

From faltet at gmail.com  Fri May  6 09:01:32 2016
From: faltet at gmail.com (Francesc Alted)
Date: Fri, 6 May 2016 15:01:32 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
In-Reply-To: <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
 <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
 <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
Message-ID: <CAFrp1vpJO+OWa=jRY0dgUD_yiMdw6am42uwxiDxcq4KWcQ68zA@mail.gmail.com>

2016-05-05 22:10 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com>:

> Thanks for your answer, Francesc. Knowing that there is no numpy solution
> saves the work of searching for this. I've not tried the solution described
> at SO, but it looks like a real performance killer. I'll rather try to
> override malloc with glibs malloc_hooks or LD_PRELOAD tricks. Do you think
> that will do it? I'll try it and report back.
>

I don't think you need that much weaponry.  Just create an array with some
spare space for alignment.  Realize that you want a 64-byte aligned double
precision array.  With that, create your desired array + 64 additional
bytes (8 doubles):

In [92]: a = np.zeros(int(1e6) + 8)

In [93]: a.ctypes.data % 64
Out[93]: 16

and compute the elements to shift this:

In [94]: shift = (64 / a.itemsize) - (a.ctypes.data % 64) / a.itemsize

In [95]: shift
Out[95]: 6

now, create a view with the required elements less:

In [98]: b = a[shift:-((64 / a.itemsize)-shift)]

In [99]: len(b)
Out[99]: 1000000

In [100]: b.ctypes.data % 64
Out[100]: 0

and voila, b is now aligned to 64 bytes.  As the view is a copy-free
operation, this is fast, and you only wasted 64 bytes.  Pretty cheap indeed.

Francesc


>
> Thanks,
> -?ystein
>
> On Thu, May 5, 2016 at 1:55 PM, Francesc Alted <faltet at gmail.com> wrote:
>
>> 2016-05-05 11:38 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com
>> >:
>>
>>> Hi!
>>>
>>> I've written a little code of numpy code that does a neural network
>>> feedforward calculation:
>>>
>>>     def feedforward(self,x):
>>>         for activation, w, b in zip( self.activations, self.weights,
>>> self.biases ):
>>>             x = activation( np.dot(w, x) + b)
>>>
>>> This works fine when my activation functions are in Python, however I've
>>> wrapped the activation functions from a C implementation that requires the
>>> array to be memory aligned. (due to simd instructions in the C
>>> implementation.) So I need the operation np.dot( w, x) + b to return a
>>> ndarray where the data pointer is aligned. How can I do that? Is it
>>> possible at all?
>>>
>>
>> Yes.  np.dot() does accept an `out` parameter where you can pass your
>> aligned array.  The way for testing if numpy is returning you an aligned
>> array is easy:
>>
>> In [15]: x = np.arange(6).reshape(2,3)
>>
>> In [16]: x.ctypes.data % 16
>> Out[16]: 0
>>
>> but:
>>
>> In [17]: x.ctypes.data % 32
>> Out[17]: 16
>>
>> so, in this case NumPy returned a 16-byte aligned array which should be
>> enough for 128 bit SIMD (SSE family).  This kind of alignment is pretty
>> common in modern computers.  If you need 256 bit (32-byte) alignment then
>> you will need to build your container manually.  See here for an example:
>> http://stackoverflow.com/questions/9895787/memory-alignment-for-fast-fft-in-python-using-shared-arrrays
>>
>> Francesc
>>
>>
>>>
>>> (BTW: the function works  correctly about 20% of the time I run it, and
>>> else it segfaults on the simd instruction in the the C function)
>>>
>>> Thanks,
>>> -?ystein
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>>
>> --
>> Francesc Alted
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Francesc Alted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160506/8c8bd529/attachment.html>

From jtaylor.debian at googlemail.com  Fri May  6 16:22:38 2016
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 6 May 2016 22:22:38 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
In-Reply-To: <CAB6mnxJPCgmsagcANW_aKXeon6NTJEeT8HO2AEKo21Bnyx+G9w@mail.gmail.com>
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
 <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
 <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
 <CAB6mnxJPCgmsagcANW_aKXeon6NTJEeT8HO2AEKo21Bnyx+G9w@mail.gmail.com>
Message-ID: <572CFD0E.1080301@googlemail.com>

note that anything larger than 16 bytes alignment is unnecessary for 
simd purposes on current hardware (>= haswell). 16 byte is default 
malloc alignment on amd64.
And even on older ones (sandy bridge) the penalty is pretty minor.

On 05.05.2016 22:32, Charles R Harris wrote:
>
>
> On Thu, May 5, 2016 at 2:10 PM, ?ystein Sch?nning-Johansen
> <oysteijo at gmail.com <mailto:oysteijo at gmail.com>> wrote:
>
>     Thanks for your answer, Francesc. Knowing that there is no numpy
>     solution saves the work of searching for this. I've not tried the
>     solution described at SO, but it looks like a real performance
>     killer. I'll rather try to override malloc with glibs malloc_hooks
>     or LD_PRELOAD tricks. Do you think that will do it? I'll try it and
>     report back.
>
>     Thanks,
>     -?ystein
>
>
> Might take a look at how numpy handles this in
> `numpy/core/src/umath/simd.inc.src`.
>
> <snip>
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From solipsis at pitrou.net  Sat May  7 07:02:14 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 7 May 2016 13:02:14 +0200
Subject: [Numpy-discussion] Calling C code that assumes SIMD aligned
	data.
References: <CAOzpFnSTN183o7FZzBaq52c+WzCNGWakYwcxHByS8K3=xh9TBw@mail.gmail.com>
 <CAFrp1vpKXgHNtxkUL99VBcROP4g9b4ojTp0WUgmJq-r03YPPVQ@mail.gmail.com>
 <CAOzpFnR91BD+2jbuv53V5Mfv2mHN6aZuSky=UbmhQSB+8fDrBg@mail.gmail.com>
 <CAFrp1vpJO+OWa=jRY0dgUD_yiMdw6am42uwxiDxcq4KWcQ68zA@mail.gmail.com>
Message-ID: <20160507130214.3ccf5af8@fsol>


Here's an obligatory plug for the two following PRs:
https://github.com/numpy/numpy/pull/5457
https://github.com/numpy/numpy/pull/5470

Regards

Antoine.


On Fri, 6 May 2016 15:01:32 +0200
Francesc Alted <faltet at gmail.com> wrote:
> 2016-05-05 22:10 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com>:
> 
> > Thanks for your answer, Francesc. Knowing that there is no numpy solution
> > saves the work of searching for this. I've not tried the solution described
> > at SO, but it looks like a real performance killer. I'll rather try to
> > override malloc with glibs malloc_hooks or LD_PRELOAD tricks. Do you think
> > that will do it? I'll try it and report back.
> >
> 
> I don't think you need that much weaponry.  Just create an array with some
> spare space for alignment.  Realize that you want a 64-byte aligned double
> precision array.  With that, create your desired array + 64 additional
> bytes (8 doubles):
> 
> In [92]: a = np.zeros(int(1e6) + 8)
> 
> In [93]: a.ctypes.data % 64
> Out[93]: 16
> 
> and compute the elements to shift this:
> 
> In [94]: shift = (64 / a.itemsize) - (a.ctypes.data % 64) / a.itemsize
> 
> In [95]: shift
> Out[95]: 6
> 
> now, create a view with the required elements less:
> 
> In [98]: b = a[shift:-((64 / a.itemsize)-shift)]
> 
> In [99]: len(b)
> Out[99]: 1000000
> 
> In [100]: b.ctypes.data % 64
> Out[100]: 0
> 
> and voila, b is now aligned to 64 bytes.  As the view is a copy-free
> operation, this is fast, and you only wasted 64 bytes.  Pretty cheap indeed.
> 
> Francesc
> 
> 
> >
> > Thanks,
> > -?ystein
> >
> > On Thu, May 5, 2016 at 1:55 PM, Francesc Alted <faltet at gmail.com> wrote:
> >
> >> 2016-05-05 11:38 GMT+02:00 ?ystein Sch?nning-Johansen <oysteijo at gmail.com
> >> >:
> >>
> >>> Hi!
> >>>
> >>> I've written a little code of numpy code that does a neural network
> >>> feedforward calculation:
> >>>
> >>>     def feedforward(self,x):
> >>>         for activation, w, b in zip( self.activations, self.weights,
> >>> self.biases ):
> >>>             x = activation( np.dot(w, x) + b)
> >>>
> >>> This works fine when my activation functions are in Python, however I've
> >>> wrapped the activation functions from a C implementation that requires the
> >>> array to be memory aligned. (due to simd instructions in the C
> >>> implementation.) So I need the operation np.dot( w, x) + b to return a
> >>> ndarray where the data pointer is aligned. How can I do that? Is it
> >>> possible at all?
> >>>
> >>
> >> Yes.  np.dot() does accept an `out` parameter where you can pass your
> >> aligned array.  The way for testing if numpy is returning you an aligned
> >> array is easy:
> >>
> >> In [15]: x = np.arange(6).reshape(2,3)
> >>
> >> In [16]: x.ctypes.data % 16
> >> Out[16]: 0
> >>
> >> but:
> >>
> >> In [17]: x.ctypes.data % 32
> >> Out[17]: 16
> >>
> >> so, in this case NumPy returned a 16-byte aligned array which should be
> >> enough for 128 bit SIMD (SSE family).  This kind of alignment is pretty
> >> common in modern computers.  If you need 256 bit (32-byte) alignment then
> >> you will need to build your container manually.  See here for an example:
> >> http://stackoverflow.com/questions/9895787/memory-alignment-for-fast-fft-in-python-using-shared-arrrays
> >>
> >> Francesc
> >>
> >>
> >>>
> >>> (BTW: the function works  correctly about 20% of the time I run it, and
> >>> else it segfaults on the simd instruction in the the C function)
> >>>
> >>> Thanks,
> >>> -?ystein
> >>>
> >>> _______________________________________________
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion at scipy.org
> >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>
> >>>
> >>
> >>
> >> --
> >> Francesc Alted
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >>
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> 
> 





From zbyszek at in.waw.pl  Tue May 10 13:29:35 2016
From: zbyszek at in.waw.pl (Zbigniew =?utf-8?Q?J=C4=99drzejewski-Szmek?=)
Date: Tue, 10 May 2016 17:29:35 +0000
Subject: [Numpy-discussion] =?utf-8?q?=5BANN=5D_Reminder=3A_Summer_School_?=
 =?utf-8?q?=22Advanced_Scientific_Programming_in_Python=22_in_Reading=2C_U?=
 =?utf-8?q?K=2C_September_5=E2=80=9411=2C_2016?=
Message-ID: <20160510172935.GA3290@in.waw.pl>

Reminder: Deadline for application is 23:59 UTC, May 15, 2016.


Advanced Scientific Programming in Python
=========================================
a Summer School by the G-Node, and the Centre for Integrative Neuroscience and Neurodynamics, School of Psychology and Clinical Language Sciences, University of Reading, UK

Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project ? an entertaining computer game.

We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist.

This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course.

We are striving hard to get a pool of students which is international and gender-balanced.

You can apply online: https://python.g-node.org
Application deadline: 23:59 UTC, May 15, 2016.
Be sure to read the FAQ before applying. 

Participation is for free, i.e. no fee is charged! Participants however should take care of travel, living, and accommodation expenses by themselves. Travel grants may be available.

Date & Location
===============
September 5?11, 2016. Reading, UK

Program
=======
- Best Programming Practices
  ? Best practices for scientific programming
  ? Version control with git and how to contribute to open source projects with GitHub
  ? Best practices in data visualization

- Software Carpentry
  ? Test-driven development
  ? Debugging with a debuggger
  ? Profiling code

- Scientific Tools for Python
  ? Advanced NumPy

- Advanced Python
  ? Decorators
  ? Context managers
  ? Generators

- The Quest for Speed
  ? Writing parallel applications
  ? Interfacing to C with Cython
  ? Memory-bound problems and memory profiling
  ? Data containers: storage and fast access to large data

- Practical Software Development
  ? Group project

Faculty
=======

? Francesc Alted, freelance consultant, author of Blosc, Spain
? Pietro Berkes, Enthought Inc., Cambridge, UK
? Zbigniew J?drzejewski-Szmek, Krasnow Institute, George Mason University, Fairfax, VA, USA
? Eilif Muller, Blue Brain Project, ?cole Polytechnique F?d?rale de Lausanne, Switzerland
? Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universit?t zu Berlin, Germany
? Bartosz Tele?czuk, European Institute for Theoretical Neuroscience, CNRS, Paris, France
? St?fan van der Walt, Berkeley Institute for Data Science, UC Berkeley, CA, USA
? Nelle Varoquaux, Centre for Computational Biology Mines ParisTech, Institut Curie, U900 INSERM, Paris, France
? Tiziano Zito, freelance consultant, Germany

Organizers
==========
For the German Neuroinformatics Node of the INCF (G-Node) Germany:
? Tiziano Zito, freelance consultant, Germany
? Zbigniew J?drzejewski-Szmek, Krasnow Institute, George Mason University, Fairfax, USA
? Jakob Jordan, Institute of Neuroscience and Medicine (INM-6), Forschungszentrum J?lich GmbH, Germany

For the Centre for Integrative Neuroscience and Neurodynamics, School of Psychology and Clinical Language Sciences, University of Reading UK:
? Etienne Roesch, Centre for Integrative Neuroscience and Neurodynamics, University of Reading, UK

Website: https://python.g-node.org
Contact: python-info at g-node.org


From sturla.molden at gmail.com  Wed May 11 04:29:02 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 08:29:02 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
Message-ID: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>

I did some work on this some years ago. I have more or less concluded that
it was a waste of effort. But first let me explain what the suggested
approach do not work. As it uses memory mapping to create shared memory
(i.e. shared segments are not named), they must be created ahead of
spawning processes. But if you really want this to work smoothly, you want
named shared memory (Sys V IPC or posix shm_open), so that shared arrays
can be created in the spawned processes and passed back.

Now for the reason I don't care about shared memory arrays anymore, and
what I am currently working on instead:

1. I have come across very few cases where threaded code cannot be used in
numerical computing. In fact, multithreading nearly always happens in the
code where I write pure C or Fortran anyway. Most often it happens in
library code that are already multithreaded (Intel MKL, Apple Accelerate
Framework, OpenBLAS, etc.), which means using it requires no extra effort
from my side. A multithreaded LAPACK library is not less multithreaded if I
call it from Python.

2. Getting shared memory right can be difficult because of hierarchical
memory and false sharing. You might not see it if you only have a multicore
CPU with a shared cache. But your code might not scale up on computers with
more than one physical processor. False sharing acts like the GIL, except
it happens in hardware and affects your C code invisibly without any
explicit locking you can pinpoint. This is also why MPI code tends to scale
much better than OpenMP code. If nothing is shared there will be no false
sharing.

3. Raw C level IPC is cheap ? very, very cheap. Even if you use pipes or
sockets instead of shared memory it is cheap. There are very few cases
where the IPC tends to be a bottleneck. 

4. The reason IPC appears expensive with NumPy is because multiprocessing
pickles the arrays. It is pickle that is slow, not the IPC. Some would say
that the pickle overhead is an integral part of the IPC ovearhead, but i
will argue that it is not. The slowness of pickle is a separate problem
alltogether.

5. Share memory does not improve on the pickle overhead because also NumPy
arrays with shared memory must be pickled. Multiprocessing can bypass
pickling the RawArray object, but the rest of the NumPy array is pickled.
Using shared memory arrays have no speed advantage over normal NumPy arrays
when we use multiprocessing.

6. It is much easier to write concurrent code that uses queues for message
passing than anything else. That is why using a Queue object has been the
popular Pythonic approach to both multitreading and multiprocessing. I
would like this to continue.

I am therefore focusing my effort on the multiprocessing.Queue object. If
you understand the six points I listed you will see where this is going:
What we really need is a specialized queue that has knowledge about NumPy
arrays and can bypass pickle. I am therefore focusing my efforts on
creating a NumPy aware queue object.

We are not doing the users a favor by encouraging the use of shared memory
arrays. They help with nothing.


Sturla Molden



Mat?j  T?? <matej.tyc at gmail.com> wrote:
> Dear Numpy developers,
> I propose a pull request https://github.com/numpy/numpy/pull/7533 that
> features numpy arrays that can be shared among processes (with some
> effort).
> 
> Why:
> In CPython, multiprocessing is the only way of how to exploit
> multi-core CPUs if your parallel code can't avoid creating Python
> objects. In that case, CPython's GIL makes threads unusable. However,
> unlike with threading, sharing data among processes is something that
> is non-trivial and platform-dependent.
> 
> Although numpy (and certainly some other packages) implement some
> operations in a way that GIL is not a concern, consider another case:
> You have a large amount of data in a form of a numpy array and you
> want to pass it to a function of an arbitrary Python module that also
> expects numpy array (e.g. list of vertices coordinates as an input and
> array of the corresponding polygon as an output). Here, it is clear
> GIL is an issue you and since you want a numpy array on both ends, now
> you would have to copy your numpy array to a multiprocessing.Array (to
> pass the data) and then to convert it back to ndarray in the worker
> process.
> This contribution would streamline it a bit - you would create an
> array as you are used to, pass it to the subprocess as you would do
> with the multiprocessing.Array, and the process can work with a numpy
> array right away.
> 
> How:
> The idea is to create a numpy array in a buffer that can be shared
> among processes. Python has support for this in its standard library,
> so the current solution creates a multiprocessing.Array and then
> passes it as the "buffer" to the ndarray.__new__. That would be it on
> Unixes, but on Windows, there has to be a a custom pickle method,
> otherwise the array "forgets" that its buffer is that special and the
> sharing doesn't work.
> 
> Some of what has been said in the pull request & my answer to that:
> 
> * ... I do see some value in providing a canonical right way to
> construct shared memory arrays in NumPy, but I'm not very happy with
> this solution, ... terrible code organization (with the global
> variables):
> * I understand that, however this is a pattern of Python
> multiprocessing and everybody who wants to use the Pool and shared
> data either is familiar with this approach or has to become familiar
> with[2, 3]. The good compromise is to have a separate module for each
> parallel calculation, so global variables are not a problem.
> 
> * Can you explain why the ndarray subclass is needed? Subclasses can
> be rather annoying to get right, and also for other reasons.
> * The shmarray class needs the custom pickler (but only on Windows).
> 
> * If there's some way to we can paper over the boilerplate such that
> users can use it without understanding the arcana of multiprocessing,
> then yes, that would be great. But otherwise I'm not sure there's
> anything to be gained by putting it in a library rather than referring
> users to the examples on StackOverflow [1] [2].
> * What about telling users: "You can use numpy with multiprocessing.
> Remeber the multiprocessing.Value and multiprocessing.Aray classes?
> numpy.shm works exactly the same way, which means that it shares their
> limitations. Refer to an example: <link to numpy doc>." Notice that
> although those SO links contain all of the information, it is very
> difficult to get it up and running for a newcomer like me few years
> ago.
> 
> * This needs tests and justification for custom pickling methods,
> which are not used in any of the current examples. ...
> * I am sorry, but don't fully understand that point. The custom
> pickling method of shmarray has to be there on Windows, but users
> don't have to know about it at all. As noted earlier, the global
> variable is the only way of using standard Python multiprocessing.Pool
> with shared objects.
> 
> [1]:
> http://stackoverflow.com/questions/10721915/shared-memory-objects-in-python-multiprocessing
> [2]:
> http://stackoverflow.com/questions/7894791/use-numpy-array-in-shared-memory-for-multiprocessing
> [3]:
> http://stackoverflow.com/questions/1675766/how-to-combine-pool-map-with-array-shared-memory-in-python-multiprocessing



From Permafacture at gmail.com  Wed May 11 10:41:46 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Wed, 11 May 2016 09:41:46 -0500
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
Message-ID: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>

Strula, this sounds brilliant!  To be clear, you're talking about
serializing the numpy array and reconstructing it in a way that's faster
than pickle? Or using shared memory and signaling array creation around
that shared memory rather than using pickle?

For what it's worth, I have used shared memory with numpy arrays as IPC (no
queue), with one process writing to it and one process reading from it, and
liked it.  Your point #5 did not apply because I was reusing the shared
memory.

Do you have a public repo where you are working on this?

Thanks!
  Elliot

On Wed, May 11, 2016 at 3:29 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> I did some work on this some years ago. I have more or less concluded that
> it was a waste of effort. But first let me explain what the suggested
> approach do not work. As it uses memory mapping to create shared memory
> (i.e. shared segments are not named), they must be created ahead of
> spawning processes. But if you really want this to work smoothly, you want
> named shared memory (Sys V IPC or posix shm_open), so that shared arrays
> can be created in the spawned processes and passed back.
>
> Now for the reason I don't care about shared memory arrays anymore, and
> what I am currently working on instead:
>
> 1. I have come across very few cases where threaded code cannot be used in
> numerical computing. In fact, multithreading nearly always happens in the
> code where I write pure C or Fortran anyway. Most often it happens in
> library code that are already multithreaded (Intel MKL, Apple Accelerate
> Framework, OpenBLAS, etc.), which means using it requires no extra effort
> from my side. A multithreaded LAPACK library is not less multithreaded if I
> call it from Python.
>
> 2. Getting shared memory right can be difficult because of hierarchical
> memory and false sharing. You might not see it if you only have a multicore
> CPU with a shared cache. But your code might not scale up on computers with
> more than one physical processor. False sharing acts like the GIL, except
> it happens in hardware and affects your C code invisibly without any
> explicit locking you can pinpoint. This is also why MPI code tends to scale
> much better than OpenMP code. If nothing is shared there will be no false
> sharing.
>
> 3. Raw C level IPC is cheap ? very, very cheap. Even if you use pipes or
> sockets instead of shared memory it is cheap. There are very few cases
> where the IPC tends to be a bottleneck.
>
> 4. The reason IPC appears expensive with NumPy is because multiprocessing
> pickles the arrays. It is pickle that is slow, not the IPC. Some would say
> that the pickle overhead is an integral part of the IPC ovearhead, but i
> will argue that it is not. The slowness of pickle is a separate problem
> alltogether.
>
> 5. Share memory does not improve on the pickle overhead because also NumPy
> arrays with shared memory must be pickled. Multiprocessing can bypass
> pickling the RawArray object, but the rest of the NumPy array is pickled.
> Using shared memory arrays have no speed advantage over normal NumPy arrays
> when we use multiprocessing.
>
> 6. It is much easier to write concurrent code that uses queues for message
> passing than anything else. That is why using a Queue object has been the
> popular Pythonic approach to both multitreading and multiprocessing. I
> would like this to continue.
>
> I am therefore focusing my effort on the multiprocessing.Queue object. If
> you understand the six points I listed you will see where this is going:
> What we really need is a specialized queue that has knowledge about NumPy
> arrays and can bypass pickle. I am therefore focusing my efforts on
> creating a NumPy aware queue object.
>
> We are not doing the users a favor by encouraging the use of shared memory
> arrays. They help with nothing.
>
>
> Sturla Molden
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160511/21cc04c2/attachment.html>

From allanhaldane at gmail.com  Wed May 11 14:01:02 2016
From: allanhaldane at gmail.com (Allan Haldane)
Date: Wed, 11 May 2016 14:01:02 -0400
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
Message-ID: <5733735E.2040309@gmail.com>

On 05/11/2016 04:29 AM, Sturla Molden wrote:
> 4. The reason IPC appears expensive with NumPy is because multiprocessing
> pickles the arrays. It is pickle that is slow, not the IPC. Some would say
> that the pickle overhead is an integral part of the IPC ovearhead, but i
> will argue that it is not. The slowness of pickle is a separate problem
> alltogether.

That's interesting. I've also used multiprocessing with numpy and didn't
realize that. Is this true in python3 too?

In python2 it appears that multiprocessing uses pickle protocol 0 which
must cause a big slowdown (a factor of 100) relative to protocol 2, and
uses pickle instead of cPickle.

a = np.arange(40*40)

%timeit pickle.dumps(a)
1000 loops, best of 3: 1.63 ms per loop

%timeit cPickle.dumps(a)
1000 loops, best of 3: 1.56 ms per loop

%timeit cPickle.dumps(a, protocol=2)
100000 loops, best of 3: 18.9 ?s per loop

Python 3 uses protocol 3 by default:

%timeit pickle.dumps(a)
10000 loops, best of 3: 20 ?s per loop


> 5. Share memory does not improve on the pickle overhead because also NumPy
> arrays with shared memory must be pickled. Multiprocessing can bypass
> pickling the RawArray object, but the rest of the NumPy array is pickled.
> Using shared memory arrays have no speed advantage over normal NumPy arrays
> when we use multiprocessing.
> 
> 6. It is much easier to write concurrent code that uses queues for message
> passing than anything else. That is why using a Queue object has been the
> popular Pythonic approach to both multitreading and multiprocessing. I
> would like this to continue.
> 
> I am therefore focusing my effort on the multiprocessing.Queue object. If
> you understand the six points I listed you will see where this is going:
> What we really need is a specialized queue that has knowledge about NumPy
> arrays and can bypass pickle. I am therefore focusing my efforts on
> creating a NumPy aware queue object.
> 
> We are not doing the users a favor by encouraging the use of shared memory
> arrays. They help with nothing.
> 
> 
> Sturla Molden




From ben.v.root at gmail.com  Wed May 11 14:22:54 2016
From: ben.v.root at gmail.com (Benjamin Root)
Date: Wed, 11 May 2016 14:22:54 -0400
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <5733735E.2040309@gmail.com>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
Message-ID: <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>

Oftentimes, if one needs to share numpy arrays for multiprocessing, I would
imagine that it is because the array is huge, right? So, the pickling
approach would copy that array for each process, which defeats the purpose,
right?

Ben Root

On Wed, May 11, 2016 at 2:01 PM, Allan Haldane <allanhaldane at gmail.com>
wrote:

> On 05/11/2016 04:29 AM, Sturla Molden wrote:
> > 4. The reason IPC appears expensive with NumPy is because multiprocessing
> > pickles the arrays. It is pickle that is slow, not the IPC. Some would
> say
> > that the pickle overhead is an integral part of the IPC ovearhead, but i
> > will argue that it is not. The slowness of pickle is a separate problem
> > alltogether.
>
> That's interesting. I've also used multiprocessing with numpy and didn't
> realize that. Is this true in python3 too?
>
> In python2 it appears that multiprocessing uses pickle protocol 0 which
> must cause a big slowdown (a factor of 100) relative to protocol 2, and
> uses pickle instead of cPickle.
>
> a = np.arange(40*40)
>
> %timeit pickle.dumps(a)
> 1000 loops, best of 3: 1.63 ms per loop
>
> %timeit cPickle.dumps(a)
> 1000 loops, best of 3: 1.56 ms per loop
>
> %timeit cPickle.dumps(a, protocol=2)
> 100000 loops, best of 3: 18.9 ?s per loop
>
> Python 3 uses protocol 3 by default:
>
> %timeit pickle.dumps(a)
> 10000 loops, best of 3: 20 ?s per loop
>
>
> > 5. Share memory does not improve on the pickle overhead because also
> NumPy
> > arrays with shared memory must be pickled. Multiprocessing can bypass
> > pickling the RawArray object, but the rest of the NumPy array is pickled.
> > Using shared memory arrays have no speed advantage over normal NumPy
> arrays
> > when we use multiprocessing.
> >
> > 6. It is much easier to write concurrent code that uses queues for
> message
> > passing than anything else. That is why using a Queue object has been the
> > popular Pythonic approach to both multitreading and multiprocessing. I
> > would like this to continue.
> >
> > I am therefore focusing my effort on the multiprocessing.Queue object. If
> > you understand the six points I listed you will see where this is going:
> > What we really need is a specialized queue that has knowledge about NumPy
> > arrays and can bypass pickle. I am therefore focusing my efforts on
> > creating a NumPy aware queue object.
> >
> > We are not doing the users a favor by encouraging the use of shared
> memory
> > arrays. They help with nothing.
> >
> >
> > Sturla Molden
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160511/45776874/attachment.html>

From rainwoodman at gmail.com  Wed May 11 18:38:15 2016
From: rainwoodman at gmail.com (Feng Yu)
Date: Wed, 11 May 2016 15:38:15 -0700
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
Message-ID: <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>

Hi,

I've been thinking and exploring this for some time. If we are to
start some effort I'd like to help. Here are my comments, mostly
regarding to Sturla's comments.

1. If we are talking about shared memory and copy-on-write
inheritance, then we are using 'fork'. If we are free to use fork,
then a large chunk of the concerns regarding the python std library
multiprocessing is no longer relevant. Especially those functions must
be in a module limitation that tends to impose a special requirement
on the software design.

2. Picking of inherited shared memory array can be done minimally by
just picking the array_interface and the pointer address. It is
because the child process and the parent share the same address space
layout, guarenteed by the fork call.

3. The RawArray and RawValue implementation in std multiprocessing has
its own memory allocator for managing small variables. It is a huge
overkill (in terms of implementation) if we only care about very large
memory chunks.

4. Hidden sychronization cost on multi-cpu (NUMA?) systems. A choice
is to defer the responsibility of avoiding racing to the developer.
Simple structs for working on slices of array in parallel can cover a
huge fraction of use cases and fully avoid this issue.

5. Whether to delegate parallelism to underlying low level
implementation or to implement the paralellism in python while
maintaining the underlying low level implementation sequential is
probably dependent on the problem. It may be convenient as of the
current state of parallelism support in Python to delegate, but will
it forever be the case?

For example, after the MPI FFTW binding stuck for a long time, someone
wrote a parallel python FFT package
(https://github.com/spectralDNS/mpiFFT4py)  that uses FFTW for
sequential and write all parallel semantics in Python with mpi4py, and
it uses a more efficient domain decomposition.

6. If we are to define a set of operations I would recommend take a
look at OpenMP as a reference -- It has been out there for decades and
used widely. An equiavlant to the 'omp parallel for' construct in
Python will be a very good starting point and immediately useful.

- Yu

On Wed, May 11, 2016 at 11:22 AM, Benjamin Root <ben.v.root at gmail.com> wrote:
> Oftentimes, if one needs to share numpy arrays for multiprocessing, I would
> imagine that it is because the array is huge, right? So, the pickling
> approach would copy that array for each process, which defeats the purpose,
> right?
>
> Ben Root
>
> On Wed, May 11, 2016 at 2:01 PM, Allan Haldane <allanhaldane at gmail.com>
> wrote:
>>
>> On 05/11/2016 04:29 AM, Sturla Molden wrote:
>> > 4. The reason IPC appears expensive with NumPy is because
>> > multiprocessing
>> > pickles the arrays. It is pickle that is slow, not the IPC. Some would
>> > say
>> > that the pickle overhead is an integral part of the IPC ovearhead, but i
>> > will argue that it is not. The slowness of pickle is a separate problem
>> > alltogether.
>>
>> That's interesting. I've also used multiprocessing with numpy and didn't
>> realize that. Is this true in python3 too?
>>
>> In python2 it appears that multiprocessing uses pickle protocol 0 which
>> must cause a big slowdown (a factor of 100) relative to protocol 2, and
>> uses pickle instead of cPickle.
>>
>> a = np.arange(40*40)
>>
>> %timeit pickle.dumps(a)
>> 1000 loops, best of 3: 1.63 ms per loop
>>
>> %timeit cPickle.dumps(a)
>> 1000 loops, best of 3: 1.56 ms per loop
>>
>> %timeit cPickle.dumps(a, protocol=2)
>> 100000 loops, best of 3: 18.9 ?s per loop
>>
>> Python 3 uses protocol 3 by default:
>>
>> %timeit pickle.dumps(a)
>> 10000 loops, best of 3: 20 ?s per loop
>>
>>
>> > 5. Share memory does not improve on the pickle overhead because also
>> > NumPy
>> > arrays with shared memory must be pickled. Multiprocessing can bypass
>> > pickling the RawArray object, but the rest of the NumPy array is
>> > pickled.
>> > Using shared memory arrays have no speed advantage over normal NumPy
>> > arrays
>> > when we use multiprocessing.
>> >
>> > 6. It is much easier to write concurrent code that uses queues for
>> > message
>> > passing than anything else. That is why using a Queue object has been
>> > the
>> > popular Pythonic approach to both multitreading and multiprocessing. I
>> > would like this to continue.
>> >
>> > I am therefore focusing my effort on the multiprocessing.Queue object.
>> > If
>> > you understand the six points I listed you will see where this is going:
>> > What we really need is a specialized queue that has knowledge about
>> > NumPy
>> > arrays and can bypass pickle. I am therefore focusing my efforts on
>> > creating a NumPy aware queue object.
>> >
>> > We are not doing the users a favor by encouraging the use of shared
>> > memory
>> > arrays. They help with nothing.
>> >
>> >
>> > Sturla Molden
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From joferkington at gmail.com  Wed May 11 18:39:49 2016
From: joferkington at gmail.com (Joe Kington)
Date: Wed, 11 May 2016 15:39:49 -0700
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <5733735E.2040309@gmail.com>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
Message-ID: <CACe1pJedq8AVPiqXRZfrF8iqgKeU_5E0qo9Xg8GdyC-Wa+Lh1g@mail.gmail.com>

In python2 it appears that multiprocessing uses pickle protocol 0 which
> must cause a big slowdown (a factor of 100) relative to protocol 2, and
> uses pickle instead of cPickle.
>
>
Even on Python 2.x, multiprocessing uses protocol 2, not protocol 0.  The
default for the `pickle` module changed, but multiprocessing has always
used a binary pickle protocol to communicate between processes.  Have a
look at multiprocessing's forking.py in Python 2.7.

As some context here for folks that may not be aware, Sturla is referring
to his earlier shared memory implementation
<https://github.com/sturlamolden/sharedmem-numpy> he wrote that avoids
actually pickling the data, and instead essentially pickles a pointer to an
array in shared memory.  As Sturla very nicely summed up, it saves memory
usage, but doesn't help the deeper issues.  You're far better off just
communicating between processes as opposed to using shared memory.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160511/68a03305/attachment.html>

From sturla.molden at gmail.com  Wed May 11 18:48:22 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 22:48:22 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
Message-ID: <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>

Elliot Hallmark <Permafacture at gmail.com> wrote:
> Strula, this sounds brilliant!  To be clear, you're talking about
> serializing the numpy array and reconstructing it in a way that's faster
> than pickle?

Yes. We know the binary format of NumPy arrays. We don't need to invoke the
machinery of pickle to serialize an array and write the bytes to some IPC
mechanism (pipe, tcp socket, unix socket, shared memory). The choise of IPC
mechanism might not even be relevant, and could even be deferred to a
library like ZeroMQ. The point is that if multiple peocesses are to
cooperate efficiently, we need a way to let them communicate NumPy arrays
quickly. That is where using multiprocessing hurts today, and shared memory
does not help here.

Sturla



From sturla.molden at gmail.com  Wed May 11 18:48:23 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 22:48:23 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
Message-ID: <1865553010484698763.952860sturla.molden-gmail.com@news.gmane.org>

Allan Haldane <allanhaldane at gmail.com> wrote:

> That's interesting. I've also used multiprocessing with numpy and didn't
> realize that. Is this true in python3 too?

I am not sure. As you have noticed, pickle is faster by to orders of
magnitude on Python 3. But several microseconds is also a lot, particularly
if we are going to do this often during a computation.

Sturla



From sturla.molden at gmail.com  Wed May 11 18:48:23 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 22:48:23 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
Message-ID: <1393378874484697930.449371sturla.molden-gmail.com@news.gmane.org>

Benjamin Root <ben.v.root at gmail.com> wrote:

> Oftentimes, if one needs to share numpy arrays for multiprocessing, I would
> imagine that it is because the array is huge, right? 

That is a case for shared memory, but what. i was taking about is more
common than this. In order for processes to cooperate, they must
communicate. So we need a way to pass around NumPy arrays quickly.
Sometimes we want to use shared memory because of the size of the data, but
more often it is just used as a form of inexpensive IPC.

> So, the pickling
> approach would copy that array for each process, which defeats the purpose,
> right?

I am not sure what you mean. When I made shared memory arrays I used named
segments, and made sure only the name of the segments were pickled, not the
contents of the buffers.

Sturla



From sturla.molden at gmail.com  Wed May 11 19:02:05 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 23:02:05 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CACe1pJedq8AVPiqXRZfrF8iqgKeU_5E0qo9Xg8GdyC-Wa+Lh1g@mail.gmail.com>
Message-ID: <771832558484700460.287024sturla.molden-gmail.com@news.gmane.org>

Joe Kington <joferkington at gmail.com> wrote:

> You're far better off just
> communicating between processes as opposed to using shared memory.

Yes.



From sturla.molden at gmail.com  Wed May 11 19:02:06 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 11 May 2016 23:02:06 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
Message-ID: <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>

Feng Yu <rainwoodman at gmail.com> wrote:

> 1. If we are talking about shared memory and copy-on-write
> inheritance, then we are using 'fork'. 

Not available on Windows. On Unix it only allows one-way communication,
from parent to child.

> 2. Picking of inherited shared memory array can be done minimally by
> just picking the array_interface and the pointer address. It is
> because the child process and the parent share the same address space
> layout, guarenteed by the fork call.

Again, not everyone uses Unix. 

And on Unix it is not trival to pass data back from the child process. I
solved that problem with Sys V IPC (pickling the name of the segment).

> 6. If we are to define a set of operations I would recommend take a
> look at OpenMP as a reference -- It has been out there for decades and
> used widely. An equiavlant to the 'omp parallel for' construct in
> Python will be a very good starting point and immediately useful.

If you are on Unix, you can just use a context manager. Call os.fork in
__enter__ and os.waitpid in __exit__. 

Sturla



From allanhaldane at gmail.com  Wed May 11 19:26:03 2016
From: allanhaldane at gmail.com (Allan Haldane)
Date: Wed, 11 May 2016 19:26:03 -0400
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <CACe1pJedq8AVPiqXRZfrF8iqgKeU_5E0qo9Xg8GdyC-Wa+Lh1g@mail.gmail.com>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CACe1pJedq8AVPiqXRZfrF8iqgKeU_5E0qo9Xg8GdyC-Wa+Lh1g@mail.gmail.com>
Message-ID: <5733BF8B.50707@gmail.com>

On 05/11/2016 06:39 PM, Joe Kington wrote:
> 
> 
>     In python2 it appears that multiprocessing uses pickle protocol 0 which
>     must cause a big slowdown (a factor of 100) relative to protocol 2, and
>     uses pickle instead of cPickle.
> 
> 
> Even on Python 2.x, multiprocessing uses protocol 2, not protocol 0. 
> The default for the `pickle` module changed, but multiprocessing has
> always used a binary pickle protocol to communicate between processes. 
> Have a look at multiprocessing's forking.py <http://forking.py> in
> Python 2.7.

Are you sure? As far as I understood the code, it uses the default
protocol 0. The file forking.py no longer exists, also.

https://github.com/python/cpython/tree/master/Lib/multiprocessing
(see reduction.py and queue.py)
http://bugs.python.org/issue23403


From allanhaldane at gmail.com  Wed May 11 19:30:14 2016
From: allanhaldane at gmail.com (Allan Haldane)
Date: Wed, 11 May 2016 19:30:14 -0400
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
Message-ID: <5733C086.9020403@gmail.com>

On 05/11/2016 06:48 PM, Sturla Molden wrote:
> Elliot Hallmark <Permafacture at gmail.com> wrote:
>> Strula, this sounds brilliant!  To be clear, you're talking about
>> serializing the numpy array and reconstructing it in a way that's faster
>> than pickle?
> 
> Yes. We know the binary format of NumPy arrays. We don't need to invoke the
> machinery of pickle to serialize an array and write the bytes to some IPC
> mechanism (pipe, tcp socket, unix socket, shared memory). The choise of IPC
> mechanism might not even be relevant, and could even be deferred to a
> library like ZeroMQ. The point is that if multiple peocesses are to
> cooperate efficiently, we need a way to let them communicate NumPy arrays
> quickly. That is where using multiprocessing hurts today, and shared memory
> does not help here.
> 
> Sturla

You probably already know this, but I just wanted to note that the
mpi4py module has worked around pickle too. They discuss how they
efficiently transfer numpy arrays in mpi messages here:
http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data

Of course not everyone is able to install mpi easily.




From sturla.molden at gmail.com  Thu May 12 02:27:43 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 12 May 2016 06:27:43 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
Message-ID: <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>

Allan Haldane <allanhaldane at gmail.com> wrote:

> You probably already know this, but I just wanted to note that the
> mpi4py module has worked around pickle too. They discuss how they
> efficiently transfer numpy arrays in mpi messages here:
> http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data

Unless I am mistaken, they use the PEP 3118 buffer interface to support
NumPy as well as a number of other Python objects. However, this protocol
makes buffer aquisition an expensive operation. You can see this in Cython
if you use typed memory views. Assigning a NumPy array to a typed
memoryview (i,e, buffer acqisition) is slow. They are correct that avoiding
pickle means we save some memory. It also avoids creating and destroying
temporary Python objects, and associated reference counting. However,
because of the expensive buffer acquisition, I am not sure how much faster
their apporach will be. I prefer to use the NumPy C API, and bypass any
unneccesary overhead. The idea is to make IPC of NumPy arrays fast, and
then we cannot have an expensive buffer acquisition in there.

Sturla



From niki.spahiev at gmail.com  Thu May 12 03:06:27 2016
From: niki.spahiev at gmail.com (Niki Spahiev)
Date: Thu, 12 May 2016 10:06:27 +0300
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
In-Reply-To: <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
Message-ID: <nh19vk$iih$1@ger.gmane.org>

On 12.05.2016 02:02, Sturla Molden wrote:
> Feng Yu <rainwoodman at gmail.com> wrote:
>
>> 1. If we are talking about shared memory and copy-on-write
>> inheritance, then we are using 'fork'.
>
> Not available on Windows. On Unix it only allows one-way communication,
> from parent to child.

Apparently next Win10 will have fork as part of bash integration.

Niki



From cimrman3 at ntc.zcu.cz  Thu May 12 03:51:35 2016
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Thu, 12 May 2016 09:51:35 +0200
Subject: [Numpy-discussion] ANN: SfePy 2016.2
Message-ID: <57343607.2020100@ntc.zcu.cz>

I am pleased to announce release 2016.2 of SfePy.

Description
-----------

SfePy (simple finite elements in Python) is a software for solving systems of
coupled partial differential equations by the finite element method or by the
isogeometric analysis (preliminary support). It is distributed under the new
BSD license.

Home page: http://sfepy.org
Mailing list: http://groups.google.com/group/sfepy-devel
Git (source) repository, issue tracker, wiki: http://github.com/sfepy

Highlights of this release
--------------------------

- partial shell10x element implementation
- parallel computation of homogenized coefficients
- clean up of elastic terms
- read support for msh file mesh format of gmsh

For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1
(rather long and technical).

Best regards,
Robert Cimrman on behalf of the SfePy development team

---

Contributors to this release in alphabetical order:

Robert Cimrman
Vladimir Lukes


From evgeny.burovskiy at gmail.com  Thu May 12 09:02:03 2016
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Thu, 12 May 2016 14:02:03 +0100
Subject: [Numpy-discussion] scipy 0.17.1 release
Message-ID: <CAMRo0ivDDsDeSdSo91LDk4_Wo3aXRi+yrKpvHe1pnYYRkm8dzQ@mail.gmail.com>

Hi,

On behalf of the scipy development team, I'm pleased to announce the
availability of scipy 0.17.1.

This is a bugfix release with no new features compared to 0.17.0.

Source tarballs and OS X wheels are available from PyPI or from GitHub
releases at https://github.com/scipy/scipy/releases/tag/v0.17.1

We recommend that all users upgrade from scipy 0.17.0.

Cheers,

Evgeni



==========================
SciPy 0.17.1 Release Notes
==========================

SciPy 0.17.1 is a bug-fix release with no new features compared to 0.17.0.


Issues closed for 0.17.1
------------------------

- `#5817 <https://github.com/scipy/scipy/issues/5817>`__: BUG: skew,
kurtosis return np.nan instead of "propagate"
- `#5850 <https://github.com/scipy/scipy/issues/5850>`__: Test failed
with sgelsy
- `#5898 <https://github.com/scipy/scipy/issues/5898>`__:
interpolate.interp1d crashes using float128
- `#5953 <https://github.com/scipy/scipy/issues/5953>`__: Massive
performance regression in cKDTree.query with L_inf distance...
- `#6062 <https://github.com/scipy/scipy/issues/6062>`__: mannwhitneyu
breaks backward compatibility in 0.17.0
- `#6134 <https://github.com/scipy/scipy/issues/6134>`__: T test does
not handle nans


Pull requests for 0.17.1
------------------------

- `#5902 <https://github.com/scipy/scipy/pull/5902>`__: BUG:
interpolate: make interp1d handle np.float128 again
- `#5957 <https://github.com/scipy/scipy/pull/5957>`__: BUG: slow down
with p=np.inf in 0.17 cKDTree.query
- `#5970 <https://github.com/scipy/scipy/pull/5970>`__: Actually
propagate nans through stats functions with nan_policy="propagate"
- `#5971 <https://github.com/scipy/scipy/pull/5971>`__: BUG: linalg:
fix lwork check in *gelsy
- `#6074 <https://github.com/scipy/scipy/pull/6074>`__: BUG: special:
fixed violation of strict aliasing rules.
- `#6083 <https://github.com/scipy/scipy/pull/6083>`__: BUG: Fix dtype
for sum of linear operators
- `#6100 <https://github.com/scipy/scipy/pull/6100>`__: BUG: Fix
mannwhitneyu to be backward compatible
- `#6135 <https://github.com/scipy/scipy/pull/6135>`__: Don't pass
null pointers to LAPACK, even during workspace queries.
- `#6148 <https://github.com/scipy/scipy/pull/6148>`__: stats: fix
handling of nan values in T tests and kendalltau


From solipsis at pitrou.net  Thu May 12 11:38:10 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 May 2016 17:38:10 +0200
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
 <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>
Message-ID: <20160512173810.79fb21b5@fsol>

On Thu, 12 May 2016 06:27:43 +0000 (UTC)
Sturla Molden <sturla.molden at gmail.com> wrote:

> Allan Haldane <allanhaldane at gmail.com> wrote:
> 
> > You probably already know this, but I just wanted to note that the
> > mpi4py module has worked around pickle too. They discuss how they
> > efficiently transfer numpy arrays in mpi messages here:
> > http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data
> 
> Unless I am mistaken, they use the PEP 3118 buffer interface to support
> NumPy as well as a number of other Python objects. However, this protocol
> makes buffer aquisition an expensive operation.

Can you define "expensive"?

> You can see this in Cython
> if you use typed memory views. Assigning a NumPy array to a typed
> memoryview (i,e, buffer acqisition) is slow.

You're assuming this is the cost of "buffer acquisition", while most
likely it's the cost of creating the memoryview object itself.

Buffer acquisition itself only calls a single C callback and uses a
stack-allocated C structure. It shouldn't be "expensive".

Regards

Antoine.




From rainwoodman at gmail.com  Thu May 12 16:46:14 2016
From: rainwoodman at gmail.com (Feng Yu)
Date: Thu, 12 May 2016 13:46:14 -0700
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CANGhFjZ6HKb=hLtLiUFot6jVkt6rKGxXoK4yLWg_8hnbJPrL4g@mail.gmail.com>

> Again, not everyone uses Unix.
>
> And on Unix it is not trival to pass data back from the child process. I
> solved that problem with Sys V IPC (pickling the name of the segment).
>

I wonder if it is neccessary insist being able to pass large amount of data
back from child to the parent process.

In most (half?) situations the result can be directly write back via
preallocated shared array before works are spawned. Then there is no
need to pass data back with named segments.

Here I am just doodling some possible use cases along the OpenMP line.
The sample would just copy the data from s to r, in two different
ways. On systems that does not support multiprocess + fork, the
semantics is still well preserved if threading is used.

```
import ...... as mp

# the access attribute of inherited variables is at least 'privatecopy'
# but with threading backend it becomes 'shared'
s = numpy.arange(10000)

with mp.parallel(num_threads=8) as section:
    r = section.empty(10000) # variables defined via section.empty
will always be 'shared'
    def work():
         # variables defined in the body is 'private'
         tid = section.get_thread_num()
         size = section.get_num_threads()
         sl = slice(tid * r.size // size, (tid + 1) * r.size // size)
         r[sl] = s[sl]

    status = section.run(work)
    assert not any(status.errors)

    # the support to the following could be implemented with section.run

    chunksize = 1000
    def work(i):
          sl = slice(i, i + chunksize)
          r[sl] = s[sl]
          return s[sl].sum()
    status = section.loop(work, range(0, r.size, chunksize), schedule='static')
    assert not any(status.errors)
    total = sum(status.results)
```

>> 6. If we are to define a set of operations I would recommend take a
>> look at OpenMP as a reference -- It has been out there for decades and
>> used widely. An equiavlant to the 'omp parallel for' construct in
>> Python will be a very good starting point and immediately useful.
>
> If you are on Unix, you can just use a context manager. Call os.fork in
> __enter__ and os.waitpid in __exit__.
>
> Sturla
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion


From sturla.molden at gmail.com  Thu May 12 19:14:36 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 12 May 2016 23:14:36 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
 <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>
 <20160512173810.79fb21b5@fsol>
Message-ID: <115932182484786870.778106sturla.molden-gmail.com@news.gmane.org>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> Can you define "expensive"?

Slow enough to cause complaints on the Cython mailing list.

> You're assuming this is the cost of "buffer acquisition", while most
> likely it's the cost of creating the memoryview object itself.

Constructing a typed memoryview from a typed memoryview or a slice is fast.
Numerical code doing this intensively is still within 80-90% of the speed
of plain c code using pointer arithmetics.

 
> Buffer acquisition itself only calls a single C callback and uses a
> stack-allocated C structure. It shouldn't be "expensive".

I don't know the reason, only that buffer acquisition from NumPy arrays
with typed memoryviews is very expensive compared to assigning a typed
memoryview to another or slicing a typed memoryview.

Sturla



From sturla.molden at gmail.com  Thu May 12 19:14:35 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 12 May 2016 23:14:35 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
 <nh19vk$iih$1@ger.gmane.org>
Message-ID: <1352160884484787255.075220sturla.molden-gmail.com@news.gmane.org>

Niki Spahiev <niki.spahiev at gmail.com> wrote:

> Apparently next Win10 will have fork as part of bash integration.

That would be great. The lack of fork on Windows is very annoying.

Sturla



From dave.hirschfeld at gmail.com  Thu May 12 19:25:55 2016
From: dave.hirschfeld at gmail.com (Dave)
Date: Thu, 12 May 2016 23:25:55 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Numpy_arrays_shareable_among_related?=
	=?utf-8?b?CXByb2Nlc3NlcyAoUFIgIzc1MzMp?=
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
 <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>
 <20160512173810.79fb21b5@fsol>
Message-ID: <loom.20160513T011421-324@post.gmane.org>

Antoine Pitrou <solipsis <at> pitrou.net> writes:

> 
> On Thu, 12 May 2016 06:27:43 +0000 (UTC)
> Sturla Molden <sturla.molden <at> gmail.com> wrote:
> 
> > Allan Haldane <allanhaldane <at> gmail.com> wrote:
> > 
> > > You probably already know this, but I just wanted to note that the
> > > mpi4py module has worked around pickle too. They discuss how they
> > > efficiently transfer numpy arrays in mpi messages here:
> > > http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-
python-objects-and-array-data
> > 
> > Unless I am mistaken, they use the PEP 3118 buffer interface to 
support
> > NumPy as well as a number of other Python objects. However, this 
protocol
> > makes buffer aquisition an expensive operation.
> 
> Can you define "expensive"?
> 
> > You can see this in Cython
> > if you use typed memory views. Assigning a NumPy array to a typed
> > memoryview (i,e, buffer acqisition) is slow.
> 
> You're assuming this is the cost of "buffer acquisition", while most
> likely it's the cost of creating the memoryview object itself.
> 
> Buffer acquisition itself only calls a single C callback and uses a
> stack-allocated C structure. It shouldn't be "expensive".
> 
> Regards
> 
> Antoine.
> 


When I looked at it, using a typed memoryview was between 7-50 times 
slower than using numpy directly:

http://thread.gmane.org/gmane.comp.python.cython.devel/14626


It looks like there was some improvement since then:

https://github.com/numpy/numpy/pull/3779


...and repeating my experiment shows the deficit is down to 3-11 times 
slower.


In [5]: x = randn(10000)

In [6]: %timeit echo_memview(x)
The slowest run took 14.98 times longer than the fastest. This could 
mean that an intermediate result is being cached.
100000 loops, best of 3: 5.31 ?s per loop

In [7]: %timeit echo_memview_nocast(x)
The slowest run took 10.80 times longer than the fastest. This could 
mean that an intermediate result is being cached.
1000000 loops, best of 3: 1.58 ?s per loop

In [8]: %timeit echo_numpy(x)
The slowest run took 58.81 times longer than the fastest. This could 
mean that an intermediate result is being cached.
1000000 loops, best of 3: 474 ns per loop



-Dave

From sturla.molden at gmail.com  Thu May 12 19:32:35 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 12 May 2016 23:32:35 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
 <CANGhFjZ6HKb=hLtLiUFot6jVkt6rKGxXoK4yLWg_8hnbJPrL4g@mail.gmail.com>
Message-ID: <864303179484787737.086354sturla.molden-gmail.com@news.gmane.org>

Feng Yu <rainwoodman at gmail.com> wrote:
 
> In most (half?) situations the result can be directly write back via
> preallocated shared array before works are spawned. Then there is no
> need to pass data back with named segments.

You can work around it in various ways, this being one of them.

Personally I prefer a parallel programming style with queues ? either to
scatter arrays to workers and collecting arrays from workers, or to chain
workers together in a pipeline (without using coroutines). But exactly how
you program is a matter of taste. I want to make it as inexpensive as
possible to pass a NumPy array through a queue. If anyone else wants to
help improve parallel programming with NumPy using a different paradigm,
that is fine too. I just wanted to clarify why I stopped working on shared
memory arrays.

(As for the implementation, I am also experimenting with platform dependent
asynchronous I/O (IOCP, GCD or kqueue, epoll) to pass NumPy arrays though a
queue as inexpensively and scalably as possible. And no, there is no public
repo, as I like to experiment with my pet project undisturbed before I let
it out in the wild.)


Sturla



From sturla.molden at gmail.com  Fri May 13 13:29:13 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Fri, 13 May 2016 17:29:13 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
 <nh19vk$iih$1@ger.gmane.org>
Message-ID: <444540179484853062.734900sturla.molden-gmail.com@news.gmane.org>

Niki Spahiev <niki.spahiev at gmail.com> wrote:

> Apparently next Win10 will have fork as part of bash integration.

It is Interix/SUA rebranded "Subsystem for Linux". It remains to be seen
how long it will stay this time. Also a Python built for this subsystem
will not run on the Win32 subsystem, so there is no graphics. Also it will
not be installed by default, just like SUA.



From rainwoodman at gmail.com  Fri May 13 15:44:34 2016
From: rainwoodman at gmail.com (Feng Yu)
Date: Fri, 13 May 2016 12:44:34 -0700
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <864303179484787737.086354sturla.molden-gmail.com@news.gmane.org>
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
 <CANGhFjZ6HKb=hLtLiUFot6jVkt6rKGxXoK4yLWg_8hnbJPrL4g@mail.gmail.com>
 <864303179484787737.086354sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CANGhFjYaO3c_LNtC2n1wmhn8TQ8MPp21ChoXRgjgz=B1f_pRSQ@mail.gmail.com>

>
> Personally I prefer a parallel programming style with queues ? either to
> scatter arrays to workers and collecting arrays from workers, or to chain
> workers together in a pipeline (without using coroutines). But exactly how
> you program is a matter of taste. I want to make it as inexpensive as
> possible to pass a NumPy array through a queue. If anyone else wants to
> help improve parallel programming with NumPy using a different paradigm,
> that is fine too. I just wanted to clarify why I stopped working on shared
> memory arrays.

Even I am not very obsessed with functional and queues, I still have
to agree with you
queues tend to produce more readable and less verbose code -- if there
is the right tool.

>
> (As for the implementation, I am also experimenting with platform dependent
> asynchronous I/O (IOCP, GCD or kqueue, epoll) to pass NumPy arrays though a
> queue as inexpensively and scalably as possible. And no, there is no public
> repo, as I like to experiment with my pet project undisturbed before I let
> it out in the wild.)

It will be wonderful if there is a way to pass numpy array around
without a huge dependency list.

After all, we know the address of the array and, in principle we are
able to find the physical pages and map them in the receiver side.

Also, did you checkout http://zeromq.org/blog:zero-copy ?
ZeroMQ is a dependency of Jupyter, so it is quite available.

- Yu

>

>
> Sturla
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion


From sturla.molden at gmail.com  Fri May 13 19:15:59 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Fri, 13 May 2016 23:15:59 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <5733735E.2040309@gmail.com>
 <CANNq6Fn3d9AVDz87zHRsTM2rYrSHDgNubr9O2mD40T2dY0bhYA@mail.gmail.com>
 <CANGhFja+6PWFMHHbbudWFihvpsO9EONruUAr9Oa-N5AzQrWhFg@mail.gmail.com>
 <1338223165484699842.726170sturla.molden-gmail.com@news.gmane.org>
 <CANGhFjZ6HKb=hLtLiUFot6jVkt6rKGxXoK4yLWg_8hnbJPrL4g@mail.gmail.com>
 <864303179484787737.086354sturla.molden-gmail.com@news.gmane.org>
 <CANGhFjYaO3c_LNtC2n1wmhn8TQ8MPp21ChoXRgjgz=B1f_pRSQ@mail.gmail.com>
Message-ID: <1163774665484873595.652105sturla.molden-gmail.com@news.gmane.org>

Feng Yu <rainwoodman at gmail.com> wrote:

> Also, did you checkout http://zeromq.org/blog:zero-copy ?
> ZeroMQ is a dependency of Jupyter, so it is quite available.

ZeroMQ is great, but it lacks some crucial features. In particular it does
not support IPC on Windows. Ideally one should e.g. use Unix doman sockets
on Linux and named pipes on Windows. Most MPI implementations seems to
prefer shared memory over these mechanisms, though. Also I am not sure
about ZeroMQ and asynch i/o. I would e.g. like to use IOCP on Windows, GCD
on Mac, and a threadpool plus epoll on Linux. 

Sturla



From phillip.m.feldman at gmail.com  Sat May 14 03:23:07 2016
From: phillip.m.feldman at gmail.com (Phillip Feldman)
Date: Sat, 14 May 2016 00:23:07 -0700
Subject: [Numpy-discussion] three-way comparisons
Message-ID: <CAB2ViTa2GnK7UF11frHHZ8RvxSgy_HyN-+p_vrhw9GJGg_pp2g@mail.gmail.com>

I often find a need to do the type of comparison done by function shown
below.  I suspect that this would be more efficient for large arrays if
implemented direction in C.  Is there any possibility of adding something
like this to NumPy?

def three_way(x, y):
   """
   This function performs a 3-way comparison on `x` and `y`, which must be
   either lists or arrays of compatible shape.  Each pair of items or
elements--
   let's call them x[i] and y[i]--are compared.  The corresponding element
in
   the output array is 1 if `x[i]` is greater then `y[i]`, -1 of `x[i]` is
less,
   and zero if the two are equal.
   """
   numpy.greater(y, x).astype(int) - numpy.less(y, x).astype(int)

Phillip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/b2a986dc/attachment.html>

From yellowhat46 at gmail.com  Sat May 14 09:27:14 2016
From: yellowhat46 at gmail.com (Vasco Gervasi)
Date: Sat, 14 May 2016 15:27:14 +0200
Subject: [Numpy-discussion] FFT and reconstruct
Message-ID: <CANtTwcFbJt-OrQCir-G4c0MfppnjCP2qv=GW15yWVjWbd7aHHA@mail.gmail.com>

Hi all,
I am trying to understand how FFT work, so I wrote the attached script.
The idea is to extract amplitude and phase of a signal and then reconstruct
using amplitude and phase information.
As you can see, I create some cosine curve on the interval t0-t1.
Let's start with t0=1.0 and t1=3.0 and consider just y['1']
= cos(1.0*omega*t), the signal is.
[image: Immagine incorporata 1]
The amplitude and phase for each order are:
[image: Immagine incorporata 2]
But if I try to reconstruct the signal using amplitude and phase:
[image: Immagine incorporata 3]
So as you can see there is a shifting of 180 deg.

Now let's consider another case, t0=2 and t1=3, the signal is
y['Signal'] = 1.0*cos(1.0*omega*t) + 2.0*cos(2.0*omega*t) +
3.0*cos(3.0*omega*t + pi/4) + 4.0*cos(4.0*omega*t) + 5.0*cos(5.0*omega*t) +
1.0
The reconstructed signal is very similar to the initial one:
[image: Immagine incorporata 4]
but is not exactly the same.
Any advice?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 138857 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 128252 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 160672 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 153579 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FFT.py
Type: application/octet-stream
Size: 3190 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/81e42b98/attachment.obj>

From josef.pktd at gmail.com  Sat May 14 11:03:29 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 14 May 2016 11:03:29 -0400
Subject: [Numpy-discussion] three-way comparisons
In-Reply-To: <CAB2ViTa2GnK7UF11frHHZ8RvxSgy_HyN-+p_vrhw9GJGg_pp2g@mail.gmail.com>
References: <CAB2ViTa2GnK7UF11frHHZ8RvxSgy_HyN-+p_vrhw9GJGg_pp2g@mail.gmail.com>
Message-ID: <CAMMTP+AL3xN5_AMr-7VTj6pHUkTqGB+Rwrkyfn62xpgJhrb8jQ@mail.gmail.com>

On Sat, May 14, 2016 at 3:23 AM, Phillip Feldman <
phillip.m.feldman at gmail.com> wrote:

> I often find a need to do the type of comparison done by function shown
> below.  I suspect that this would be more efficient for large arrays if
> implemented direction in C.  Is there any possibility of adding something
> like this to NumPy?
>
> def three_way(x, y):
>    """
>    This function performs a 3-way comparison on `x` and `y`, which must be
>    either lists or arrays of compatible shape.  Each pair of items or
> elements--
>    let's call them x[i] and y[i]--are compared.  The corresponding element
> in
>    the output array is 1 if `x[i]` is greater then `y[i]`, -1 of `x[i]` is
> less,
>    and zero if the two are equal.
>    """
>    numpy.greater(y, x).astype(int) - numpy.less(y, x).astype(int)
>


isn't that the same as sign( x- y) ?

Josef


>
> Phillip
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/109c126b/attachment.html>

From Permafacture at gmail.com  Sat May 14 14:25:44 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Sat, 14 May 2016 13:25:44 -0500
Subject: [Numpy-discussion] Why is this old bug still present?
In-Reply-To: <CAHedU1TTyE9m0DJnFQt+9mHrn302nPWzD8ZuAKvvmB2eGQJqjQ@mail.gmail.com>
References: <CAHedU1SXY+W_w=s70x+s_SLdX240+LB1KdbSdOCnWhNkZWp5aA@mail.gmail.com>
 <CAHedU1TTyE9m0DJnFQt+9mHrn302nPWzD8ZuAKvvmB2eGQJqjQ@mail.gmail.com>
Message-ID: <CAHedU1SGPmTbLeY0kXRcHM9AqGL+47rJ42Kzvq-UeWMhxRafgw@mail.gmail.com>

Sorry for the noob question.

On numpy 10.4.1,  I am bit by this:
https://github.com/numpy/numpy/issues/4185

But it has been fixed 6 months ago:
https://github.com/numpy/numpy/issues/6740

Do I need to compile numpy to get this fix on debian sid? Would anaconda be
up to date enough?

Elliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160514/0fdb0017/attachment.html>

From sean at mehta.io  Sun May 15 10:30:39 2016
From: sean at mehta.io (Sean Mehta)
Date: Sun, 15 May 2016 10:30:39 -0400
Subject: [Numpy-discussion] Why is this old bug still present?
In-Reply-To: <CAHedU1SGPmTbLeY0kXRcHM9AqGL+47rJ42Kzvq-UeWMhxRafgw@mail.gmail.com>
References: <CAHedU1SXY+W_w=s70x+s_SLdX240+LB1KdbSdOCnWhNkZWp5aA@mail.gmail.com>
 <CAHedU1TTyE9m0DJnFQt+9mHrn302nPWzD8ZuAKvvmB2eGQJqjQ@mail.gmail.com>
 <CAHedU1SGPmTbLeY0kXRcHM9AqGL+47rJ42Kzvq-UeWMhxRafgw@mail.gmail.com>
Message-ID: <1463322639.3418552.608328433.5AA59FD4@webmail.messagingengine.com>

On Sat, May 14, 2016, at 14:25, Elliot Hallmark wrote:
> On numpy 10.4.1,? I am bit by this: https://github.com/numpy/numpy/issues/4185
?
I assume you mean 1.10.4?
?
> But it has been fixed 6 months ago: https://github.com/numpy/numpy/issues/6740
> Do I need to compile numpy to get this fix on debian sid? Would anaconda be up to date enough?
?
This fix was included in 1.11.0. My Anaconda installation hasn't updated
to 1.11.0 yet, although their package list [0] includes the 1.11.0
packages. Although I don't use Debian, a quick search in the Debian
unstable package list [1] seems to suggest that 1.11.0 is available.

I ended up building 1.11.0 manually for myself.

[0] http://repo.continuum.io/pkgs/free/linux-64/
[1]
https://packages.debian.org/search?keywords=numpy&searchon=names&suite=unstable&section=all
?


From Permafacture at gmail.com  Sun May 15 21:42:58 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Sun, 15 May 2016 20:42:58 -0500
Subject: [Numpy-discussion] Why is this old bug still present?
In-Reply-To: <1463322639.3418552.608328433.5AA59FD4@webmail.messagingengine.com>
References: <CAHedU1SXY+W_w=s70x+s_SLdX240+LB1KdbSdOCnWhNkZWp5aA@mail.gmail.com>
 <CAHedU1TTyE9m0DJnFQt+9mHrn302nPWzD8ZuAKvvmB2eGQJqjQ@mail.gmail.com>
 <CAHedU1SGPmTbLeY0kXRcHM9AqGL+47rJ42Kzvq-UeWMhxRafgw@mail.gmail.com>
 <1463322639.3418552.608328433.5AA59FD4@webmail.messagingengine.com>
Message-ID: <CAHedU1QOLwuBMatuA0iE4SV9UYoziz9kgL7aS-0qxzBqErB6QQ@mail.gmail.com>

Thanks! Man, I got every one of those numbers in the wrong place...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/9397c80b/attachment.html>

From rainwoodman at gmail.com  Mon May 16 02:46:18 2016
From: rainwoodman at gmail.com (Feng Yu)
Date: Sun, 15 May 2016 23:46:18 -0700
Subject: [Numpy-discussion] FFT and reconstruct
In-Reply-To: <CANtTwcFbJt-OrQCir-G4c0MfppnjCP2qv=GW15yWVjWbd7aHHA@mail.gmail.com>
References: <CANtTwcFbJt-OrQCir-G4c0MfppnjCP2qv=GW15yWVjWbd7aHHA@mail.gmail.com>
Message-ID: <CANGhFjYiTYwN33PMbn6zkNESDw2FM+__UC_1EOOeq3kno4EbLg@mail.gmail.com>

Hi Vasco,

It looks slightly strange that you are using cos instead of exp in the
reconstruction of the signal.

I'd recommend you take a look at
http://docs.scipy.org/doc/numpy-1.10.1/reference/routines.fft.html

Also the documents of fftfreq, fftshift, and ifft.

Best,

- Yu

On Sat, May 14, 2016 at 6:27 AM, Vasco Gervasi <yellowhat46 at gmail.com>
wrote:

> Hi all,
> I am trying to understand how FFT work, so I wrote the attached script.
> The idea is to extract amplitude and phase of a signal and then
> reconstruct using amplitude and phase information.
> As you can see, I create some cosine curve on the interval t0-t1.
> Let's start with t0=1.0 and t1=3.0 and consider just y['1']
> = cos(1.0*omega*t), the signal is.
> [image: Immagine incorporata 1]
> The amplitude and phase for each order are:
> [image: Immagine incorporata 2]
> But if I try to reconstruct the signal using amplitude and phase:
> [image: Immagine incorporata 3]
> So as you can see there is a shifting of 180 deg.
>
> Now let's consider another case, t0=2 and t1=3, the signal is
> y['Signal'] = 1.0*cos(1.0*omega*t) + 2.0*cos(2.0*omega*t) +
> 3.0*cos(3.0*omega*t + pi/4) + 4.0*cos(4.0*omega*t) + 5.0*cos(5.0*omega*t) +
> 1.0
> The reconstructed signal is very similar to the initial one:
> [image: Immagine incorporata 4]
> but is not exactly the same.
> Any advice?
>
> Thanks
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/8c88a224/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 160672 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/8c88a224/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 153579 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/8c88a224/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 138857 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/8c88a224/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 128252 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160515/8c88a224/attachment-0003.png>

From mailinglists at xgm.de  Mon May 16 10:08:07 2016
From: mailinglists at xgm.de (Florian Lindner)
Date: Mon, 16 May 2016 16:08:07 +0200
Subject: [Numpy-discussion] Remove a random sample from array
Message-ID: <3366973.oDrVMeR9Kb@horus>

Hello,

I have an array of shape (n, 2) from which I want to extract a random sample 
of 20% of rows. The choosen samples should be removed the original array and 
moved to a new array of the same shape (n, 2).

What is the most clever way to do with numpy?

Thanks,
Florian


From Permafacture at gmail.com  Mon May 16 11:01:38 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Mon, 16 May 2016 10:01:38 -0500
Subject: [Numpy-discussion] Remove a random sample from array
In-Reply-To: <3366973.oDrVMeR9Kb@horus>
References: <3366973.oDrVMeR9Kb@horus>
Message-ID: <CAHedU1SbiLjGMFeLJrSUQuU=K80fcVto1=4g=1QuZjaE0xnr+A@mail.gmail.com>

What do you mean remove them from the array? Replace with zero or NaN?
On May 16, 2016 9:08 AM, "Florian Lindner" <mailinglists at xgm.de> wrote:

> Hello,
>
> I have an array of shape (n, 2) from which I want to extract a random
> sample
> of 20% of rows. The choosen samples should be removed the original array
> and
> moved to a new array of the same shape (n, 2).
>
> What is the most clever way to do with numpy?
>
> Thanks,
> Florian
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/39681290/attachment.html>

From mailinglists at xgm.de  Mon May 16 11:45:04 2016
From: mailinglists at xgm.de (Florian Lindner)
Date: Mon, 16 May 2016 17:45:04 +0200
Subject: [Numpy-discussion] Remove a random sample from array
In-Reply-To: <CAHedU1SbiLjGMFeLJrSUQuU=K80fcVto1=4g=1QuZjaE0xnr+A@mail.gmail.com>
References: <3366973.oDrVMeR9Kb@horus>
 <CAHedU1SbiLjGMFeLJrSUQuU=K80fcVto1=4g=1QuZjaE0xnr+A@mail.gmail.com>
Message-ID: <1689041.8OOJlUXQ1U@horus>

Am Montag, 16. Mai 2016, 10:01:38 CEST schrieb Elliot Hallmark:
> What do you mean remove them from the array? Replace with zero or NaN?

Removed like when 10 samples are taken from a (100, 2) array it becomes a (90, 
2) array. Copying the array is no problem, it removing inplace is not 
possible.

Best,
Florian
 
> On May 16, 2016 9:08 AM, "Florian Lindner" <mailinglists at xgm.de> wrote:
> > Hello,
> > 
> > I have an array of shape (n, 2) from which I want to extract a random
> > sample
> > of 20% of rows. The choosen samples should be removed the original array
> > and
> > moved to a new array of the same shape (n, 2).
> > 
> > What is the most clever way to do with numpy?
> > 
> > Thanks,
> > Florian
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion


From martin.noblia at openmailbox.org  Mon May 16 12:04:41 2016
From: martin.noblia at openmailbox.org (Martin Noblia)
Date: Mon, 16 May 2016 13:04:41 -0300
Subject: [Numpy-discussion] Remove a random sample from array
In-Reply-To: <3366973.oDrVMeR9Kb@horus>
References: <3366973.oDrVMeR9Kb@horus>
Message-ID: <5739EF99.2030304@openmailbox.org>

I think with `np.random.choice`

http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html

On 05/16/2016 11:08 AM, Florian Lindner wrote:
> Hello,
>
> I have an array of shape (n, 2) from which I want to extract a random sample
> of 20% of rows. The choosen samples should be removed the original array and
> moved to a new array of the same shape (n, 2).
>
> What is the most clever way to do with numpy?
>
> Thanks,
> Florian
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

-- 
*Martin Noblia*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/5bfd0a95/attachment.html>

From Permafacture at gmail.com  Mon May 16 12:24:55 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Mon, 16 May 2016 11:24:55 -0500
Subject: [Numpy-discussion] Remove a random sample from array
In-Reply-To: <5739EF99.2030304@openmailbox.org>
References: <3366973.oDrVMeR9Kb@horus>
	<5739EF99.2030304@openmailbox.org>
Message-ID: <CAHedU1QQojwmpZtYegrwFSPb0ZxBzF9rV+M0psqdV4jdtNWX4g@mail.gmail.com>

Use `random.shuffle(range(len(arr))` to make a list of indices.  Use a
slices to get your 20/80.  Convert to integer arrays and index your
original array with them.  Use sorted on the 80% list if you need to
preserve the order.

-Elliot

On Mon, May 16, 2016 at 11:04 AM, Martin Noblia <
martin.noblia at openmailbox.org> wrote:

> I think with `np.random.choice`
>
>
> http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html
>
>
> On 05/16/2016 11:08 AM, Florian Lindner wrote:
>
> Hello,
>
> I have an array of shape (n, 2) from which I want to extract a random sample
> of 20% of rows. The choosen samples should be removed the original array and
> moved to a new array of the same shape (n, 2).
>
> What is the most clever way to do with numpy?
>
> Thanks,
> Florian
> _______________________________________________
> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttps://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> --
> *Martin Noblia*
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/0f95f112/attachment.html>

From mdboom at gmail.com  Mon May 16 13:49:43 2016
From: mdboom at gmail.com (Michael Droettboom)
Date: Mon, 16 May 2016 17:49:43 +0000
Subject: [Numpy-discussion] Scipy John Hunter Plotting Contest: DEADLINE
	EXTENDED TO JUNE 3
Message-ID: <CAJd6-4-LM9LuZ8FB0UrwxtQvnU4XpmuDD2+5LHL0aD9kUug6xg@mail.gmail.com>

The Scipy John Hunter Excellence in Plotting Contest is a great opportunity
to showcase advancement in the start of the art of visualization.

Entries are still welcome, as the deadline has been extended to June 3,
2016.
Entry Instructions

   -

   Participants are invited to submit scientific plots to be judged by a
   panel.
   -

   Entries must be submitted by June 3, 2016 via e-mail to
   plotting-contest at scipy.org
   -

   Plots may be produced with any combination of Python-based tools. (It is
   not required that they use matplotlib, for example.)
   -

   Source code for the plot must be provided, in the form of Python code
   and/or IPython notebook, along with a rendering of the plot in PDF format.
   If the original data can not be shared for reasons of size or licensing,
   ?fake? data may be substituted, along with an image of the plot using real
   data.
   -

   Each entry must include a 300-500 word abstract describing the plot and
   its scientific importance for a general scientific audience.
   -

   Entries will be judged on their clarity, innovation and aesthetics, but
   most importantly for their effectiveness in illuminating real scientific
   work. Entrants are encouraged to submit plots that were used during the
   course of research, rather than merely being hypothetical.
   -

   SciPy reserves the right to display any and all entries, whether
   prize-winning or not, at the conference, use in any materials or on its
   website, with attribution to the original author(s).

Michael Droettboom, chair
Jacob Vanderplas
Phil Elson
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/59683114/attachment.html>

From yellowhat46 at gmail.com  Mon May 16 15:27:27 2016
From: yellowhat46 at gmail.com (Vasco Gervasi)
Date: Mon, 16 May 2016 21:27:27 +0200
Subject: [Numpy-discussion] FFT and reconstruct
In-Reply-To: <CANGhFjYiTYwN33PMbn6zkNESDw2FM+__UC_1EOOeq3kno4EbLg@mail.gmail.com>
References: <CANtTwcFbJt-OrQCir-G4c0MfppnjCP2qv=GW15yWVjWbd7aHHA@mail.gmail.com>
 <CANGhFjYiTYwN33PMbn6zkNESDw2FM+__UC_1EOOeq3kno4EbLg@mail.gmail.com>
Message-ID: <CANtTwcFbYeu930+YxfQCTD_bCOp_gFc-GGPLUSByh=wFDmL8Ag@mail.gmail.com>

I would like to use cos, to reconstruct the signal, to verify phase angle.
I will try ifft.

Thanks,

2016-05-16 8:46 GMT+02:00 Feng Yu <rainwoodman at gmail.com>:

> Hi Vasco,
>
> It looks slightly strange that you are using cos instead of exp in the
> reconstruction of the signal.
>
> I'd recommend you take a look at
> http://docs.scipy.org/doc/numpy-1.10.1/reference/routines.fft.html
>
> Also the documents of fftfreq, fftshift, and ifft.
>
> Best,
>
> - Yu
>
> On Sat, May 14, 2016 at 6:27 AM, Vasco Gervasi <yellowhat46 at gmail.com>
> wrote:
>
>> Hi all,
>> I am trying to understand how FFT work, so I wrote the attached script.
>> The idea is to extract amplitude and phase of a signal and then
>> reconstruct using amplitude and phase information.
>> As you can see, I create some cosine curve on the interval t0-t1.
>> Let's start with t0=1.0 and t1=3.0 and consider just y['1']
>> = cos(1.0*omega*t), the signal is.
>> [image: Immagine incorporata 1]
>> The amplitude and phase for each order are:
>> [image: Immagine incorporata 2]
>> But if I try to reconstruct the signal using amplitude and phase:
>> [image: Immagine incorporata 3]
>> So as you can see there is a shifting of 180 deg.
>>
>> Now let's consider another case, t0=2 and t1=3, the signal is
>> y['Signal'] = 1.0*cos(1.0*omega*t) + 2.0*cos(2.0*omega*t) +
>> 3.0*cos(3.0*omega*t + pi/4) + 4.0*cos(4.0*omega*t) + 5.0*cos(5.0*omega*t) +
>> 1.0
>> The reconstructed signal is very similar to the initial one:
>> [image: Immagine incorporata 4]
>> but is not exactly the same.
>> Any advice?
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/dac1eb49/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 160672 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/dac1eb49/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 138857 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/dac1eb49/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 128252 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/dac1eb49/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 153579 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/dac1eb49/attachment-0003.png>

From josef.pktd at gmail.com  Mon May 16 23:00:07 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 16 May 2016 23:00:07 -0400
Subject: [Numpy-discussion] Remove a random sample from array
In-Reply-To: <CAHedU1QQojwmpZtYegrwFSPb0ZxBzF9rV+M0psqdV4jdtNWX4g@mail.gmail.com>
References: <3366973.oDrVMeR9Kb@horus> <5739EF99.2030304@openmailbox.org>
 <CAHedU1QQojwmpZtYegrwFSPb0ZxBzF9rV+M0psqdV4jdtNWX4g@mail.gmail.com>
Message-ID: <CAMMTP+D=HLs7KL_FX=VPJ5fi4Q+A3ZexmqHdW620-N=O5puTrQ@mail.gmail.com>

On Mon, May 16, 2016 at 12:24 PM, Elliot Hallmark <Permafacture at gmail.com>
wrote:

> Use `random.shuffle(range(len(arr))` to make a list of indices.  Use a
> slices to get your 20/80.  Convert to integer arrays and index your
> original array with them.  Use sorted on the 80% list if you need to
> preserve the order.
>

similar but simpler

You can just random permute/shuffle an array with 20% ones (True) and 80%
zeros (False) and use it as a mask to select from the original array.

Josef



>
>
> -Elliot
>
> On Mon, May 16, 2016 at 11:04 AM, Martin Noblia <
> martin.noblia at openmailbox.org> wrote:
>
>> I think with `np.random.choice`
>>
>>
>> http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html
>>
>>
>> On 05/16/2016 11:08 AM, Florian Lindner wrote:
>>
>> Hello,
>>
>> I have an array of shape (n, 2) from which I want to extract a random sample
>> of 20% of rows. The choosen samples should be removed the original array and
>> moved to a new array of the same shape (n, 2).
>>
>> What is the most clever way to do with numpy?
>>
>> Thanks,
>> Florian
>> _______________________________________________
>> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttps://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>> --
>> *Martin Noblia*
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/034f6cd2/attachment.html>

From shoyer at gmail.com  Mon May 16 23:54:29 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 16 May 2016 20:54:29 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
Message-ID: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>

I have recently encountered several use cases for randomly generate random
number seeds:

1. When writing a library of stochastic functions that take a seed as an
input argument, and some of these functions call multiple other such
stochastic functions. Dask is one such example [1].

2. When a library needs to produce results that are reproducible after
calling numpy.random.seed, but that do not want to use the functions in
numpy.random directly. This came up recently in a pandas pull request [2],
because we want to allow using RandomState objects as an alternative to
global state in numpy.random. A major advantage of this approach is that it
provides an obvious alternative to reusing the private numpy.random._mtrand
[3].

The implementation of this function (and the corresponding method on
RandomState) is almost trivial, and I've already written such a utility for
my code:

def random_seed():
    # numpy.random uses uint32 seeds
    np.random.randint(2 ** 32)

The advantage of adding a new method is that it avoids the need for
explanation by making the intent of code using this pattern obvious. So I
think it is a good candidate for inclusion in numpy.random.

Any opinions?

[1]
https://github.com/dask/dask/blob/e0b246221957c4bd618e57246f3a7ccc8863c494/dask/utils.py#L336
[2] https://github.com/pydata/pandas/pull/13161
[3] On a side note, if there's no longer a good reason to keep this object
private, perhaps we should expose it in our public API. It would certainly
be useful -- scikit-learn is already using it (see links in the pandas PR
above).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/da14c262/attachment.html>

From shoyer at gmail.com  Tue May 17 00:32:35 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 16 May 2016 21:32:35 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
Message-ID: <CAEQ_TvftfEFEE9=RJxLJshk1nU=3pTSAiOg=DCNTE9D-vR1r1Q@mail.gmail.com>

Looking at the dask helper function again reminds me of an important cavaet
to this approach, which was pointed out to me by Clark Fitzgerald.

If you generate a moderately large number of random seeds in this fashion,
you are quite likely to have collisions due to the Birthday Paradox. For
example, you have a 50% chance of encountering at least one collision if
you generate only 77,000 seeds:
https://en.wikipedia.org/wiki/Birthday_attack

The docstring for this function should document this limitation of the
approach, which is still appropriate for a small number of seeds. Our
implementation can also encourage creating these seeds in a single
vectorized call to random_seed, which can significantly reduce the
likelihood of collisions between seeds generated in a single call to
random_seed with something like the following:

def random_seed(size):
    base = np.random.randint(2 ** 32)
    offset = np.arange(size)
    return (base + offset) % (2 ** 32)

In principle, I believe this could generate the full 2 ** 32 unique seeds
without any collisions.

Cryptography experts, please speak up if I'm mistaken here.

On Mon, May 16, 2016 at 8:54 PM, Stephan Hoyer <shoyer at gmail.com> wrote:

> I have recently encountered several use cases for randomly generate random
> number seeds:
>
> 1. When writing a library of stochastic functions that take a seed as an
> input argument, and some of these functions call multiple other such
> stochastic functions. Dask is one such example [1].
>
> 2. When a library needs to produce results that are reproducible after
> calling numpy.random.seed, but that do not want to use the functions in
> numpy.random directly. This came up recently in a pandas pull request [2],
> because we want to allow using RandomState objects as an alternative to
> global state in numpy.random. A major advantage of this approach is that it
> provides an obvious alternative to reusing the private numpy.random._mtrand
> [3].
>
> The implementation of this function (and the corresponding method on
> RandomState) is almost trivial, and I've already written such a utility for
> my code:
>
> def random_seed():
>     # numpy.random uses uint32 seeds
>     np.random.randint(2 ** 32)
>
> The advantage of adding a new method is that it avoids the need for
> explanation by making the intent of code using this pattern obvious. So I
> think it is a good candidate for inclusion in numpy.random.
>
> Any opinions?
>
> [1]
> https://github.com/dask/dask/blob/e0b246221957c4bd618e57246f3a7ccc8863c494/dask/utils.py#L336
> [2] https://github.com/pydata/pandas/pull/13161
> [3] On a side note, if there's no longer a good reason to keep this object
> private, perhaps we should expose it in our public API. It would certainly
> be useful -- scikit-learn is already using it (see links in the pandas PR
> above).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160516/a959ea29/attachment.html>

From robert.kern at gmail.com  Tue May 17 03:18:52 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 May 2016 08:18:52 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
Message-ID: <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>

On Tue, May 17, 2016 at 4:54 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
>
> I have recently encountered several use cases for randomly generate
random number seeds:
>
> 1. When writing a library of stochastic functions that take a seed as an
input argument, and some of these functions call multiple other such
stochastic functions. Dask is one such example [1].

Can you clarify the use case here? I don't really know what you are doing
here, but I'm pretty sure this is not the right approach.

> 2. When a library needs to produce results that are reproducible after
calling numpy.random.seed, but that do not want to use the functions in
numpy.random directly. This came up recently in a pandas pull request [2],
because we want to allow using RandomState objects as an alternative to
global state in numpy.random. A major advantage of this approach is that it
provides an obvious alternative to reusing the private numpy.random._mtrand
[3].

It's only pseudo-private. This is an authorized use of it.

However, for this case, I usually just pass around the the numpy.random
module itself and let duck-typing take care of the rest.

> [3] On a side note, if there's no longer a good reason to keep this
object private, perhaps we should expose it in our public API. It would
certainly be useful -- scikit-learn is already using it (see links in the
pandas PR above).

Adding a public get_global_random_state() function might be in order.
Originally, I wanted there to be *some* barrier to entry, but just grabbing
it to use as a default RandomState object is definitely an intended use of
it. It's not going to disappear.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/573e88b2/attachment.html>

From shoyer at gmail.com  Tue May 17 04:09:28 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Tue, 17 May 2016 01:09:28 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
Message-ID: <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>

On Tue, May 17, 2016 at 12:18 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, May 17, 2016 at 4:54 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
> > 1. When writing a library of stochastic functions that take a seed as an
> input argument, and some of these functions call multiple other such
> stochastic functions. Dask is one such example [1].
>
> Can you clarify the use case here? I don't really know what you are doing
> here, but I'm pretty sure this is not the right approach.
>

Here's a contrived example. Suppose I've written a simulator for cars that
consists of a number of loosely connected components (e.g., an engine,
brakes, etc.). The behavior of each component of our simulator is
stochastic, but we want everything to be fully reproducible, so we need to
use seeds or RandomState objects.

We might write our simulate_car function like the following:

def simulate_car(engine_config, brakes_config, seed=None):
    rs = np.random.RandomState(seed)
    engine = simulate_engine(engine_config, seed=rs.random_seed())
    brakes = simulate_brakes(brakes_config, seed=rs.random_seed())
    ...

The problem with passing the same RandomState object (either explicitly or
dropping the seed argument entirely and using the  global state) to both
simulate_engine and simulate_breaks is that it breaks encapsulation -- if I
change what I do inside simulate_engine, it also effects the brakes.

The dask use case is actually pretty different -- the intent is to create
many random numbers in parallel using multiple threads or processes
(possibly in a distributed fashion). I know that skipping ahead is the
standard way to get independent number streams for parallel sampling, but
that isn't exposed in numpy.random, and setting distinct seeds seems like a
reasonable alternative for scientific computing use cases.


> It's only pseudo-private. This is an authorized use of it.
>
> However, for this case, I usually just pass around the the numpy.random
> module itself and let duck-typing take care of the rest.
>

I like the duck-typing approach. That's very elegant.

If this is an authorized use of the global RandomState object, let's
document it! Otherwise cautious library maintainers like myself will
discourage using it :).


> > [3] On a side note, if there's no longer a good reason to keep this
> object private, perhaps we should expose it in our public API. It would
> certainly be useful -- scikit-learn is already using it (see links in the
> pandas PR above).
>
> Adding a public get_global_random_state() function might be in order.
>

Yes, possibly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/9fdd354f/attachment.html>

From robert.kern at gmail.com  Tue May 17 04:49:45 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 May 2016 09:49:45 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
Message-ID: <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>

On Tue, May 17, 2016 at 9:09 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
>
> On Tue, May 17, 2016 at 12:18 AM, Robert Kern <robert.kern at gmail.com>
wrote:
>>
>> On Tue, May 17, 2016 at 4:54 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
>> > 1. When writing a library of stochastic functions that take a seed as
an input argument, and some of these functions call multiple other such
stochastic functions. Dask is one such example [1].
>>
>> Can you clarify the use case here? I don't really know what you are
doing here, but I'm pretty sure this is not the right approach.
>
> Here's a contrived example. Suppose I've written a simulator for cars
that consists of a number of loosely connected components (e.g., an engine,
brakes, etc.). The behavior of each component of our simulator is
stochastic, but we want everything to be fully reproducible, so we need to
use seeds or RandomState objects.
>
> We might write our simulate_car function like the following:
>
> def simulate_car(engine_config, brakes_config, seed=None):
>     rs = np.random.RandomState(seed)
>     engine = simulate_engine(engine_config, seed=rs.random_seed())
>     brakes = simulate_brakes(brakes_config, seed=rs.random_seed())
>     ...
>
> The problem with passing the same RandomState object (either explicitly
or dropping the seed argument entirely and using the  global state) to both
simulate_engine and simulate_breaks is that it breaks encapsulation -- if I
change what I do inside simulate_engine, it also effects the brakes.

That's a little too contrived, IMO. In most such simulations, the different
components interact with each other in the normal course of the simulation;
that's why they are both joined together in the same simulation instead of
being two separate runs. Unless if the components are being run across a
process or thread boundary (a la dask below) where true nondeterminism
comes into play, then I don't think you want these semi-independent
streams. This seems to be the advice du jour from the agent-based modeling
community.

> The dask use case is actually pretty different -- the intent is to create
many random numbers in parallel using multiple threads or processes
(possibly in a distributed fashion). I know that skipping ahead is the
standard way to get independent number streams for parallel sampling, but
that isn't exposed in numpy.random, and setting distinct seeds seems like a
reasonable alternative for scientific computing use cases.

Forget about integer seeds. Those are for human convenience. If you're not
jotting them down in your lab notebook in pen, you don't want an integer
seed.

What you want is a function that returns many RandomState objects that are
hopefully spread around the MT19937 space enough that they are essentially
independent (in the absence of true jumpahead). The better implementation
of such a function would look something like this:

def spread_out_prngs(n, root_prng=None):
    if root_prng is None:
        root_prng = np.random
    elif not isinstance(root_prng, np.random.RandomState):
        root_prng = np.random.RandomState(root_prng)
    sprouted_prngs = []
    for i in range(n):
        seed_array = root_prng.randint(1<<32, size=624)  # dtype=np.uint32
under 1.11
        sprouted_prngs.append(np.random.RandomState(seed_array))
    return spourted_prngs

Internally, this generates seed arrays of about the size of the MT19937
state so make sure that you can access more of the state space. That will
at least make the chance of collision tiny. And it can be easily rewritten
to take advantage of one of the newer PRNGs that have true independent
streams:

  https://github.com/bashtage/ng-numpy-randomstate

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/4a37ecb4/attachment.html>

From matej.tyc at gmail.com  Tue May 17 05:04:07 2016
From: matej.tyc at gmail.com (=?UTF-8?B?TWF0xJtqIFTDvcSN?=)
Date: Tue, 17 May 2016 11:04:07 +0200
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
References: <CAD_1ysid2nWSWAOMVEyobWcG-n1kNJoPJBVwPR7w-ZHwWCQaXg@mail.gmail.com>
 <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
Message-ID: <9de8219e-79de-a474-79df-647921f3344a@gmail.com>

On 11.5.2016 10:29, Sturla Molden wrote:
> I did some work on this some years ago. ...
>
I am sorry, I have missed this discussion when it started.

There are two cases when I had feeling that I had to use this functionality:

 - Parallel processing of HUGE data, and

 - using parallel processing in an application that had plug-ins which
operated on one shared array (that was updated every one and then - it
was a producer-consumer pattern thing). As everything got set up, it
worked like a charm.

The thing I especially like about the proposed module is the lack of
external dependencies + it works if one knows how to use it.

The bad thing about it is its fragility - I admit that using it as it is
is not particularly intuitive. Unlike Sturla, I think that this is not a
dead end, but it indeed feels clumsy. However, I dislike the necessity
of writing Cython or C to get true multithreading for reasons I have
mentioned - what if you want to run high-level Python functions in parallel?

So, what I would really like to see is some kind of numpy documentation
on how to approach parallel computing with numpy arrays (depending on
what kind of task one wants to achieve). Maybe just using the queue is
good enough, or there are those 3-rd party modules with known
limitations? Plenty of people start off with numpy, so some kind of
overview should be part of numpy docs.




From sturla.molden at gmail.com  Tue May 17 08:13:42 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 17 May 2016 12:13:42 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <9de8219e-79de-a474-79df-647921f3344a@gmail.com>
Message-ID: <1502440745485178811.176557sturla.molden-gmail.com@news.gmane.org>

Mat?j T?? <matej.tyc at gmail.com> wrote:

>  - Parallel processing of HUGE data, and

This is mainly a Windows problem, as copy-on-write fork() will solve this
on any other platform. I am more in favor of asking  Microsoft to fix their
broken OS. 

Also observe that the usefulness of shared memory is very limited on
Windows, as we in practice never get the same base address in a spawned
process. This prevents sharing data structures with pointers and Python
objects. Anything more complex than an array cannot be shared.

What this means is that shared memory is seldom useful for sharing huge
data, even on Windows. It is only useful for this on Unix/Linux, where base
addresses can stay they same. But on non-Windows platforms, the COW will in
99.99% of the cases be sufficient, thus make shared memory superfluous
anyway. We don't need shared memory to scatter large data on Linux, only
fork.

As I see it. shared memory is mostly useful as a means to construct an
inter-process communication (IPC) protocol. 

Sturla



From josef.pktd at gmail.com  Tue May 17 08:34:26 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 17 May 2016 08:34:26 -0400
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
Message-ID: <CAMMTP+AKvzpJLjasdxVSM==ZAepPwohbGwQg0c8hKvvdQpdrmQ@mail.gmail.com>

On Tue, May 17, 2016 at 4:49 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, May 17, 2016 at 9:09 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
> >
> > On Tue, May 17, 2016 at 12:18 AM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Tue, May 17, 2016 at 4:54 AM, Stephan Hoyer <shoyer at gmail.com>
> wrote:
> >> > 1. When writing a library of stochastic functions that take a seed as
> an input argument, and some of these functions call multiple other such
> stochastic functions. Dask is one such example [1].
> >>
> >> Can you clarify the use case here? I don't really know what you are
> doing here, but I'm pretty sure this is not the right approach.
> >
> > Here's a contrived example. Suppose I've written a simulator for cars
> that consists of a number of loosely connected components (e.g., an engine,
> brakes, etc.). The behavior of each component of our simulator is
> stochastic, but we want everything to be fully reproducible, so we need to
> use seeds or RandomState objects.
> >
> > We might write our simulate_car function like the following:
> >
> > def simulate_car(engine_config, brakes_config, seed=None):
> >     rs = np.random.RandomState(seed)
> >     engine = simulate_engine(engine_config, seed=rs.random_seed())
> >     brakes = simulate_brakes(brakes_config, seed=rs.random_seed())
> >     ...
> >
> > The problem with passing the same RandomState object (either explicitly
> or dropping the seed argument entirely and using the  global state) to both
> simulate_engine and simulate_breaks is that it breaks encapsulation -- if I
> change what I do inside simulate_engine, it also effects the brakes.
>
> That's a little too contrived, IMO. In most such simulations, the
> different components interact with each other in the normal course of the
> simulation; that's why they are both joined together in the same simulation
> instead of being two separate runs. Unless if the components are being run
> across a process or thread boundary (a la dask below) where true
> nondeterminism comes into play, then I don't think you want these
> semi-independent streams. This seems to be the advice du jour from the
> agent-based modeling community.
>


similar usecase where I had to switch to using several RandomStates

In a Monte Carlo experiment with increasing sample size, I want two random
variables, x, y, to have the same the same draws in the common initial
observations.

If I draw x and y sequentially, and then increase the number of
observations for the simulation, then it completely changes the draws for
second variable if they use a common RandomState.

With separate random states, increasing from 1000 to 1200 observations,
leaves the first 1000 draws unchanged.
(This reduces the Monte Carlo noise for example when calculating the power
of a hypothesis test as function of the sample size.)

Josef


>
>
> > The dask use case is actually pretty different -- the intent is to
> create many random numbers in parallel using multiple threads or processes
> (possibly in a distributed fashion). I know that skipping ahead is the
> standard way to get independent number streams for parallel sampling, but
> that isn't exposed in numpy.random, and setting distinct seeds seems like a
> reasonable alternative for scientific computing use cases.
>
> Forget about integer seeds. Those are for human convenience. If you're not
> jotting them down in your lab notebook in pen, you don't want an integer
> seed.
>
> What you want is a function that returns many RandomState objects that are
> hopefully spread around the MT19937 space enough that they are essentially
> independent (in the absence of true jumpahead). The better implementation
> of such a function would look something like this:
>
> def spread_out_prngs(n, root_prng=None):
>     if root_prng is None:
>         root_prng = np.random
>     elif not isinstance(root_prng, np.random.RandomState):
>         root_prng = np.random.RandomState(root_prng)
>     sprouted_prngs = []
>     for i in range(n):
>         seed_array = root_prng.randint(1<<32, size=624)  # dtype=np.uint32
> under 1.11
>         sprouted_prngs.append(np.random.RandomState(seed_array))
>     return spourted_prngs
>
> Internally, this generates seed arrays of about the size of the MT19937
> state so make sure that you can access more of the state space. That will
> at least make the chance of collision tiny. And it can be easily rewritten
> to take advantage of one of the newer PRNGs that have true independent
> streams:
>
>   https://github.com/bashtage/ng-numpy-randomstate
>
> --
> Robert Kern
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/f70f7334/attachment.html>

From sturla.molden at gmail.com  Tue May 17 09:40:56 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 17 May 2016 13:40:56 +0000 (UTC)
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
Message-ID: <1591394023485184927.488471sturla.molden-gmail.com@news.gmane.org>

Stephan Hoyer <shoyer at gmail.com> wrote:
> I have recently encountered several use cases for randomly generate random
> number seeds:
> 
> 1. When writing a library of stochastic functions that take a seed as an
> input argument, and some of these functions call multiple other such
> stochastic functions. Dask is one such example [1].
> 
> 2. When a library needs to produce results that are reproducible after
> calling numpy.random.seed, but that do not want to use the functions in
> numpy.random directly. This came up recently in a pandas pull request [2],
> because we want to allow using RandomState objects as an alternative to
> global state in numpy.random. A major advantage of this approach is that it
> provides an obvious alternative to reusing the private numpy.random._mtrand
> [3].


What about making numpy.random a finite state machine, and keeping a stack
of RandomState seeds? That is, something similar to what OpenGL does for
its matrices? Then we get two functions, numpy.random.push_seed and
numpy.random.pop_seed.


Sturla



From robert.kern at gmail.com  Tue May 17 09:48:45 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 May 2016 14:48:45 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <1591394023485184927.488471sturla.molden-gmail.com@news.gmane.org>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <1591394023485184927.488471sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAF6FJivHm85EEZTaxbuOAfQOOG0WfT+RJy8YjRv4+u+kgcJpcA@mail.gmail.com>

On Tue, May 17, 2016 at 2:40 PM, Sturla Molden <sturla.molden at gmail.com>
wrote:
>
> Stephan Hoyer <shoyer at gmail.com> wrote:
> > I have recently encountered several use cases for randomly generate
random
> > number seeds:
> >
> > 1. When writing a library of stochastic functions that take a seed as an
> > input argument, and some of these functions call multiple other such
> > stochastic functions. Dask is one such example [1].
> >
> > 2. When a library needs to produce results that are reproducible after
> > calling numpy.random.seed, but that do not want to use the functions in
> > numpy.random directly. This came up recently in a pandas pull request
[2],
> > because we want to allow using RandomState objects as an alternative to
> > global state in numpy.random. A major advantage of this approach is
that it
> > provides an obvious alternative to reusing the private
numpy.random._mtrand
> > [3].
>
> What about making numpy.random a finite state machine, and keeping a stack
> of RandomState seeds? That is, something similar to what OpenGL does for
> its matrices? Then we get two functions, numpy.random.push_seed and
> numpy.random.pop_seed.

I don't think that addresses the issues brought up here. It's just more
global state to worry about.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/82067830/attachment.html>

From ewm at redtetrahedron.org  Tue May 17 09:50:41 2016
From: ewm at redtetrahedron.org (Eric Moore)
Date: Tue, 17 May 2016 09:50:41 -0400
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <1591394023485184927.488471sturla.molden-gmail.com@news.gmane.org>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <1591394023485184927.488471sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAGeA38nTKni+UEJ3urTYptg-BQOG9jKFyVEZV-RJ4ocUAR1q3A@mail.gmail.com>

On Tue, May 17, 2016 at 9:40 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> Stephan Hoyer <shoyer at gmail.com> wrote:
> > I have recently encountered several use cases for randomly generate
> random
> > number seeds:
> >
> > 1. When writing a library of stochastic functions that take a seed as an
> > input argument, and some of these functions call multiple other such
> > stochastic functions. Dask is one such example [1].
> >
> > 2. When a library needs to produce results that are reproducible after
> > calling numpy.random.seed, but that do not want to use the functions in
> > numpy.random directly. This came up recently in a pandas pull request
> [2],
> > because we want to allow using RandomState objects as an alternative to
> > global state in numpy.random. A major advantage of this approach is that
> it
> > provides an obvious alternative to reusing the private
> numpy.random._mtrand
> > [3].
>
>
> What about making numpy.random a finite state machine, and keeping a stack
> of RandomState seeds? That is, something similar to what OpenGL does for
> its matrices? Then we get two functions, numpy.random.push_seed and
> numpy.random.pop_seed.
>

I don't like the idea of adding this kind of internal state.  Having it
built into the module means that it is shared by all callers, libraries
user code etc. That's not the right choice when a stack of seeds could be
easily built around the RandomState object if that is really what someone
needs.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/3a8c8106/attachment.html>

From njs at pobox.com  Tue May 17 13:24:05 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 17 May 2016 10:24:05 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
Message-ID: <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>

On May 17, 2016 1:50 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
>
[...]
> What you want is a function that returns many RandomState objects that
are hopefully spread around the MT19937 space enough that they are
essentially independent (in the absence of true jumpahead). The better
implementation of such a function would look something like this:
>
> def spread_out_prngs(n, root_prng=None):
>     if root_prng is None:
>         root_prng = np.random
>     elif not isinstance(root_prng, np.random.RandomState):
>         root_prng = np.random.RandomState(root_prng)
>     sprouted_prngs = []
>     for i in range(n):
>         seed_array = root_prng.randint(1<<32, size=624)  #
dtype=np.uint32 under 1.11
>         sprouted_prngs.append(np.random.RandomState(seed_array))
>     return spourted_prngs

Maybe a nice way to encapsulate this in the RandomState interface would be
a method RandomState.random_state() that generates and returns a new child
RandomState.

> Internally, this generates seed arrays of about the size of the MT19937
state so make sure that you can access more of the state space. That will
at least make the chance of collision tiny. And it can be easily rewritten
to take advantage of one of the newer PRNGs that have true independent
streams:
>
>   https://github.com/bashtage/ng-numpy-randomstate

... But unfortunately I'm not sure how to make my interface suggestion
above work on top of one of these RNGs, because for
RandomState.random_state you really want a tree of independent RNGs and the
fancy new PRNGs only provide a single flat namespace :-/. And even more
annoyingly, the tree API is actually a nicer API, because with a flat
namespace you have to know up front about all possible RNGs your code will
use, which is an unfortunate global coupling that makes it difficult to
compose programs out of independent pieces, while the
RandomState.random_state approach composes beautifully. Maybe there's some
clever way to allocate a 64-bit namespace to make it look tree-like? I'm
not sure 64 bits is really enough...

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/cc57fd75/attachment.html>

From robert.kern at gmail.com  Tue May 17 13:41:31 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 May 2016 18:41:31 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
Message-ID: <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>

On Tue, May 17, 2016 at 6:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
> On May 17, 2016 1:50 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
> >
> [...]
> > What you want is a function that returns many RandomState objects that
are hopefully spread around the MT19937 space enough that they are
essentially independent (in the absence of true jumpahead). The better
implementation of such a function would look something like this:
> >
> > def spread_out_prngs(n, root_prng=None):
> >     if root_prng is None:
> >         root_prng = np.random
> >     elif not isinstance(root_prng, np.random.RandomState):
> >         root_prng = np.random.RandomState(root_prng)
> >     sprouted_prngs = []
> >     for i in range(n):
> >         seed_array = root_prng.randint(1<<32, size=624)  #
dtype=np.uint32 under 1.11
> >         sprouted_prngs.append(np.random.RandomState(seed_array))
> >     return spourted_prngs
>
> Maybe a nice way to encapsulate this in the RandomState interface would
be a method RandomState.random_state() that generates and returns a new
child RandomState.

I disagree. This is a workaround in the absence of proper jumpahead or
guaranteed-independent streams. I would not encourage it.

> > Internally, this generates seed arrays of about the size of the MT19937
state so make sure that you can access more of the state space. That will
at least make the chance of collision tiny. And it can be easily rewritten
to take advantage of one of the newer PRNGs that have true independent
streams:
> >
> >   https://github.com/bashtage/ng-numpy-randomstate
>
> ... But unfortunately I'm not sure how to make my interface suggestion
above work on top of one of these RNGs, because for
RandomState.random_state you really want a tree of independent RNGs and the
fancy new PRNGs only provide a single flat namespace :-/. And even more
annoyingly, the tree API is actually a nicer API, because with a flat
namespace you have to know up front about all possible RNGs your code will
use, which is an unfortunate global coupling that makes it difficult to
compose programs out of independent pieces, while the
RandomState.random_state approach composes beautifully. Maybe there's some
clever way to allocate a 64-bit namespace to make it look tree-like? I'm
not sure 64 bits is really enough...

MT19937 doesn't have a "tree" any more than the others. It's the same flat
state space. You are just getting the illusion of a tree by hoping that you
never collide. You ought to think about precisely the same global coupling
issues with MT19937 as you do with guaranteed-independent streams.
Hope-and-prayer isn't really a substitute for properly engineering your
problem. It's just a moral hazard to promote this method to the main API.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160517/a21d4c7f/attachment.html>

From matej.tyc at gmail.com  Tue May 17 16:49:20 2016
From: matej.tyc at gmail.com (=?UTF-8?B?TWF0xJtqIFTDvcSN?=)
Date: Tue, 17 May 2016 22:49:20 +0200
Subject: [Numpy-discussion] Numpy arrays shareable among related
 processes (PR #7533)
In-Reply-To: <1502440745485178811.176557sturla.molden-gmail.com@news.gmane.org>
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <9de8219e-79de-a474-79df-647921f3344a@gmail.com>
 <1502440745485178811.176557sturla.molden-gmail.com@news.gmane.org>
Message-ID: <00ae9046-1228-5209-0c14-984deb528643@gmail.com>

On 17.5.2016 14:13, Sturla Molden wrote:

> Mat?j T?? <matej.tyc at gmail.com> wrote:
>
>>  - Parallel processing of HUGE data, and
> This is mainly a Windows problem, as copy-on-write fork() will solve this
> on any other platform. ...
That sounds interesting, could you elaborate on it a bit? Does it mean
that if you pass the numpy array to the child process using Queue, no
significant amount of data will flow through it? Or I shouldn't pass it
using Queue at all and just rely on inheritance? Finally, I assume that
passing it as an argument to the Process class is the worst option,
because it will be pickled and unpickled.

Or maybe you refer to modules s.a. joblib that use this functionality
and expose only a nice interface?
And finally, cow means that returning large arrays still involves data
moving between processes, whereas the shm approach has the workaround
that you can preallocate the result array by the parent process, where
the worker process can write to.
> What this means is that shared memory is seldom useful for sharing huge
> data, even on Windows. It is only useful for this on Unix/Linux, where base
> addresses can stay they same. But on non-Windows platforms, the COW will in
> 99.99% of the cases be sufficient, thus make shared memory superfluous
> anyway. We don't need shared memory to scatter large data on Linux, only
> fork.
I am actually quite comfortable with sharing numpy arrays only. It is a
nice format for sharing large amounts of numbers, which is what I want
and what many modules accept as input (e.g. the "shapely" module).



From sturla.molden at gmail.com  Tue May 17 17:03:04 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 17 May 2016 21:03:04 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <1624973999484645450.688294sturla.molden-gmail.com@news.gmane.org>
 <9de8219e-79de-a474-79df-647921f3344a@gmail.com>
 <1502440745485178811.176557sturla.molden-gmail.com@news.gmane.org>
 <00ae9046-1228-5209-0c14-984deb528643@gmail.com>
Message-ID: <1126141486485211291.448744sturla.molden-gmail.com@news.gmane.org>

Mat?j T?? <matej.tyc at gmail.com> wrote:

> Does it mean
> that if you pass the numpy array to the child process using Queue, no
> significant amount of data will flow through it? 

This is what my shared memory arrayes do.

> Or I shouldn't pass it
> using Queue at all and just rely on inheritance? 

This is what David Baddeley's shared memory arrays do.

> Finally, I assume that
> passing it as an argument to the Process class is the worst option,
> because it will be pickled and unpickled.
 
My shared memory arrays only pickles the metadata, and can be used in this
way.

> Or maybe you refer to modules s.a. joblib that use this functionality
> and expose only a nice interface?

Joblib creates "share memory" by memory mapping a temporary file, which is
back by RAM on Libux (tempfs). It is backed by a physical file on disk on
Mac and Windows. In this resepect, joblib is much better on Linux than Mac
or Windows.


> And finally, cow means that returning large arrays still involves data
> moving between processes, whereas the shm approach has the workaround
> that you can preallocate the result array by the parent process, where
> the worker process can write to.

My shared memory arrays need no workaround dor this. They also allow shared
memory arrays to be returned to the parent process. No preallocation is
needed.


Sturla



From njs at pobox.com  Tue May 17 20:14:33 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 17 May 2016 17:14:33 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
Message-ID: <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>

On Tue, May 17, 2016 at 10:41 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, May 17, 2016 at 6:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On May 17, 2016 1:50 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
>> >
>> [...]
>> > What you want is a function that returns many RandomState objects that
>> > are hopefully spread around the MT19937 space enough that they are
>> > essentially independent (in the absence of true jumpahead). The better
>> > implementation of such a function would look something like this:
>> >
>> > def spread_out_prngs(n, root_prng=None):
>> >     if root_prng is None:
>> >         root_prng = np.random
>> >     elif not isinstance(root_prng, np.random.RandomState):
>> >         root_prng = np.random.RandomState(root_prng)
>> >     sprouted_prngs = []
>> >     for i in range(n):
>> >         seed_array = root_prng.randint(1<<32, size=624)  #
>> > dtype=np.uint32 under 1.11
>> >         sprouted_prngs.append(np.random.RandomState(seed_array))
>> >     return spourted_prngs
>>
>> Maybe a nice way to encapsulate this in the RandomState interface would be
>> a method RandomState.random_state() that generates and returns a new child
>> RandomState.
>
> I disagree. This is a workaround in the absence of proper jumpahead or
> guaranteed-independent streams. I would not encourage it.
>
>> > Internally, this generates seed arrays of about the size of the MT19937
>> > state so make sure that you can access more of the state space. That will at
>> > least make the chance of collision tiny. And it can be easily rewritten to
>> > take advantage of one of the newer PRNGs that have true independent streams:
>> >
>> >   https://github.com/bashtage/ng-numpy-randomstate
>>
>> ... But unfortunately I'm not sure how to make my interface suggestion
>> above work on top of one of these RNGs, because for RandomState.random_state
>> you really want a tree of independent RNGs and the fancy new PRNGs only
>> provide a single flat namespace :-/. And even more annoyingly, the tree API
>> is actually a nicer API, because with a flat namespace you have to know up
>> front about all possible RNGs your code will use, which is an unfortunate
>> global coupling that makes it difficult to compose programs out of
>> independent pieces, while the RandomState.random_state approach composes
>> beautifully. Maybe there's some clever way to allocate a 64-bit namespace to
>> make it look tree-like? I'm not sure 64 bits is really enough...
>
> MT19937 doesn't have a "tree" any more than the others. It's the same flat
> state space. You are just getting the illusion of a tree by hoping that you
> never collide. You ought to think about precisely the same global coupling
> issues with MT19937 as you do with guaranteed-independent streams.
> Hope-and-prayer isn't really a substitute for properly engineering your
> problem. It's just a moral hazard to promote this method to the main API.

Nonsense.

If your definition of "hope and prayer" includes assuming that we
won't encounter a random collision in a 2**19937 state space, then
literally all engineering is hope-and-prayer. A collision could
happen, but if it does it's overwhelmingly more likely to happen
because of a flaw in the mathematical analysis, or a bug in the
implementation, or because random quantum fluctuations caused you and
your program to suddenly be transported to a parallel world where 1 +
1 = 1, than that you just got unlucky with your random state. And all
of these hazards apply equally to both MT19937 and more modern PRNGs.

...anyway, the real reason I'm a bit grumpy is because there are solid
engineering reasons why users *want* this API, so whether or not it
turns out to be possible I think we should at least be allowed to have
a discussion about whether there's some way to give it to them. It's
not even 100% out of the question that we conclude that existing PRNGs
are buggy because they don't take this use case into account -- it
would be far from the first time that numpy found itself going beyond
the limits of older numerical tools that weren't designed to build the
kind of large composable systems that numpy gets used for.

MT19937's state space is large enough that you could explicitly encode
a "tree seed" into it, even if you don't trust the laws of probability
-- e.g., you start with a RandomState with id [], then its children
have id [0], [1], [2], ..., and their children have ids [0, 0], [0,
1], ..., [1, 0], ..., and you write these into the state (probably
after sending them through some bit-mixing permutation), to guarantee
non-collision.

Or if you do trust the laws of probability, then the
randomly-sample-a-PRNG approach is not 100% out of the question even
for more modern PRNGs. For example, let's take a PRNG with a 64-bit
stream id and a 64-bit state space per stream. Suppose that we know
that in our application each PRNG will be used to draw 2**48 64-bit
samples (= 2 pebibytes of random data), and that we will use 2**32
PRNGs (= total of 8 yobibytes of random data). If we randomly
initialize each PRNG with a 128-bit (stream id, state) pair, then the
chance that two of our streams will overlap is about the chance of
getting a collision in an 80-bit state space (128 - 48) on 2**32
draws, which can be approximated[1] as

1 - exp(-(2**32)**2 / (2 * 2**80))
= 1 - exp(-2**64 / 2 ** 81)
= 1 - exp(-1/2**17)
= 7.6-chances-in-a-million

To put this number in some kind of context, Sandia says that on 128
pebibyte clusters (which I assume is the kind of system you use for a
simulation that needs multiple yobibytes of random data!) they see a
double-bit (!) DRAM fault ~every 4 minutes [2]. Also, by the time
we've drawn 2**48 samples from a single 64-bit stream than we've
already violated independence because in 2**48 draws you should see a
repeated value with probability ~1.0 (by the same birthday
approximation as above), but we see them with probability 0.

Alternatively, for a PRNG with a 256 bit state space, or 128-bit
stream id + 128-bit state space, then one can draw 2**64 64-bit
samples without violating independence, *and* do this for, say, 2**70
randomly initialized streams, and the probability of collision would
still be on the order of 10**-16.

Unless I totally messed up these calculations, I'd conclude that
almost everyone would be totally fine with a .random_state() method
combined with any reasonable generator, and for some reasonable
generators (those with more than, like, ~192 bits of state?) it's just
unconditionally safe for all physically realistic problem sizes.

-n

[1] https://en.wikipedia.org/wiki/Birthday_problem#Approximations
[2] http://www.fiala.me/pubs/papers/sc12-redmpi.pdf

-- 
Nathaniel J. Smith -- https://vorpus.org


From robert.kern at gmail.com  Wed May 18 08:07:30 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 May 2016 13:07:30 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
Message-ID: <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>

On Wed, May 18, 2016 at 1:14 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
> On Tue, May 17, 2016 at 10:41 AM, Robert Kern <robert.kern at gmail.com>
wrote:
> > On Tue, May 17, 2016 at 6:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> On May 17, 2016 1:50 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
> >> >
> >> [...]
> >> > What you want is a function that returns many RandomState objects
that
> >> > are hopefully spread around the MT19937 space enough that they are
> >> > essentially independent (in the absence of true jumpahead). The
better
> >> > implementation of such a function would look something like this:
> >> >
> >> > def spread_out_prngs(n, root_prng=None):
> >> >     if root_prng is None:
> >> >         root_prng = np.random
> >> >     elif not isinstance(root_prng, np.random.RandomState):
> >> >         root_prng = np.random.RandomState(root_prng)
> >> >     sprouted_prngs = []
> >> >     for i in range(n):
> >> >         seed_array = root_prng.randint(1<<32, size=624)  #
> >> > dtype=np.uint32 under 1.11
> >> >         sprouted_prngs.append(np.random.RandomState(seed_array))
> >> >     return spourted_prngs
> >>
> >> Maybe a nice way to encapsulate this in the RandomState interface
would be
> >> a method RandomState.random_state() that generates and returns a new
child
> >> RandomState.
> >
> > I disagree. This is a workaround in the absence of proper jumpahead or
> > guaranteed-independent streams. I would not encourage it.
> >
> >> > Internally, this generates seed arrays of about the size of the
MT19937
> >> > state so make sure that you can access more of the state space. That
will at
> >> > least make the chance of collision tiny. And it can be easily
rewritten to
> >> > take advantage of one of the newer PRNGs that have true independent
streams:
> >> >
> >> >   https://github.com/bashtage/ng-numpy-randomstate
> >>
> >> ... But unfortunately I'm not sure how to make my interface suggestion
> >> above work on top of one of these RNGs, because for
RandomState.random_state
> >> you really want a tree of independent RNGs and the fancy new PRNGs only
> >> provide a single flat namespace :-/. And even more annoyingly, the
tree API
> >> is actually a nicer API, because with a flat namespace you have to
know up
> >> front about all possible RNGs your code will use, which is an
unfortunate
> >> global coupling that makes it difficult to compose programs out of
> >> independent pieces, while the RandomState.random_state approach
composes
> >> beautifully. Maybe there's some clever way to allocate a 64-bit
namespace to
> >> make it look tree-like? I'm not sure 64 bits is really enough...
> >
> > MT19937 doesn't have a "tree" any more than the others. It's the same
flat
> > state space. You are just getting the illusion of a tree by hoping that
you
> > never collide. You ought to think about precisely the same global
coupling
> > issues with MT19937 as you do with guaranteed-independent streams.
> > Hope-and-prayer isn't really a substitute for properly engineering your
> > problem. It's just a moral hazard to promote this method to the main
API.
>
> Nonsense.
>
> If your definition of "hope and prayer" includes assuming that we
> won't encounter a random collision in a 2**19937 state space, then
> literally all engineering is hope-and-prayer. A collision could
> happen, but if it does it's overwhelmingly more likely to happen
> because of a flaw in the mathematical analysis, or a bug in the
> implementation, or because random quantum fluctuations caused you and
> your program to suddenly be transported to a parallel world where 1 +
> 1 = 1, than that you just got unlucky with your random state. And all
> of these hazards apply equally to both MT19937 and more modern PRNGs.

Granted.

> ...anyway, the real reason I'm a bit grumpy is because there are solid
> engineering reasons why users *want* this API,

I remain unconvinced on this mark. Grumpily.

> so whether or not it
> turns out to be possible I think we should at least be allowed to have
> a discussion about whether there's some way to give it to them.

I'm not shutting down discussion of the option. I *implemented* the option.
I think that discussing whether it should be part of the main API is
premature. There probably ought to be a paper or three out there supporting
its safety and utility first. Let the utility function version flourish
first.

> It's
> not even 100% out of the question that we conclude that existing PRNGs
> are buggy because they don't take this use case into account -- it
> would be far from the first time that numpy found itself going beyond
> the limits of older numerical tools that weren't designed to build the
> kind of large composable systems that numpy gets used for.
>
> MT19937's state space is large enough that you could explicitly encode
> a "tree seed" into it, even if you don't trust the laws of probability
> -- e.g., you start with a RandomState with id [], then its children
> have id [0], [1], [2], ..., and their children have ids [0, 0], [0,
> 1], ..., [1, 0], ..., and you write these into the state (probably
> after sending them through some bit-mixing permutation), to guarantee
> non-collision.

Sure. Not entirely sure if that can be done without preallocating the
branching factor or depth, but I'm sure there's some fancy combinatoric
representation of an unbounded tree that could be exploited here. It seems
likely to me that such a thing could be done with the stream IDs of PRNGs
that support that.

> Or if you do trust the laws of probability, then the
> randomly-sample-a-PRNG approach is not 100% out of the question even
> for more modern PRNGs.

Tread carefully here. PRNGs are not RNGs. Using the root PRNG to generate N
new seeds for the same PRNG does not necessarily generate N good,
uncorrelated streams with high probability. Overlap is not the only factor
in play. Long-term correlations happen even in the case where the N streams
do not overlap, and the structure of the root PRNG could well generate N
such correlated seeds.

http://www.adv-radio-sci.net/12/75/2014/

See section 4.4. It does look like one can get good MT19937 streams with
sequential integer seeds, when expanded by the standard sub-PRNG that we
use for that purpose (yay!). However, if you use a different (but otherwise
good-quality and general purpose) sub-PRNG to expand the key, you get
increasing numbers of failures as you increase the number of parallel
streams. It is not explored in the paper exactly why this is the case, but
I suspect that the similarity of the second sub-PRNG algorithm to that of
the MT itself is part of the problem.

It is *likely* that the scheme I implemented will be okay (our array-seed
expansion algorithm is similar to the integer-seed expansion and *probably*
insulates the sprouted MTs from their MT generating source), but that
remains to be demonstrated.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/fa9ffb22/attachment.html>

From chris.barker at noaa.gov  Wed May 18 11:50:25 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 18 May 2016 08:50:25 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
Message-ID: <CALGmxEKn_-gz2MmY3TSqrVKrjiaqUsc9X-QWRExxNsz2XkTLTA@mail.gmail.com>

>
> > ...anyway, the real reason I'm a bit grumpy is because there are solid
> > engineering reasons why users *want* this API,
>

Honestly, I am lost in the math -- but like any good engineer, I want to
accomplish something anyway :-) I trust you guys to get this right -- or at
least document what's "wrong" with it.

But, if I'm reading the use case that started all this correctly, it
closely matches my use-case. That is, I have a complex model with multiple
independent "random" processes. And we want to be able to re-produce
EXACTLY simulations -- our users get confused when the results are
"different" even if in a statistically insignificant way.

At the moment we are using one RNG, with one seed for everything. So we get
reproducible results, but if one thing is changed, then the entire
simulation is different -- which is OK, but it would be nicer to have each
process using its own RNG stream with it's own seed. However, it matters
not one whit if those seeds are independent -- the processes are different,
you'd never notice if they were using the same PRN stream -- because they
are used differently. So a "fairly low probability of a clash" would be
totally fine.

Granted, in a Monte Carlo simulation, it could be disastrous... :-)

I guess the point is -- do something reasonable, and document its
limitations, and we're all fine :-)

And thanks for giving your attention to this.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/715846e7/attachment.html>

From robert.kern at gmail.com  Wed May 18 12:01:44 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 May 2016 17:01:44 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CALGmxEKn_-gz2MmY3TSqrVKrjiaqUsc9X-QWRExxNsz2XkTLTA@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CALGmxEKn_-gz2MmY3TSqrVKrjiaqUsc9X-QWRExxNsz2XkTLTA@mail.gmail.com>
Message-ID: <CAF6FJitDRson-KDkoZphpKZ_RBs_z_6n+CJ0PNOUBjdtUhhk+g@mail.gmail.com>

On Wed, May 18, 2016 at 4:50 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>
>> > ...anyway, the real reason I'm a bit grumpy is because there are solid
>> > engineering reasons why users *want* this API,
>
> Honestly, I am lost in the math -- but like any good engineer, I want to
accomplish something anyway :-) I trust you guys to get this right -- or at
least document what's "wrong" with it.
>
> But, if I'm reading the use case that started all this correctly, it
closely matches my use-case. That is, I have a complex model with multiple
independent "random" processes. And we want to be able to re-produce
EXACTLY simulations -- our users get confused when the results are
"different" even if in a statistically insignificant way.
>
> At the moment we are using one RNG, with one seed for everything. So we
get reproducible results, but if one thing is changed, then the entire
simulation is different -- which is OK, but it would be nicer to have each
process using its own RNG stream with it's own seed. However, it matters
not one whit if those seeds are independent -- the processes are different,
you'd never notice if they were using the same PRN stream -- because they
are used differently. So a "fairly low probability of a clash" would be
totally fine.

Well, the main question is: do you need to be able to spawn dependent
streams at arbitrary points to an arbitrary depth without coordination
between processes? The necessity for multiple independent streams per se is
not contentious.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/bf1910eb/attachment.html>

From josef.pktd at gmail.com  Wed May 18 13:20:57 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 18 May 2016 13:20:57 -0400
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJitDRson-KDkoZphpKZ_RBs_z_6n+CJ0PNOUBjdtUhhk+g@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CALGmxEKn_-gz2MmY3TSqrVKrjiaqUsc9X-QWRExxNsz2XkTLTA@mail.gmail.com>
 <CAF6FJitDRson-KDkoZphpKZ_RBs_z_6n+CJ0PNOUBjdtUhhk+g@mail.gmail.com>
Message-ID: <CAMMTP+CGsmR5b5jm4+j8_psfa9=ceg6AWFeX9j7R1GNDPhhzHw@mail.gmail.com>

On Wed, May 18, 2016 at 12:01 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, May 18, 2016 at 4:50 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> >>
> >> > ...anyway, the real reason I'm a bit grumpy is because there are solid
> >> > engineering reasons why users *want* this API,
> >
> > Honestly, I am lost in the math -- but like any good engineer, I want to
> accomplish something anyway :-) I trust you guys to get this right -- or at
> least document what's "wrong" with it.
> >
> > But, if I'm reading the use case that started all this correctly, it
> closely matches my use-case. That is, I have a complex model with multiple
> independent "random" processes. And we want to be able to re-produce
> EXACTLY simulations -- our users get confused when the results are
> "different" even if in a statistically insignificant way.
> >
> > At the moment we are using one RNG, with one seed for everything. So we
> get reproducible results, but if one thing is changed, then the entire
> simulation is different -- which is OK, but it would be nicer to have each
> process using its own RNG stream with it's own seed. However, it matters
> not one whit if those seeds are independent -- the processes are different,
> you'd never notice if they were using the same PRN stream -- because they
> are used differently. So a "fairly low probability of a clash" would be
> totally fine.
>
> Well, the main question is: do you need to be able to spawn dependent
> streams at arbitrary points to an arbitrary depth without coordination
> between processes? The necessity for multiple independent streams per se is
> not contentious.
>


I'm similar to Chris, and didn't try to figure out the details of what you
are talking about.

However, if there are functions getting into numpy that help in using a
best practice even if it's not bullet proof, then it's still better than
home made approaches.
If it get's in soon, then we can use it in a few years (given dependency
lag). At that point there should be more distributed, nested simulation
based algorithms where we don't know in advance how far we have to go to
get reliable numbers or convergence.

(But I don't see anything like that right now.)

Josef



>
> --
> Robert Kern
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/f5c2b4d9/attachment.html>

From robert.kern at gmail.com  Wed May 18 13:56:50 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 May 2016 18:56:50 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAMMTP+CGsmR5b5jm4+j8_psfa9=ceg6AWFeX9j7R1GNDPhhzHw@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CALGmxEKn_-gz2MmY3TSqrVKrjiaqUsc9X-QWRExxNsz2XkTLTA@mail.gmail.com>
 <CAF6FJitDRson-KDkoZphpKZ_RBs_z_6n+CJ0PNOUBjdtUhhk+g@mail.gmail.com>
 <CAMMTP+CGsmR5b5jm4+j8_psfa9=ceg6AWFeX9j7R1GNDPhhzHw@mail.gmail.com>
Message-ID: <CAF6FJiv5ntRgHucA5yxeaSZu8GEjyKiL-RCHs+eH+iyz4QP8Kw@mail.gmail.com>

On Wed, May 18, 2016 at 6:20 PM, <josef.pktd at gmail.com> wrote:
>
> On Wed, May 18, 2016 at 12:01 PM, Robert Kern <robert.kern at gmail.com>
wrote:
>>
>> On Wed, May 18, 2016 at 4:50 PM, Chris Barker <chris.barker at noaa.gov>
wrote:
>> >>
>> >> > ...anyway, the real reason I'm a bit grumpy is because there are
solid
>> >> > engineering reasons why users *want* this API,
>> >
>> > Honestly, I am lost in the math -- but like any good engineer, I want
to accomplish something anyway :-) I trust you guys to get this right -- or
at least document what's "wrong" with it.
>> >
>> > But, if I'm reading the use case that started all this correctly, it
closely matches my use-case. That is, I have a complex model with multiple
independent "random" processes. And we want to be able to re-produce
EXACTLY simulations -- our users get confused when the results are
"different" even if in a statistically insignificant way.
>> >
>> > At the moment we are using one RNG, with one seed for everything. So
we get reproducible results, but if one thing is changed, then the entire
simulation is different -- which is OK, but it would be nicer to have each
process using its own RNG stream with it's own seed. However, it matters
not one whit if those seeds are independent -- the processes are different,
you'd never notice if they were using the same PRN stream -- because they
are used differently. So a "fairly low probability of a clash" would be
totally fine.
>>
>> Well, the main question is: do you need to be able to spawn dependent
streams at arbitrary points to an arbitrary depth without coordination
between processes? The necessity for multiple independent streams per se is
not contentious.
>
> I'm similar to Chris, and didn't try to figure out the details of what
you are talking about.
>
> However, if there are functions getting into numpy that help in using a
best practice even if it's not bullet proof, then it's still better than
home made approaches.
> If it get's in soon, then we can use it in a few years (given dependency
lag). At that point there should be more distributed, nested simulation
based algorithms where we don't know in advance how far we have to go to
get reliable numbers or convergence.
>
> (But I don't see anything like that right now.)

Current best practice is to use PRNGs with settable streams (or fixed
jumpahead for those PRNGs cursed to not have settable streams but blessed
to have super-long periods). The way to get those into numpy is to help
Kevin Sheppard finish:

  https://github.com/bashtage/ng-numpy-randomstate

He's done nearly all of the hard work already.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/7ac24b98/attachment.html>

From njs at pobox.com  Wed May 18 14:56:17 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 18 May 2016 11:56:17 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
Message-ID: <CAPJVwB=T1P27i2LDyXj3xdiZrJ4pEcq8Ac-pZQ7Ryh58YHDOTw@mail.gmail.com>

On Wed, May 18, 2016 at 5:07 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, May 18, 2016 at 1:14 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Tue, May 17, 2016 at 10:41 AM, Robert Kern <robert.kern at gmail.com>
>> wrote:
>> > On Tue, May 17, 2016 at 6:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> >>
>> >> On May 17, 2016 1:50 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
>> >> >
>> >> [...]
>> >> > What you want is a function that returns many RandomState objects
>> >> > that
>> >> > are hopefully spread around the MT19937 space enough that they are
>> >> > essentially independent (in the absence of true jumpahead). The
>> >> > better
>> >> > implementation of such a function would look something like this:
>> >> >
>> >> > def spread_out_prngs(n, root_prng=None):
>> >> >     if root_prng is None:
>> >> >         root_prng = np.random
>> >> >     elif not isinstance(root_prng, np.random.RandomState):
>> >> >         root_prng = np.random.RandomState(root_prng)
>> >> >     sprouted_prngs = []
>> >> >     for i in range(n):
>> >> >         seed_array = root_prng.randint(1<<32, size=624)  #
>> >> > dtype=np.uint32 under 1.11
>> >> >         sprouted_prngs.append(np.random.RandomState(seed_array))
>> >> >     return spourted_prngs
>> >>
>> >> Maybe a nice way to encapsulate this in the RandomState interface would
>> >> be
>> >> a method RandomState.random_state() that generates and returns a new
>> >> child
>> >> RandomState.
>> >
>> > I disagree. This is a workaround in the absence of proper jumpahead or
>> > guaranteed-independent streams. I would not encourage it.
>> >
>> >> > Internally, this generates seed arrays of about the size of the
>> >> > MT19937
>> >> > state so make sure that you can access more of the state space. That
>> >> > will at
>> >> > least make the chance of collision tiny. And it can be easily
>> >> > rewritten to
>> >> > take advantage of one of the newer PRNGs that have true independent
>> >> > streams:
>> >> >
>> >> >   https://github.com/bashtage/ng-numpy-randomstate
>> >>
>> >> ... But unfortunately I'm not sure how to make my interface suggestion
>> >> above work on top of one of these RNGs, because for
>> >> RandomState.random_state
>> >> you really want a tree of independent RNGs and the fancy new PRNGs only
>> >> provide a single flat namespace :-/. And even more annoyingly, the tree
>> >> API
>> >> is actually a nicer API, because with a flat namespace you have to know
>> >> up
>> >> front about all possible RNGs your code will use, which is an
>> >> unfortunate
>> >> global coupling that makes it difficult to compose programs out of
>> >> independent pieces, while the RandomState.random_state approach
>> >> composes
>> >> beautifully. Maybe there's some clever way to allocate a 64-bit
>> >> namespace to
>> >> make it look tree-like? I'm not sure 64 bits is really enough...
>> >
>> > MT19937 doesn't have a "tree" any more than the others. It's the same
>> > flat
>> > state space. You are just getting the illusion of a tree by hoping that
>> > you
>> > never collide. You ought to think about precisely the same global
>> > coupling
>> > issues with MT19937 as you do with guaranteed-independent streams.
>> > Hope-and-prayer isn't really a substitute for properly engineering your
>> > problem. It's just a moral hazard to promote this method to the main
>> > API.
>>
>> Nonsense.
>>
>> If your definition of "hope and prayer" includes assuming that we
>> won't encounter a random collision in a 2**19937 state space, then
>> literally all engineering is hope-and-prayer. A collision could
>> happen, but if it does it's overwhelmingly more likely to happen
>> because of a flaw in the mathematical analysis, or a bug in the
>> implementation, or because random quantum fluctuations caused you and
>> your program to suddenly be transported to a parallel world where 1 +
>> 1 = 1, than that you just got unlucky with your random state. And all
>> of these hazards apply equally to both MT19937 and more modern PRNGs.
>
> Granted.
>
>> ...anyway, the real reason I'm a bit grumpy is because there are solid
>> engineering reasons why users *want* this API,
>
> I remain unconvinced on this mark. Grumpily.

Sorry for getting grumpy :-). The engineering reasons seem pretty
obvious to me though? If you have any use case for independent streams
at all, and you're writing code that's intended to live inside a
library's abstraction barrier, then you need some way to choose your
streams to avoid colliding with arbitrary other code that the end-user
might assemble alongside yours as part of their final program. So
AFAICT you have two options: either you need a "tree-style" API for
allocating these streams, or else you need to add some explicit API to
your library that lets the end-user control in detail which streams
you use. Both are possible, but the latter is obviously undesireable
if you can avoid it, since it breaks the abstraction barrier, making
your library more complicated to use and harder to evolve.

>> so whether or not it
>> turns out to be possible I think we should at least be allowed to have
>> a discussion about whether there's some way to give it to them.
>
> I'm not shutting down discussion of the option. I *implemented* the option.
> I think that discussing whether it should be part of the main API is
> premature. There probably ought to be a paper or three out there supporting
> its safety and utility first. Let the utility function version flourish
> first.

OK -- I guess this particularly makes sense given how
extra-tightly-constrained we currently are in fixing mistakes in
np.random. But I feel like in the end the right place for this really
is inside the RandomState interface, because the person implementing
RandomState is the one best placed to understand (a) the gnarly
technical details here, and (b) how those change depending on the
particular PRNG in use. I don't want to end up with a bunch of
subtly-buggy utility functions in non-specialist libraries like dask
-- so we should be trying to help downstream users figure out how to
actually get this into np.random?

>> It's
>> not even 100% out of the question that we conclude that existing PRNGs
>> are buggy because they don't take this use case into account -- it
>> would be far from the first time that numpy found itself going beyond
>> the limits of older numerical tools that weren't designed to build the
>> kind of large composable systems that numpy gets used for.
>>
>> MT19937's state space is large enough that you could explicitly encode
>> a "tree seed" into it, even if you don't trust the laws of probability
>> -- e.g., you start with a RandomState with id [], then its children
>> have id [0], [1], [2], ..., and their children have ids [0, 0], [0,
>> 1], ..., [1, 0], ..., and you write these into the state (probably
>> after sending them through some bit-mixing permutation), to guarantee
>> non-collision.
>
> Sure. Not entirely sure if that can be done without preallocating the
> branching factor or depth, but I'm sure there's some fancy combinatoric
> representation of an unbounded tree that could be exploited here. It seems
> likely to me that such a thing could be done with the stream IDs of PRNGs
> that support that.

I'm pretty sure you do have to preallocate both the branching factor
and depth, since getting the collision-free guarantee requires that
across the universe of possible tree addresses, each state id gets
used at most once -- and there are finitely many state ids. But as a
practical matter, saying "you can only sprout up to 2**32 new states
out of any given state, and you can only nest them 600 deep, and
exceeding these bounds is an error" would still be "composable enough"
for ~all practical purposes.

>> Or if you do trust the laws of probability, then the
>> randomly-sample-a-PRNG approach is not 100% out of the question even
>> for more modern PRNGs.
>
> Tread carefully here. PRNGs are not RNGs. Using the root PRNG to generate N
> new seeds for the same PRNG does not necessarily generate N good,
> uncorrelated streams with high probability. Overlap is not the only factor
> in play. Long-term correlations happen even in the case where the N streams
> do not overlap, and the structure of the root PRNG could well generate N
> such correlated seeds.
>
> http://www.adv-radio-sci.net/12/75/2014/
>
> See section 4.4. It does look like one can get good MT19937 streams with
> sequential integer seeds, when expanded by the standard sub-PRNG that we use
> for that purpose (yay!). However, if you use a different (but otherwise
> good-quality and general purpose) sub-PRNG to expand the key, you get
> increasing numbers of failures as you increase the number of parallel
> streams. It is not explored in the paper exactly why this is the case, but I
> suspect that the similarity of the second sub-PRNG algorithm to that of the
> MT itself is part of the problem.
>
> It is *likely* that the scheme I implemented will be okay (our array-seed
> expansion algorithm is similar to the integer-seed expansion and *probably*
> insulates the sprouted MTs from their MT generating source), but that
> remains to be demonstrated.

Right... I think the way to think about this is to split it into two
pieces. First, there's the issue that correlated streams exist at all.
That they do for MT is annoying and closely related to why we prefer
other PRNGs these days. When doing the analysis of the collision
probability of randomly sprouted generators, this effectively reduces
the size of the space and increases the risk of collision. For MT this
by itself isn't a big deal because the space is so ludicrously large,
but for the more modern PRNGs with smaller state spaces you definitely
want these correlated streams not to exist at all. Fortunately I
believe this is a design criterion that modern PRNGs take into
account, but yeah, one needs to check this. (This kind of issue is
also why I have a fondness for PRNGs designed on the "get a bigger
hammer" principle, like AES-CTR :-).)

Second, there's the issue that that paper talks about, where *if*
correlated streams exist, then the structure of your PRNG might be
such that when sampling a new seed from an old PRNG, you're
unreasonably likely to hit one of them. (The most obvious example of
this would be if you use an archaic PRNG design that works by
outputting its internal state and then mixing it -- if you use the
output of this PRNG to seed a new PRNG then the two will be identical
modulo some lag!) If your sprouting procedure is subject to this kind
of phenomenon, then it totally invalidates the use of simple
probability theory to analyze the chance of collisions.

But, I ignored this in my analysis because it's definitely solveable
:-). All we need is some deterministic function that we can apply to
the output of our original PRNG that will give us back something
"effectively random" to use as our sprouted seed, totally destroying
all correlations. And this problem of destroying correlations is
well-studied in the crypto world (basically destroying these
correlations *is* the entire problem of cryptographic primitive
design), so even if we have doubts about the specific initialization
functions usually used with MT, we have available to us lots of
primitives that we're *very* confident will do a *very* thorough job,
and sprouting isn't a performance sensitive operations so it doesn't
matter if it takes a little longer using some crypto hash instead of
the classic MT seeding algorithm. And then the simple birthday
calculations are appropriate again.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org


From charlesr.harris at gmail.com  Wed May 18 17:09:36 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 May 2016 15:09:36 -0600
Subject: [Numpy-discussion] Scipy 2016 attending
Message-ID: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>

Hi All,

Out of curiosity, who all here intends to be at Scipy 2016?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/b0c4b07d/attachment.html>

From chris.barker at noaa.gov  Wed May 18 18:02:24 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 18 May 2016 15:02:24 -0700
Subject: [Numpy-discussion] Scipy 2016 attending
In-Reply-To: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>
References: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>
Message-ID: <CALGmxELyPiZ57RGLh8ucsVFXp7=8RtfOfsnoB4r_1q61aS3jiw@mail.gmail.com>

I'll be there.

-CHB


On Wed, May 18, 2016 at 2:09 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> Hi All,
>
> Out of curiosity, who all here intends to be at Scipy 2016?
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/39df922c/attachment.html>

From njs at pobox.com  Wed May 18 18:03:31 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 18 May 2016 15:03:31 -0700
Subject: [Numpy-discussion] Scipy 2016 attending
In-Reply-To: <CALGmxELyPiZ57RGLh8ucsVFXp7=8RtfOfsnoB4r_1q61aS3jiw@mail.gmail.com>
References: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>
 <CALGmxELyPiZ57RGLh8ucsVFXp7=8RtfOfsnoB4r_1q61aS3jiw@mail.gmail.com>
Message-ID: <CAPJVwBmJM7CevJ8+O6Szwhn4bHJy_QuaxanBUDYTHvKsyQMZdw@mail.gmail.com>

Me too.

On Wed, May 18, 2016 at 3:02 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> I'll be there.
>
> -CHB
>
>
> On Wed, May 18, 2016 at 2:09 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>> Hi All,
>>
>> Out of curiosity, who all here intends to be at Scipy 2016?
>>
>> Chuck
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Nathaniel J. Smith -- https://vorpus.org


From waterbug at pangalactic.us  Wed May 18 19:04:15 2016
From: waterbug at pangalactic.us (Steve Waterbury)
Date: Wed, 18 May 2016 19:04:15 -0400
Subject: [Numpy-discussion] Scipy 2016 attending
In-Reply-To: <CAPJVwBmJM7CevJ8+O6Szwhn4bHJy_QuaxanBUDYTHvKsyQMZdw@mail.gmail.com>
References: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>
 <CALGmxELyPiZ57RGLh8ucsVFXp7=8RtfOfsnoB4r_1q61aS3jiw@mail.gmail.com>
 <CAPJVwBmJM7CevJ8+O6Szwhn4bHJy_QuaxanBUDYTHvKsyQMZdw@mail.gmail.com>
Message-ID: <573CF4EF.4000306@pangalactic.us>

Me 3!  ;)

Steve

On 05/18/2016 06:03 PM, Nathaniel Smith wrote:
> Me too.
>
> On Wed, May 18, 2016 at 3:02 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>> I'll be there.
>>
>> -CHB
>>
>>
>> On Wed, May 18, 2016 at 2:09 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>> Hi All,
>>>
>>> Out of curiosity, who all here intends to be at Scipy 2016?
>>>
>>> Chuck
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/e2384d31/attachment.html>

From rmay31 at gmail.com  Wed May 18 21:45:26 2016
From: rmay31 at gmail.com (Ryan May)
Date: Wed, 18 May 2016 19:45:26 -0600
Subject: [Numpy-discussion] Scipy 2016 attending
In-Reply-To: <573CF4EF.4000306@pangalactic.us>
References: <CAB6mnxKhTBVHPZzx_Rf_hTQnt-hD6-xQ2iBDEbEpJANd0+3qrg@mail.gmail.com>
 <CALGmxELyPiZ57RGLh8ucsVFXp7=8RtfOfsnoB4r_1q61aS3jiw@mail.gmail.com>
 <CAPJVwBmJM7CevJ8+O6Szwhn4bHJy_QuaxanBUDYTHvKsyQMZdw@mail.gmail.com>
 <573CF4EF.4000306@pangalactic.us>
Message-ID: <CAKH0P+Usi=EZ2HDKJ5eGCwNWKrnuw90S-m9qQtoMaYQr8JWpYQ@mail.gmail.com>

Yup.

On Wed, May 18, 2016 at 5:04 PM, Steve Waterbury <waterbug at pangalactic.us>
wrote:

> Me 3!  ;)
>
> Steve
>
>
> On 05/18/2016 06:03 PM, Nathaniel Smith wrote:
>
> Me too.
>
> On Wed, May 18, 2016 at 3:02 PM, Chris Barker <chris.barker at noaa.gov> <chris.barker at noaa.gov> wrote:
>
> I'll be there.
>
> -CHB
>
>
> On Wed, May 18, 2016 at 2:09 PM, Charles R Harris<charlesr.harris at gmail.com> <charlesr.harris at gmail.com> wrote:
>
> Hi All,
>
> Out of curiosity, who all here intends to be at Scipy 2016?
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttps://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> Chris.Barker at noaa.gov
>
> _______________________________________________
> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttps://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Ryan May
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160518/0196bfd7/attachment.html>

From cgodshall at enthought.com  Thu May 19 14:07:45 2016
From: cgodshall at enthought.com (Courtenay Godshall (Enthought))
Date: Thu, 19 May 2016 13:07:45 -0500
Subject: [Numpy-discussion] ANN: SciPy 2016 Conference (Scientific Computing
	with Python): Tutorials and Talks Announced
Message-ID: <010c01d1b1f9$58adc010$0a094030$@enthought.com>

**ANN: SciPy 2016 Conference (Scientific Computing with Python): Tutorials
and Talks Announced**

 

We're excited to announce this year's accepted Talks
<http://scipy2016.scipy.org/ehome/146062/332963/> & Posters and Tutorial
Schedule <http://scipy2016.scipy.org/ehome/146062/332960/> !  This year's 3
major talk tracks include Python in Data Science, High Performance
Computing, and general Scientific Computing. Our six mini-symposia include:
Earth and Space Science, Engineering, Medicine and Biology, Case Studies in
Industry, Education, and Reproducibility. For tutorials, you can choose from
<http://scipy2016.scipy.org/ehome/146062/332960/>  18 different SciPy
tutorials, including a  <https://katyhuff.github.io/2016-07-11-scipy> 1 day
Software Carpentry Scientific Python course that assumes some programming
experience but no Python knowledge, or a 2-day Software Carpentry Instructor
Training.

 

We hope you'll join us - early bird registration ENDS May 22, 2016. Register
at: http://scipy2016.scipy.org/ehome/146062/332936/

 

About SciPy 2016

SciPy 2016 <http://www.scipy2016.scipy.org> , the 15th annual Scientific
Computing with Python conference, will be held July 11-17, 2016 in Austin,
Texas. SciPy is a community dedicated to the advancement of scientific
computing through open source Python software for mathematics, science, and
engineering. The annual SciPy Conference brings together over 650
participants from industry, academia, and government to showcase their
latest projects, learn from skilled users and developers, and collaborate on
code development. The full program will consist of
<http://scipy2016.scipy.org/ehome/146062/332960/>  2 days of tutorials (July
11-12), <http://scipy2016.scipy.org/ehome/146062/332963/>  3 days of talks
(July 13-15), and 2days of
<http://scipy2016.scipy.org/ehome/146062/332969/?&&>  developer sprints
(July 16-17). More info is available on the conference website at
<http://www.scipy2016.scipy.org>  http://scipy2016.scipy.org (where you can
sign up for the mailing list); or follow <https://twitter.com/scipyconf>
@scipyconf on Twitter.  

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160519/098c8b3d/attachment.html>

From charlesr.harris at gmail.com  Thu May 19 17:37:05 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 19 May 2016 15:37:05 -0600
Subject: [Numpy-discussion] Integers to integer powers
Message-ID: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>

Hi All,

There are currently several pull requests apropos integer arrays/scalars to
integer powers and, because the area is messy and involves tradeoffs, I'd
like to see some discussion here on the list before proceeding.

*Scalars in 1.10*

In [1]: 1 ** -1
Out[1]: 1.0

In [2]: int16(1) ** -1
Out[2]: 1

In [3]: int32(1) ** -1
Out[3]: 1

In [4]: int64(1) ** -1
Out[4]: 1.0

In [5]: 2 ** -1
Out[5]: 0.5

In [6]: int16(2) ** -1
Out[6]: 0

In [7]: int32(2) ** -1
Out[7]: 0

In [8]: int64(2) ** -1
Out[8]: 0.5

In [9]: 0 ** -1
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-9-fd405d6cf4bc> in <module>()
----> 1 0 ** -1

ZeroDivisionError: 0.0 cannot be raised to a negative power

In [10]: int16(0) ** -1
/home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
encountered in power
  #!/usr/bin/python
/home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
encountered in power
  #!/usr/bin/python
Out[10]: -9223372036854775808

In [11]: int32(0) ** -1
Out[11]: -9223372036854775808

In [12]: int64(0) ** -1
/home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
encountered in long_scalars
  #!/usr/bin/python
Out[12]: inf

Proposed

   - for non-zero numbers the return type should be float.
   - for zero numbers a zero division error should be raised.




*Scalar Arrays in 1.10*
In [1]: array(1, dtype=int16) ** -1
Out[1]: 1

In [2]: array(1, dtype=int32) ** -1
Out[2]: 1

In [3]: array(1, dtype=int64) ** -1
Out[3]: 1

In [4]: array(2, dtype=int16) ** -1
Out[4]: 0

In [5]: array(2, dtype=int32) ** -1
Out[5]: 0

In [6]: array(2, dtype=int64) ** -1
Out[6]: 0

In [7]: array(0, dtype=int16) ** -1
/home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
encountered in power
  #!/usr/bin/python
/home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
encountered in power
  #!/usr/bin/python
Out[7]: -9223372036854775808

In [8]: array(0, dtype=int32) ** -1
Out[8]: -9223372036854775808

In [9]: array(0, dtype=int64) ** -1
Out[9]: -9223372036854775808

In [10]: type(array(1, dtype=int64) ** -1)
Out[10]: numpy.int64

In [11]: type(array(1, dtype=int32) ** -1)
Out[11]: numpy.int64

In [12]: type(array(1, dtype=int16) ** -1)
Out[12]: numpy.int64

Note that the return type is always int64 in all these cases. However, type
is preserved in non-scalar arrays, although the value of int16 is not
compatible with int32 and int64 for zero division.

In [22]: array([0]*2, dtype=int16) ** -1
Out[22]: array([0, 0], dtype=int16)

In [23]: array([0]*2, dtype=int32) ** -1
Out[23]: array([-2147483648, -2147483648], dtype=int32)

In [24]: array([0]*2, dtype=int64) ** -1
Out[24]: array([-9223372036854775808, -9223372036854775808])

Proposed:

   - Raise an ZeroDivisionError for zero division, that is, in the ufunc.
   - Scalar arrays to return scalar arrays


Thoughts?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160519/0d2caab1/attachment.html>

From josef.pktd at gmail.com  Thu May 19 22:16:35 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 19 May 2016 22:16:35 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
Message-ID: <CAMMTP+AkK1QONmYhn37nU-oXKLq7zZhYY+0=a1iHY6DQz+X1iA@mail.gmail.com>

On Thu, May 19, 2016 at 5:37 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> Hi All,
>
> There are currently several pull requests apropos integer arrays/scalars
> to integer powers and, because the area is messy and involves tradeoffs,
> I'd like to see some discussion here on the list before proceeding.
>
> *Scalars in 1.10*
>
> In [1]: 1 ** -1
> Out[1]: 1.0
>
> In [2]: int16(1) ** -1
> Out[2]: 1
>
> In [3]: int32(1) ** -1
> Out[3]: 1
>
> In [4]: int64(1) ** -1
> Out[4]: 1.0
>
> In [5]: 2 ** -1
> Out[5]: 0.5
>
> In [6]: int16(2) ** -1
> Out[6]: 0
>
> In [7]: int32(2) ** -1
> Out[7]: 0
>
> In [8]: int64(2) ** -1
> Out[8]: 0.5
>
> In [9]: 0 ** -1
> ---------------------------------------------------------------------------
> ZeroDivisionError                         Traceback (most recent call last)
> <ipython-input-9-fd405d6cf4bc> in <module>()
> ----> 1 0 ** -1
>
> ZeroDivisionError: 0.0 cannot be raised to a negative power
>
> In [10]: int16(0) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in power
>   #!/usr/bin/python
> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
> encountered in power
>   #!/usr/bin/python
> Out[10]: -9223372036854775808
>
> In [11]: int32(0) ** -1
> Out[11]: -9223372036854775808
>
> In [12]: int64(0) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in long_scalars
>   #!/usr/bin/python
> Out[12]: inf
>
> Proposed
>
>    - for non-zero numbers the return type should be float.
>    - for zero numbers a zero division error should be raised.
>
>
>
>
> *Scalar Arrays in 1.10*
> In [1]: array(1, dtype=int16) ** -1
> Out[1]: 1
>
> In [2]: array(1, dtype=int32) ** -1
> Out[2]: 1
>
> In [3]: array(1, dtype=int64) ** -1
> Out[3]: 1
>
> In [4]: array(2, dtype=int16) ** -1
> Out[4]: 0
>
> In [5]: array(2, dtype=int32) ** -1
> Out[5]: 0
>
> In [6]: array(2, dtype=int64) ** -1
> Out[6]: 0
>
> In [7]: array(0, dtype=int16) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in power
>   #!/usr/bin/python
> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
> encountered in power
>   #!/usr/bin/python
> Out[7]: -9223372036854775808
>
> In [8]: array(0, dtype=int32) ** -1
> Out[8]: -9223372036854775808
>
> In [9]: array(0, dtype=int64) ** -1
> Out[9]: -9223372036854775808
>
> In [10]: type(array(1, dtype=int64) ** -1)
> Out[10]: numpy.int64
>
> In [11]: type(array(1, dtype=int32) ** -1)
> Out[11]: numpy.int64
>
> In [12]: type(array(1, dtype=int16) ** -1)
> Out[12]: numpy.int64
>
> Note that the return type is always int64 in all these cases. However,
> type is preserved in non-scalar arrays, although the value of int16 is not
> compatible with int32 and int64 for zero division.
>
> In [22]: array([0]*2, dtype=int16) ** -1
> Out[22]: array([0, 0], dtype=int16)
>
> In [23]: array([0]*2, dtype=int32) ** -1
> Out[23]: array([-2147483648, -2147483648], dtype=int32)
>
> In [24]: array([0]*2, dtype=int64) ** -1
> Out[24]: array([-9223372036854775808, -9223372036854775808])
>
> Proposed:
>
>    - Raise an ZeroDivisionError for zero division, that is, in the ufunc.
>    - Scalar arrays to return scalar arrays
>
>
> Thoughts?
>
Why does negative exponent not upcast to float like division?
sounds like python 2 to me

Josef



> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160519/3a1b1f06/attachment.html>

From josef.pktd at gmail.com  Thu May 19 22:17:54 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 19 May 2016 22:17:54 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAMMTP+AkK1QONmYhn37nU-oXKLq7zZhYY+0=a1iHY6DQz+X1iA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAMMTP+AkK1QONmYhn37nU-oXKLq7zZhYY+0=a1iHY6DQz+X1iA@mail.gmail.com>
Message-ID: <CAMMTP+CyV1sUQqANySaNfE1mtqO0BVkFT39Wq3Wy036EBh-iaw@mail.gmail.com>

On Thu, May 19, 2016 at 10:16 PM, <josef.pktd at gmail.com> wrote:

>
>
> On Thu, May 19, 2016 at 5:37 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> Hi All,
>>
>> There are currently several pull requests apropos integer arrays/scalars
>> to integer powers and, because the area is messy and involves tradeoffs,
>> I'd like to see some discussion here on the list before proceeding.
>>
>> *Scalars in 1.10*
>>
>> In [1]: 1 ** -1
>> Out[1]: 1.0
>>
>> In [2]: int16(1) ** -1
>> Out[2]: 1
>>
>> In [3]: int32(1) ** -1
>> Out[3]: 1
>>
>> In [4]: int64(1) ** -1
>> Out[4]: 1.0
>>
>> In [5]: 2 ** -1
>> Out[5]: 0.5
>>
>> In [6]: int16(2) ** -1
>> Out[6]: 0
>>
>> In [7]: int32(2) ** -1
>> Out[7]: 0
>>
>> In [8]: int64(2) ** -1
>> Out[8]: 0.5
>>
>> In [9]: 0 ** -1
>>
>> ---------------------------------------------------------------------------
>> ZeroDivisionError                         Traceback (most recent call
>> last)
>> <ipython-input-9-fd405d6cf4bc> in <module>()
>> ----> 1 0 ** -1
>>
>> ZeroDivisionError: 0.0 cannot be raised to a negative power
>>
>> In [10]: int16(0) ** -1
>> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
>> encountered in power
>>   #!/usr/bin/python
>> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
>> encountered in power
>>   #!/usr/bin/python
>> Out[10]: -9223372036854775808
>>
>> In [11]: int32(0) ** -1
>> Out[11]: -9223372036854775808
>>
>> In [12]: int64(0) ** -1
>> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
>> encountered in long_scalars
>>   #!/usr/bin/python
>> Out[12]: inf
>>
>> Proposed
>>
>>    - for non-zero numbers the return type should be float.
>>    - for zero numbers a zero division error should be raised.
>>
>>
>>
>>
>> *Scalar Arrays in 1.10*
>> In [1]: array(1, dtype=int16) ** -1
>> Out[1]: 1
>>
>> In [2]: array(1, dtype=int32) ** -1
>> Out[2]: 1
>>
>> In [3]: array(1, dtype=int64) ** -1
>> Out[3]: 1
>>
>> In [4]: array(2, dtype=int16) ** -1
>> Out[4]: 0
>>
>> In [5]: array(2, dtype=int32) ** -1
>> Out[5]: 0
>>
>> In [6]: array(2, dtype=int64) ** -1
>> Out[6]: 0
>>
>> In [7]: array(0, dtype=int16) ** -1
>> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
>> encountered in power
>>   #!/usr/bin/python
>> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
>> encountered in power
>>   #!/usr/bin/python
>> Out[7]: -9223372036854775808
>>
>> In [8]: array(0, dtype=int32) ** -1
>> Out[8]: -9223372036854775808
>>
>> In [9]: array(0, dtype=int64) ** -1
>> Out[9]: -9223372036854775808
>>
>> In [10]: type(array(1, dtype=int64) ** -1)
>> Out[10]: numpy.int64
>>
>> In [11]: type(array(1, dtype=int32) ** -1)
>> Out[11]: numpy.int64
>>
>> In [12]: type(array(1, dtype=int16) ** -1)
>> Out[12]: numpy.int64
>>
>> Note that the return type is always int64 in all these cases. However,
>> type is preserved in non-scalar arrays, although the value of int16 is not
>> compatible with int32 and int64 for zero division.
>>
>> In [22]: array([0]*2, dtype=int16) ** -1
>> Out[22]: array([0, 0], dtype=int16)
>>
>> In [23]: array([0]*2, dtype=int32) ** -1
>> Out[23]: array([-2147483648, -2147483648], dtype=int32)
>>
>> In [24]: array([0]*2, dtype=int64) ** -1
>> Out[24]: array([-9223372036854775808, -9223372036854775808])
>>
>> Proposed:
>>
>>    - Raise an ZeroDivisionError for zero division, that is, in the ufunc.
>>    - Scalar arrays to return scalar arrays
>>
>>
>> Thoughts?
>>
> Why does negative exponent not upcast to float like division?
> sounds like python 2 to me
>

from __future__ import negative_power

Josef


>
> Josef
>
>
>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160519/7fb92db9/attachment.html>

From njs at pobox.com  Thu May 19 23:30:40 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 19 May 2016 20:30:40 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
Message-ID: <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>

So I guess what makes this tricky is that:

- We want the behavior to the same for multiple-element arrays,
single-element arrays, zero-dimensional arrays, and scalars -- the
shape of the data shouldn't affect the semantics of **

- We also want the numpy scalar behavior to match the Python scalar behavior

- For Python scalars, int ** (positive int) returns an int, but int **
(negative int) returns a float.

- For arrays, int ** (positive int) and int ** (negative int) _have_
to return the same type, because in general output types are always a
function of the input types and *can't* look at the specific values
involved, and in specific because if you do array([2, 3]) ** array([2,
-2]) you can't return an array where the first element is int and the
second is float.

Given these immutable and contradictory constraints, the last bad
option IMHO would be that we make int ** (negative int) an error in
all cases, and the error message can suggest that instead of writing

    np.array(2) ** -2

they should instead write

    np.array(2) ** -2.0

(And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)

Definitely annoying, but all the other options seem even more
inconsistent and confusing, and likely to encourage the writing of
subtly buggy code...

(I especially have in mind numpy's habit of silently switching between
scalars and zero-dimensional arrays -- so it's easy to write code that
you think handles arbitrary array dimensions, and it even passes all
your tests, but then it fails when someone passes in a different shape
data and triggers some scalar/array inconsistency. E.g. if we make **
-2 work for scalars but not arrays, then this code:

def f(arr):
    return np.sum(arr, axis=0) ** -2

works as expected for 1-d input, tests pass, everyone's happy... but
as soon as you try to pass in higher dimensional integer input it will
fail.)

-n

On Thu, May 19, 2016 at 2:37 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Hi All,
>
> There are currently several pull requests apropos integer arrays/scalars to
> integer powers and, because the area is messy and involves tradeoffs, I'd
> like to see some discussion here on the list before proceeding.
>
> Scalars in 1.10
>
> In [1]: 1 ** -1
> Out[1]: 1.0
>
> In [2]: int16(1) ** -1
> Out[2]: 1
>
> In [3]: int32(1) ** -1
> Out[3]: 1
>
> In [4]: int64(1) ** -1
> Out[4]: 1.0
>
> In [5]: 2 ** -1
> Out[5]: 0.5
>
> In [6]: int16(2) ** -1
> Out[6]: 0
>
> In [7]: int32(2) ** -1
> Out[7]: 0
>
> In [8]: int64(2) ** -1
> Out[8]: 0.5
>
> In [9]: 0 ** -1
> ---------------------------------------------------------------------------
> ZeroDivisionError                         Traceback (most recent call last)
> <ipython-input-9-fd405d6cf4bc> in <module>()
> ----> 1 0 ** -1
>
> ZeroDivisionError: 0.0 cannot be raised to a negative power
>
> In [10]: int16(0) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in power
>   #!/usr/bin/python
> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
> encountered in power
>   #!/usr/bin/python
> Out[10]: -9223372036854775808
>
> In [11]: int32(0) ** -1
> Out[11]: -9223372036854775808
>
> In [12]: int64(0) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in long_scalars
>   #!/usr/bin/python
> Out[12]: inf
>
> Proposed
>
> for non-zero numbers the return type should be float.
> for zero numbers a zero division error should be raised.
>
>
> Scalar Arrays in 1.10
>
> In [1]: array(1, dtype=int16) ** -1
> Out[1]: 1
>
> In [2]: array(1, dtype=int32) ** -1
> Out[2]: 1
>
> In [3]: array(1, dtype=int64) ** -1
> Out[3]: 1
>
> In [4]: array(2, dtype=int16) ** -1
> Out[4]: 0
>
> In [5]: array(2, dtype=int32) ** -1
> Out[5]: 0
>
> In [6]: array(2, dtype=int64) ** -1
> Out[6]: 0
>
> In [7]: array(0, dtype=int16) ** -1
> /home/charris/.local/bin/ipython:1: RuntimeWarning: divide by zero
> encountered in power
>   #!/usr/bin/python
> /home/charris/.local/bin/ipython:1: RuntimeWarning: invalid value
> encountered in power
>   #!/usr/bin/python
> Out[7]: -9223372036854775808
>
> In [8]: array(0, dtype=int32) ** -1
> Out[8]: -9223372036854775808
>
> In [9]: array(0, dtype=int64) ** -1
> Out[9]: -9223372036854775808
>
> In [10]: type(array(1, dtype=int64) ** -1)
> Out[10]: numpy.int64
>
> In [11]: type(array(1, dtype=int32) ** -1)
> Out[11]: numpy.int64
>
> In [12]: type(array(1, dtype=int16) ** -1)
> Out[12]: numpy.int64
>
> Note that the return type is always int64 in all these cases. However, type
> is preserved in non-scalar arrays, although the value of int16 is not
> compatible with int32 and int64 for zero division.
>
> In [22]: array([0]*2, dtype=int16) ** -1
> Out[22]: array([0, 0], dtype=int16)
>
> In [23]: array([0]*2, dtype=int32) ** -1
> Out[23]: array([-2147483648, -2147483648], dtype=int32)
>
> In [24]: array([0]*2, dtype=int64) ** -1
> Out[24]: array([-9223372036854775808, -9223372036854775808])
>
> Proposed:
>
> Raise an ZeroDivisionError for zero division, that is, in the ufunc.
> Scalar arrays to return scalar arrays
>
>
> Thoughts?
>
> Chuck
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Nathaniel J. Smith -- https://vorpus.org


From charlesr.harris at gmail.com  Fri May 20 14:35:47 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 May 2016 12:35:47 -0600
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
Message-ID: <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>

On Thu, May 19, 2016 at 9:30 PM, Nathaniel Smith <njs at pobox.com> wrote:

> So I guess what makes this tricky is that:
>
> - We want the behavior to the same for multiple-element arrays,
> single-element arrays, zero-dimensional arrays, and scalars -- the
> shape of the data shouldn't affect the semantics of **
>
> - We also want the numpy scalar behavior to match the Python scalar
> behavior
>
> - For Python scalars, int ** (positive int) returns an int, but int **
> (negative int) returns a float.
>
> - For arrays, int ** (positive int) and int ** (negative int) _have_
> to return the same type, because in general output types are always a
> function of the input types and *can't* look at the specific values
> involved, and in specific because if you do array([2, 3]) ** array([2,
> -2]) you can't return an array where the first element is int and the
> second is float.
>
> Given these immutable and contradictory constraints, the last bad
> option IMHO would be that we make int ** (negative int) an error in
> all cases, and the error message can suggest that instead of writing
>
>     np.array(2) ** -2
>
> they should instead write
>
>     np.array(2) ** -2.0
>
> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>
> Definitely annoying, but all the other options seem even more
> inconsistent and confusing, and likely to encourage the writing of
> subtly buggy code...
>
> (I especially have in mind numpy's habit of silently switching between
> scalars and zero-dimensional arrays -- so it's easy to write code that
> you think handles arbitrary array dimensions, and it even passes all
> your tests, but then it fails when someone passes in a different shape
> data and triggers some scalar/array inconsistency. E.g. if we make **
> -2 work for scalars but not arrays, then this code:
>
> def f(arr):
>     return np.sum(arr, axis=0) ** -2
>
> works as expected for 1-d input, tests pass, everyone's happy... but
> as soon as you try to pass in higher dimensional integer input it will
> fail.)
>
>
Hmm, the Alexandrian solution. The main difficulty with this solution that
this will likely to break working code. We could try it, or take the safe
route of raising a (Visible)DeprecationWarning. The other option is to
simply treat the negative power case uniformly as floor division and raise
an error on zero division, but the difference from Python power would be
highly confusing. I think I would vote for the second option with a
DeprecationWarning.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/87f5a966/attachment.html>

From charlesr.harris at gmail.com  Fri May 20 14:48:29 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 May 2016 12:48:29 -0600
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
Message-ID: <CAB6mnx+DB7BWU=ON2VkBpB+BfswjNxw4kpOfhV58WPy4y5Fs+g@mail.gmail.com>

On Fri, May 20, 2016 at 12:35 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Thu, May 19, 2016 at 9:30 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> So I guess what makes this tricky is that:
>>
>> - We want the behavior to the same for multiple-element arrays,
>> single-element arrays, zero-dimensional arrays, and scalars -- the
>> shape of the data shouldn't affect the semantics of **
>>
>> - We also want the numpy scalar behavior to match the Python scalar
>> behavior
>>
>> - For Python scalars, int ** (positive int) returns an int, but int **
>> (negative int) returns a float.
>>
>> - For arrays, int ** (positive int) and int ** (negative int) _have_
>> to return the same type, because in general output types are always a
>> function of the input types and *can't* look at the specific values
>> involved, and in specific because if you do array([2, 3]) ** array([2,
>> -2]) you can't return an array where the first element is int and the
>> second is float.
>>
>> Given these immutable and contradictory constraints, the last bad
>> option IMHO would be that we make int ** (negative int) an error in
>> all cases, and the error message can suggest that instead of writing
>>
>>     np.array(2) ** -2
>>
>> they should instead write
>>
>>     np.array(2) ** -2.0
>>
>> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>>
>> Definitely annoying, but all the other options seem even more
>> inconsistent and confusing, and likely to encourage the writing of
>> subtly buggy code...
>>
>> (I especially have in mind numpy's habit of silently switching between
>> scalars and zero-dimensional arrays -- so it's easy to write code that
>> you think handles arbitrary array dimensions, and it even passes all
>> your tests, but then it fails when someone passes in a different shape
>> data and triggers some scalar/array inconsistency. E.g. if we make **
>> -2 work for scalars but not arrays, then this code:
>>
>> def f(arr):
>>     return np.sum(arr, axis=0) ** -2
>>
>> works as expected for 1-d input, tests pass, everyone's happy... but
>> as soon as you try to pass in higher dimensional integer input it will
>> fail.)
>>
>>
> Hmm, the Alexandrian solution. The main difficulty with this solution that
> this will likely to break working code. We could try it, or take the safe
> route of raising a (Visible)DeprecationWarning. The other option is to
> simply treat the negative power case uniformly as floor division and raise
> an error on zero division, but the difference from Python power would be
> highly confusing. I think I would vote for the second option with a
> DeprecationWarning.
>
>
I suspect that the different behavior of int64 on my system is due to
inheritance from Python 2.7 int

In [1]: isinstance(int64(1), int)
Out[1]: True

That different behavior is also carried over for Python 3.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/492f0300/attachment.html>

From njs at pobox.com  Fri May 20 15:15:55 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 20 May 2016 12:15:55 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
Message-ID: <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>

On Fri, May 20, 2016 at 11:35 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, May 19, 2016 at 9:30 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> So I guess what makes this tricky is that:
>>
>> - We want the behavior to the same for multiple-element arrays,
>> single-element arrays, zero-dimensional arrays, and scalars -- the
>> shape of the data shouldn't affect the semantics of **
>>
>> - We also want the numpy scalar behavior to match the Python scalar
>> behavior
>>
>> - For Python scalars, int ** (positive int) returns an int, but int **
>> (negative int) returns a float.
>>
>> - For arrays, int ** (positive int) and int ** (negative int) _have_
>> to return the same type, because in general output types are always a
>> function of the input types and *can't* look at the specific values
>> involved, and in specific because if you do array([2, 3]) ** array([2,
>> -2]) you can't return an array where the first element is int and the
>> second is float.
>>
>> Given these immutable and contradictory constraints, the last bad
>> option IMHO would be that we make int ** (negative int) an error in
>> all cases, and the error message can suggest that instead of writing
>>
>>     np.array(2) ** -2
>>
>> they should instead write
>>
>>     np.array(2) ** -2.0
>>
>> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>>
>> Definitely annoying, but all the other options seem even more
>> inconsistent and confusing, and likely to encourage the writing of
>> subtly buggy code...
>>
>> (I especially have in mind numpy's habit of silently switching between
>> scalars and zero-dimensional arrays -- so it's easy to write code that
>> you think handles arbitrary array dimensions, and it even passes all
>> your tests, but then it fails when someone passes in a different shape
>> data and triggers some scalar/array inconsistency. E.g. if we make **
>> -2 work for scalars but not arrays, then this code:
>>
>> def f(arr):
>>     return np.sum(arr, axis=0) ** -2
>>
>> works as expected for 1-d input, tests pass, everyone's happy... but
>> as soon as you try to pass in higher dimensional integer input it will
>> fail.)
>>
>
> Hmm, the Alexandrian solution. The main difficulty with this solution that
> this will likely to break working code. We could try it, or take the safe
> route of raising a (Visible)DeprecationWarning.

Right, sorry, I was talking about the end goal -- there's a separate
question of how we get there. Pretty much any solution is going to
require some sort of deprecation cycle though I guess, and at least
the deprecate -> error transition is a lot easier than the working ->
working different transition.

> The other option is to
> simply treat the negative power case uniformly as floor division and raise
> an error on zero division, but the difference from Python power would be
> highly confusing. I think I would vote for the second option with a
> DeprecationWarning.

So "floor division" here would mean that k ** -n == 0 for all k and n
except for k == 1, right? In addition to the consistency issue, that
doesn't seem like a behavior that's very useful to anyone...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org


From charlesr.harris at gmail.com  Fri May 20 15:23:39 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 May 2016 13:23:39 -0600
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
 <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>
Message-ID: <CAB6mnxJQqvRjJeyBOh5va90AsRb9VQ5ZeX_5-iR2207m8ifokw@mail.gmail.com>

On Fri, May 20, 2016 at 1:15 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, May 20, 2016 at 11:35 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Thu, May 19, 2016 at 9:30 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> So I guess what makes this tricky is that:
> >>
> >> - We want the behavior to the same for multiple-element arrays,
> >> single-element arrays, zero-dimensional arrays, and scalars -- the
> >> shape of the data shouldn't affect the semantics of **
> >>
> >> - We also want the numpy scalar behavior to match the Python scalar
> >> behavior
> >>
> >> - For Python scalars, int ** (positive int) returns an int, but int **
> >> (negative int) returns a float.
> >>
> >> - For arrays, int ** (positive int) and int ** (negative int) _have_
> >> to return the same type, because in general output types are always a
> >> function of the input types and *can't* look at the specific values
> >> involved, and in specific because if you do array([2, 3]) ** array([2,
> >> -2]) you can't return an array where the first element is int and the
> >> second is float.
> >>
> >> Given these immutable and contradictory constraints, the last bad
> >> option IMHO would be that we make int ** (negative int) an error in
> >> all cases, and the error message can suggest that instead of writing
> >>
> >>     np.array(2) ** -2
> >>
> >> they should instead write
> >>
> >>     np.array(2) ** -2.0
> >>
> >> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
> >>
> >> Definitely annoying, but all the other options seem even more
> >> inconsistent and confusing, and likely to encourage the writing of
> >> subtly buggy code...
> >>
> >> (I especially have in mind numpy's habit of silently switching between
> >> scalars and zero-dimensional arrays -- so it's easy to write code that
> >> you think handles arbitrary array dimensions, and it even passes all
> >> your tests, but then it fails when someone passes in a different shape
> >> data and triggers some scalar/array inconsistency. E.g. if we make **
> >> -2 work for scalars but not arrays, then this code:
> >>
> >> def f(arr):
> >>     return np.sum(arr, axis=0) ** -2
> >>
> >> works as expected for 1-d input, tests pass, everyone's happy... but
> >> as soon as you try to pass in higher dimensional integer input it will
> >> fail.)
> >>
> >
> > Hmm, the Alexandrian solution. The main difficulty with this solution
> that
> > this will likely to break working code. We could try it, or take the safe
> > route of raising a (Visible)DeprecationWarning.
>
> Right, sorry, I was talking about the end goal -- there's a separate
> question of how we get there. Pretty much any solution is going to
> require some sort of deprecation cycle though I guess, and at least
> the deprecate -> error transition is a lot easier than the working ->
> working different transition.
>
> > The other option is to
> > simply treat the negative power case uniformly as floor division and
> raise
> > an error on zero division, but the difference from Python power would be
> > highly confusing. I think I would vote for the second option with a
> > DeprecationWarning.
>
> So "floor division" here would mean that k ** -n == 0 for all k and n
> except for k == 1, right? In addition to the consistency issue, that
> doesn't seem like a behavior that's very useful to anyone...
>

And -1 as well. The virtue is consistancy while deprecating. Or we could
just back out the current changes in master and throw in deprecation
warnings. That has the virtue of simplicity and not introducing possible
code breaks.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/442e19ff/attachment.html>

From josef.pktd at gmail.com  Fri May 20 15:44:52 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 20 May 2016 15:44:52 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAB6mnxJQqvRjJeyBOh5va90AsRb9VQ5ZeX_5-iR2207m8ifokw@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
 <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>
 <CAB6mnxJQqvRjJeyBOh5va90AsRb9VQ5ZeX_5-iR2207m8ifokw@mail.gmail.com>
Message-ID: <CAMMTP+ABHthT9uz0mnuoA5ay24NRLZhhdQZML_Fn=y_gEpSqsg@mail.gmail.com>

On Fri, May 20, 2016 at 3:23 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Fri, May 20, 2016 at 1:15 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Fri, May 20, 2016 at 11:35 AM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Thu, May 19, 2016 at 9:30 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> >>
>> >> So I guess what makes this tricky is that:
>> >>
>> >> - We want the behavior to the same for multiple-element arrays,
>> >> single-element arrays, zero-dimensional arrays, and scalars -- the
>> >> shape of the data shouldn't affect the semantics of **
>> >>
>> >> - We also want the numpy scalar behavior to match the Python scalar
>> >> behavior
>> >>
>> >> - For Python scalars, int ** (positive int) returns an int, but int **
>> >> (negative int) returns a float.
>> >>
>> >> - For arrays, int ** (positive int) and int ** (negative int) _have_
>> >> to return the same type, because in general output types are always a
>> >> function of the input types and *can't* look at the specific values
>> >> involved, and in specific because if you do array([2, 3]) ** array([2,
>> >> -2]) you can't return an array where the first element is int and the
>> >> second is float.
>> >>
>> >> Given these immutable and contradictory constraints, the last bad
>> >> option IMHO would be that we make int ** (negative int) an error in
>> >> all cases, and the error message can suggest that instead of writing
>> >>
>> >>     np.array(2) ** -2
>> >>
>> >> they should instead write
>> >>
>> >>     np.array(2) ** -2.0
>> >>
>> >> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>> >>
>> >> Definitely annoying, but all the other options seem even more
>> >> inconsistent and confusing, and likely to encourage the writing of
>> >> subtly buggy code...
>> >>
>> >> (I especially have in mind numpy's habit of silently switching between
>> >> scalars and zero-dimensional arrays -- so it's easy to write code that
>> >> you think handles arbitrary array dimensions, and it even passes all
>> >> your tests, but then it fails when someone passes in a different shape
>> >> data and triggers some scalar/array inconsistency. E.g. if we make **
>> >> -2 work for scalars but not arrays, then this code:
>> >>
>> >> def f(arr):
>> >>     return np.sum(arr, axis=0) ** -2
>> >>
>> >> works as expected for 1-d input, tests pass, everyone's happy... but
>> >> as soon as you try to pass in higher dimensional integer input it will
>> >> fail.)
>> >>
>> >
>> > Hmm, the Alexandrian solution. The main difficulty with this solution
>> that
>> > this will likely to break working code. We could try it, or take the
>> safe
>> > route of raising a (Visible)DeprecationWarning.
>>
>> Right, sorry, I was talking about the end goal -- there's a separate
>> question of how we get there. Pretty much any solution is going to
>> require some sort of deprecation cycle though I guess, and at least
>> the deprecate -> error transition is a lot easier than the working ->
>> working different transition.
>>
>> > The other option is to
>> > simply treat the negative power case uniformly as floor division and
>> raise
>> > an error on zero division, but the difference from Python power would be
>> > highly confusing. I think I would vote for the second option with a
>> > DeprecationWarning.
>>
>> So "floor division" here would mean that k ** -n == 0 for all k and n
>> except for k == 1, right? In addition to the consistency issue, that
>> doesn't seem like a behavior that's very useful to anyone...
>>
>
> And -1 as well. The virtue is consistancy while deprecating. Or we could
> just back out the current changes in master and throw in deprecation
> warnings. That has the virtue of simplicity and not introducing possible
> code breaks.
>


can numpy cast to float by default for power or **?

At least then we always get correct numbers.

Are there dominant usecases that require default return dtype int?
AFAICS, it's always possible to choose the return dtype in np.power.

Josef


>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/eeed09e3/attachment.html>

From alan.isaac at gmail.com  Fri May 20 16:22:05 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Fri, 20 May 2016 16:22:05 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
Message-ID: <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>

On 5/19/2016 11:30 PM, Nathaniel Smith wrote:
> the last bad
> option IMHO would be that we make int ** (negative int) an error in
> all cases, and the error message can suggest that instead of writing
>
>     np.array(2) ** -2
>
> they should instead write
>
>     np.array(2) ** -2.0
>
> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)



Fwiw, Haskell has three exponentiation operators
because of such ambiguities.  I don't use C, but
I think the contrasting decision there was to
always return a double, which has a clear attraction
since for any fixed-width integral type, most of the
possible input pairs overflow the type.

My core inclination would be to use (what I understand to be)
the C convention that integer exponentiation always produces
a double, but to support dtype-specific exponentiation with
a function.  But this is just a user's perspective.

Cheers,
Alan Isaac



From warren.weckesser at gmail.com  Fri May 20 16:27:19 2016
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Fri, 20 May 2016 16:27:19 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>
Message-ID: <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>

On Fri, May 20, 2016 at 4:22 PM, Alan Isaac <alan.isaac at gmail.com> wrote:

> On 5/19/2016 11:30 PM, Nathaniel Smith wrote:
>
>> the last bad
>> option IMHO would be that we make int ** (negative int) an error in
>> all cases, and the error message can suggest that instead of writing
>>
>>     np.array(2) ** -2
>>
>> they should instead write
>>
>>     np.array(2) ** -2.0
>>
>> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>>
>
>
>
> Fwiw, Haskell has three exponentiation operators
> because of such ambiguities.  I don't use C, but
> I think the contrasting decision there was to
> always return a double, which has a clear attraction
> since for any fixed-width integral type, most of the
> possible input pairs overflow the type.
>
> My core inclination would be to use (what I understand to be)
> the C convention that integer exponentiation always produces
> a double, but to support dtype-specific exponentiation with
> a function.



C doesn't have an exponentiation operator.  The C math library has pow,
powf and powl, which (like any C functions) are explicitly typed.

Warren


  But this is just a user's perspective.
>
> Cheers,
> Alan Isaac
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/e6c4377e/attachment.html>

From josef.pktd at gmail.com  Fri May 20 16:35:05 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 20 May 2016 16:35:05 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>
 <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>
Message-ID: <CAMMTP+AwJe012TOW5cyL2XF1UTG3ZsrQvzZ830Os+AhYQ0=7PA@mail.gmail.com>

On Fri, May 20, 2016 at 4:27 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
>
> On Fri, May 20, 2016 at 4:22 PM, Alan Isaac <alan.isaac at gmail.com> wrote:
>
>> On 5/19/2016 11:30 PM, Nathaniel Smith wrote:
>>
>>> the last bad
>>> option IMHO would be that we make int ** (negative int) an error in
>>> all cases, and the error message can suggest that instead of writing
>>>
>>>     np.array(2) ** -2
>>>
>>> they should instead write
>>>
>>>     np.array(2) ** -2.0
>>>
>>> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
>>>
>>
>>
>>
>> Fwiw, Haskell has three exponentiation operators
>> because of such ambiguities.  I don't use C, but
>> I think the contrasting decision there was to
>> always return a double, which has a clear attraction
>> since for any fixed-width integral type, most of the
>> possible input pairs overflow the type.
>>
>> My core inclination would be to use (what I understand to be)
>> the C convention that integer exponentiation always produces
>> a double, but to support dtype-specific exponentiation with
>> a function.
>
>
>
> C doesn't have an exponentiation operator.  The C math library has pow,
> powf and powl, which (like any C functions) are explicitly typed.
>
> Warren
>
>
>   But this is just a user's perspective.
>>
>> Cheers,
>> Alan Isaac
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

another question

uint are positive so the division problem doesn't show up. So that could
still handle a usecase for ints.

I'm getting stronger in favor of float because raising an exception (or
worse, nonsense) in half of the parameter spaces sounds ...  (maybe kind of
silly)

Josef
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/0a5515ca/attachment.html>

From m.h.vankerkwijk at gmail.com  Fri May 20 16:44:43 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Fri, 20 May 2016 16:44:43 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAMMTP+AwJe012TOW5cyL2XF1UTG3ZsrQvzZ830Os+AhYQ0=7PA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>
 <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>
 <CAMMTP+AwJe012TOW5cyL2XF1UTG3ZsrQvzZ830Os+AhYQ0=7PA@mail.gmail.com>
Message-ID: <CAJNV+9uxbBGJkK8vt8ryh7uEozz7wD8tp5UtX2=sM1UR3wvFOQ@mail.gmail.com>

Hi All,

As a final desired state, always returning float seems the best idea. It
seems quite similar to division in this regard, where integer division
works for some values, but not for all. This means not being quite
consistent with python, but as Nathan pointed out, one cannot have
value-dependent dtype's for arrays (and scalars should indeed behave the
same way).

If so, then like for division, in-place power should raise a `TypeError`.
?
Obviously, one could have a specific function for integers (like `//` for
division) for cases where it is really needed.

Now, how to get there...  Maybe we can learn from division? At least, I
guess at some point `np.divide` became equivalent to `np.true_divide`?

All the best,

Marten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/2337eb85/attachment.html>

From alan.isaac at gmail.com  Fri May 20 17:33:20 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Fri, 20 May 2016 17:33:20 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <61702e3a-b62e-8d87-ab58-d664b573aea3@gmail.com>
 <CAGzF1udG-ax8SodBxhLH_jn95XLAB1BBsaB7jZFNsu8E6d-iTA@mail.gmail.com>
Message-ID: <7815f2b0-1049-115c-c205-d8894a876397@gmail.com>

Yes, I was referring to `pow`,
but I had in mind the C++ version,
which is overloaded:
http://www.cplusplus.com/reference/cmath/pow/

Cheers,
Alan


On 5/20/2016 4:27 PM, Warren Weckesser wrote:
> C doesn't have an exponentiation operator.  The C math library has pow, powf and powl, which (like any C functions) are explicitly typed.



From njs at pobox.com  Fri May 20 18:54:41 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 20 May 2016 15:54:41 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAMMTP+ABHthT9uz0mnuoA5ay24NRLZhhdQZML_Fn=y_gEpSqsg@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
 <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>
 <CAB6mnxJQqvRjJeyBOh5va90AsRb9VQ5ZeX_5-iR2207m8ifokw@mail.gmail.com>
 <CAMMTP+ABHthT9uz0mnuoA5ay24NRLZhhdQZML_Fn=y_gEpSqsg@mail.gmail.com>
Message-ID: <CAPJVwB=yjZ0feqcYgjVSwsdqga+o6vn0-V2+L6LV+LvKCBMzkA@mail.gmail.com>

On May 20, 2016 12:44 PM, <josef.pktd at gmail.com> wrote:
[...]
>
> can numpy cast to float by default for power or **?

Maybe? The question is whether there are any valid use cases for getting
ints back:

>>> np.array([1, 2, 3]) ** 2
array([1, 4, 9])

It's not 100% obvious to me but intuitively this seems like an operation
that we probably want to support? Especially since there's a reasonable
range of int64 values that can't be represented exactly as floats.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/1eb8667e/attachment.html>

From Permafacture at gmail.com  Fri May 20 19:24:43 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Fri, 20 May 2016 18:24:43 -0500
Subject: [Numpy-discussion] A numpy based Entity-Component-System
Message-ID: <CAHedU1SQArspVHzFYCVDnFs7zqUgjGXiKqjC5r+c6b7HJmyYnw@mail.gmail.com>

I have a Data Oriented programing library I'm writing that uses the
Entity-Component-System model.

https://github.com/Permafacture/data-oriented-pyglet

I have initially called it Numpy-ECS but I don't know if that name is
okay.  The numpy license says:

Neither the name of the NumPy Developers nor the names of any contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.

but doesn't say anything about using the name of NumPy. Is this okay?

For anyone interested in what this project is, I suggest looking at the
project README.  Briefly, a Component is the Data Oriented version of what
Object Oriented calls an attribute (a Numpy array of values that represents
an attribute that Entities might have).  An Entity is the Data Oriented
version of an object instance (something subscribed to some subset of
Components, with an ID that can be used to access each instance's values in
those components).  A System is like an object method (a function that
operates on the Components of all Entities that have that Component).

This is something that one would use in a game engine or simulation.  It
allows you to have instances, but where the logic can be bulk applied to
all involved instances by operations on slices of Numpy arrays.  In an
example shown in this video <https://vimeo.com/66736654>, I went from 50
FPS in an object oriented approach to using ufuncs operating on shared
memory in multiprocessing and getting 180 space steps per second on the
physics and 500-600 FPS in the rendering.

Thanks for reading,
  Elliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/6938e530/attachment.html>

From josef.pktd at gmail.com  Fri May 20 20:11:20 2016
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 20 May 2016 20:11:20 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAPJVwB=yjZ0feqcYgjVSwsdqga+o6vn0-V2+L6LV+LvKCBMzkA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <CAB6mnx+8Z+qXLYZi3jaxmhUFEWNmU802=h4qcnX_H3Q8vqD2rA@mail.gmail.com>
 <CAPJVwB=gFDMvgp8NBrgO9=mXgg0R=xYm-CbT8Lt_ORoSCkMhGA@mail.gmail.com>
 <CAB6mnxJQqvRjJeyBOh5va90AsRb9VQ5ZeX_5-iR2207m8ifokw@mail.gmail.com>
 <CAMMTP+ABHthT9uz0mnuoA5ay24NRLZhhdQZML_Fn=y_gEpSqsg@mail.gmail.com>
 <CAPJVwB=yjZ0feqcYgjVSwsdqga+o6vn0-V2+L6LV+LvKCBMzkA@mail.gmail.com>
Message-ID: <CAMMTP+Dc_MTmFpaa-cD6_Kghq9-WVZMUHZ6ynFLkREhKqch-DA@mail.gmail.com>

On Fri, May 20, 2016 at 6:54 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On May 20, 2016 12:44 PM, <josef.pktd at gmail.com> wrote:
> [...]
> >
> > can numpy cast to float by default for power or **?
>
> Maybe? The question is whether there are any valid use cases for getting
> ints back:
>
> >>> np.array([1, 2, 3]) ** 2
> array([1, 4, 9])
>
> It's not 100% obvious to me but intuitively this seems like an operation
> that we probably want to support? Especially since there's a reasonable
> range of int64 values that can't be represented exactly as floats.
>

It would still be supported by the np.power function with dtype keyword for
users that want to strictly control the dtype.

The question is mainly for the operator ** which doesn't have options and
there I think it's more appropriate for users that want correct numbers but
not necessarily have or want to watch out for the dtype.


Related: Python 3.4 returns complex for (-1)**0.5 while numpy returns nan.
That's a similar case of upcasting if the result doesn't fit.

Longterm I think it would still be nice if numpy could do value dependent
type promotion. e.g. when a value is encountered that doesn't fit, then
upcast the result at the cost of possibly having to copy the existing
results.
In the current setting the user has to decide in advance what the necessary
maybe required dtype is supposed to be. (Of course I have no idea about the
technical problems or computational cost of this.)

(Julia dispatch seems to make it easier to construct new types. e.g. we
could have a flexible dtype that is free to upcast to whatever is required
for the calculation. just guessing)


practicality:
going from int to float is a common usecase and we would expect getting the
correct numbers  2**(-2) -> promote

complex is in most fields an unusual outcome for integer or float
calculations
(e.g. box-cox transformation for x>0 )  having suddenly complex numbers is
weird, getting nans is standard float response  -> don't promote


I'm still largely in the python 2.x habit of adding a decimal points to
numbers for float or a redundant `* 1.` in my code to avoid integer
division or other weirdness. So, I never realized that ** in numpy doesn't
always promote to float which I kind of thought it did.
Maybe it's not yet time to drop all the decimal points or `* 1.` from the
code?


Josef



> -n
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/3ad8bd6c/attachment.html>

From njs at pobox.com  Fri May 20 21:40:09 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 20 May 2016 18:40:09 -0700
Subject: [Numpy-discussion] A numpy based Entity-Component-System
In-Reply-To: <CAHedU1SQArspVHzFYCVDnFs7zqUgjGXiKqjC5r+c6b7HJmyYnw@mail.gmail.com>
References: <CAHedU1SQArspVHzFYCVDnFs7zqUgjGXiKqjC5r+c6b7HJmyYnw@mail.gmail.com>
Message-ID: <CAPJVwB=cEW6JT0nRm6_XvO=zO0qnrjmeH-z9z=+wa66X5e-neQ@mail.gmail.com>

On May 20, 2016 4:24 PM, "Elliot Hallmark" <Permafacture at gmail.com> wrote:
>
> I have a Data Oriented programing library I'm writing that uses the
Entity-Component-System model.
>
> https://github.com/Permafacture/data-oriented-pyglet
>
> I have initially called it Numpy-ECS but I don't know if that name is
okay.  The numpy license says:
>
> Neither the name of the NumPy Developers nor the names of any
contributors may be used to endorse or promote products derived from this
software without specific prior written permission.
>
> but doesn't say anything about using the name of NumPy. Is this okay?

Legally speaking, I'm pretty sure no one is going to sue you. That clause
in the license is generally taken to be mostly meaningless, because lying
and claiming that so-and-so endorses my project when they don't is
generally going to get me into legal trouble anyway, so the license text is
redundant. The main thing that would legally control use of the name
"numpy" is if we had a trademark on it, and I'm pretty sure no one has
claimed or registered one of those.

But legal issues aside, it'd probably be better to give your software a
more unique name? "Numpy-ECS" is potentially confusing (are we going to get
bug reports filed on it because it says "numpy"?), and, well, it's kind of
boring and generic, don't you think? Like naming your child
"Human-legsarms"?

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160520/e81f6869/attachment.html>

From yellowhat46 at gmail.com  Sat May 21 01:47:38 2016
From: yellowhat46 at gmail.com (Vasco Gervasi)
Date: Sat, 21 May 2016 07:47:38 +0200
Subject: [Numpy-discussion] FFT and reconstruct
In-Reply-To: <CANtTwcFbYeu930+YxfQCTD_bCOp_gFc-GGPLUSByh=wFDmL8Ag@mail.gmail.com>
References: <CANtTwcFbJt-OrQCir-G4c0MfppnjCP2qv=GW15yWVjWbd7aHHA@mail.gmail.com>
 <CANGhFjYiTYwN33PMbn6zkNESDw2FM+__UC_1EOOeq3kno4EbLg@mail.gmail.com>
 <CANtTwcFbYeu930+YxfQCTD_bCOp_gFc-GGPLUSByh=wFDmL8Ag@mail.gmail.com>
Message-ID: <CANtTwcGWdNo9yq65J6ML+LghqWVMLJRhOr+WNHZQD0g9nYNpiA@mail.gmail.com>

Maybe I found the problems;

1. t0=1.0, t1=3.0, y['1'] = cos(1.0*omega*t): I have to reconstruct the
signal using

>  yRec += a * cos(omega*i*(t-t0) + f)

not

>  yRec += a * cos(omega*i*t + f)


2.  t0=2, t1=3, y['Signal'] = 1.0*cos(1.0*omega*t) + ... +
5.0*cos(5.0*omega*t) + 1.0: starting point and end point must not be the
same, so to generate the signal I have to use

> t = linspace(t0, t1, 1000, endpoint=False)

not

> t = linspace(t0, t1, 1000)


Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160521/37d9371d/attachment.html>

From gfyoung17 at gmail.com  Sat May 21 21:05:23 2016
From: gfyoung17 at gmail.com (G Young)
Date: Sun, 22 May 2016 02:05:23 +0100
Subject: [Numpy-discussion] axis parameter for count_nonzero
Message-ID: <CAJ1_J5go3wwR2utK4UsVWR7TV72Qj7FfB969HarreNkAzype2Q@mail.gmail.com>

Hi,

I have had a PR <https://github.com/numpy/numpy/pull/7177> open (first
draft can be found here <https://github.com/numpy/numpy/pull/7138>) for
quite some time now that adds an 'axis' parameter to *count_nonzero*.
While the functionality is fully in-place, very robust, and actually
higher-performing than the original *count_nonzero* function, the obstacle
at this point is the implementation, as most of the functionality is now
surfaced at the Python level instead of at the C level.

I have made several attempts to move the code into C to no avail and have
not received much feedback from maintainers unfortunately to move this
forward, so I'm opening this up to the mailing list to see what you guys
think of the changes and whether or not it should be merged in as is or be
tabled until a more C-friendly solution can be found.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/a2462c31/attachment.html>

From gfyoung17 at gmail.com  Sat May 21 21:15:00 2016
From: gfyoung17 at gmail.com (G Young)
Date: Sun, 22 May 2016 02:15:00 +0100
Subject: [Numpy-discussion] broadcasting for randint
Message-ID: <CAJ1_J5gFs5c6eJmHhJJFz5H5zFNXFMq3XV6r1OhDmzx1JjT-dA@mail.gmail.com>

Hi,

I have had a PR <https://github.com/numpy/numpy/pull/6938> open for quite
some time now that allows arguments to broadcast in *randint*.  While the
functionality is fully in-place and very robust, the obstacle at this point
is the implementation.

When the *dtype* parameter was added to *randint* (see here
<https://github.com/numpy/numpy/pull/6910>), a big issue with the
implementation was that it created so much duplicate code that it would be
a huge maintenance nightmare.  However, this was dismissed in the original
PR message because it was believed that template-ing would be trivial,
which seemed reasonable at the time.

When I added broadcasting, I introduced a template system to the code that
dramatically cut down on the duplication.  However, the obstacle has been
whether or not this template system is too *ad hoc* to be merged into the
library.  Implementing a template in Cython was not considered sufficient
and is in fact very tricky to do, and unfortunately, I have not received
any constructive suggestions from maintainers about how to proceed, so I'm
opening this up to the mailing to see whether or not there are better
alternatives to what I did, whether this should be merged as it, or whether
this should be tabled until a better template can be found.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/64df80e6/attachment.html>

From ralf.gommers at gmail.com  Sun May 22 05:32:02 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 22 May 2016 11:32:02 +0200
Subject: [Numpy-discussion] axis parameter for count_nonzero
In-Reply-To: <CAJ1_J5go3wwR2utK4UsVWR7TV72Qj7FfB969HarreNkAzype2Q@mail.gmail.com>
References: <CAJ1_J5go3wwR2utK4UsVWR7TV72Qj7FfB969HarreNkAzype2Q@mail.gmail.com>
Message-ID: <CABL7CQj3MM891T2zMUGRLyQFws_20pj6JPbim3TV04-fgttQjw@mail.gmail.com>

On Sun, May 22, 2016 at 3:05 AM, G Young <gfyoung17 at gmail.com> wrote:

> Hi,
>
> I have had a PR <https://github.com/numpy/numpy/pull/7177> open (first
> draft can be found here <https://github.com/numpy/numpy/pull/7138>) for
> quite some time now that adds an 'axis' parameter to *count_nonzero*.
> While the functionality is fully in-place, very robust, and actually
> higher-performing than the original *count_nonzero* function, the
> obstacle at this point is the implementation, as most of the functionality
> is now surfaced at the Python level instead of at the C level.
>
> I have made several attempts to move the code into C to no avail and have
> not received much feedback from maintainers unfortunately to move this
> forward, so I'm opening this up to the mailing list to see what you guys
> think of the changes and whether or not it should be merged in as is or be
> tabled until a more C-friendly solution can be found.
>

The discussion is spread over several PRs/issues, so maybe a summary is
useful:

- adding an axis parameter was a feature request that was generally
approved of [1]
- writing the axis selection/validation code in C, like the rest of
count_nonzero, was preferred by several core devs
- Writing that C code turns out to be tricky. Jaime had a PR for doing this
for bincount [2], but closed it with final conclusion "the proper approach
seems to me to build some intermediate layer over nditer that abstracts the
complexity away".
- Julian pointed out that this adds a ufunc-like param, so why not add
other params like out/keepdims [3]
- Stephan points out that the current PR has quite a few branches, would
benefit from reusing a helper function (like _validate_axis, but that may
not do exactly the right thing), and that he doesn't want to merge it as is
without further input from other devs [4].

Points previously not raised that I can think of:
- count_nonzero is also in the C API [5], the axis parameter is now only
added to the Python API.
- Part of why the code in this PR is complex is to keep performance for
small arrays OK, but there's no benchmarks added or result given for the
existing benchmark [6]. A simple check with:
  x = np.arange(100)
  %timeit np.count_nonzero(x)
shows that that gets about 30x slower (330 ns vs 10.5 us on my machine).

It looks to me like performance is a concern, and if that can be resolved
there's the broader discussion of whether it's a good idea to merge this PR
at all. That's a trade-off of adding a useful feature vs. technical debt /
maintenance burden plus divergence Python/C API. Also, what do we do when
we merge this and then next week someone else sends a PR adding a keepdims
or out keyword? For these kinds of additions it would feel better if we
were sure that the new version is the final/desired one for the foreseeable
future.

Ralf


[1] https://github.com/numpy/numpy/issues/391
[2] https://github.com/numpy/numpy/pull/4330#issuecomment-77791250
[3] https://github.com/numpy/numpy/pull/7138#issuecomment-177202894
[4] https://github.com/numpy/numpy/pull/7177
[5]
http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_CountNonzero
[6]
https://github.com/numpy/numpy/blob/master/benchmarks/benchmarks/bench_ufunc.py#L70
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/323f561d/attachment.html>

From robert.kern at gmail.com  Sun May 22 05:35:50 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 22 May 2016 10:35:50 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAPJVwB=T1P27i2LDyXj3xdiZrJ4pEcq8Ac-pZQ7Ryh58YHDOTw@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CAPJVwB=T1P27i2LDyXj3xdiZrJ4pEcq8Ac-pZQ7Ryh58YHDOTw@mail.gmail.com>
Message-ID: <CAF6FJivq6iH9z+cQdwgJc3TLA2cd6Hu12cix=Su+JfN0j1i57w@mail.gmail.com>

On Wed, May 18, 2016 at 7:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
> On Wed, May 18, 2016 at 5:07 AM, Robert Kern <robert.kern at gmail.com>
wrote:
> > On Wed, May 18, 2016 at 1:14 AM, Nathaniel Smith <njs at pobox.com> wrote:

> >> ...anyway, the real reason I'm a bit grumpy is because there are solid
> >> engineering reasons why users *want* this API,
> >
> > I remain unconvinced on this mark. Grumpily.
>
> Sorry for getting grumpy :-).

And my apologies for some unwarranted hyperbole. I think we're both
converging on a reasonable approach, though.

> The engineering reasons seem pretty
> obvious to me though?

Well, I mean, engineers want lots of things. I suspect that most engineers
*really* just want to call `numpy.random.seed(8675309)` at the start and
never explicitly pass around separate streams. There's an upside to that in
terms of code simplicity. There are also significant limitations and
constraints. Ultimately, the upside against the alternative of passing
around RandomState objects is usually overweighed by the limitations, so
best practice is to pass around RandomState objects.

I acknowledge that there exists an upside to the splitting API, but I don't
think it's a groundbreaking improvement over the alternative current best
practice. It's also unclear to me how often situations that really
demonstrate the upside come into play; in my experience a lot of these
situations are already structured such that preallocating N streams is the
natural thing to do. The limitations and constraints are currently
underexplored, IMO; and in this conservative field, pessimism is warranted.

> If you have any use case for independent streams
> at all, and you're writing code that's intended to live inside a
> library's abstraction barrier, then you need some way to choose your
> streams to avoid colliding with arbitrary other code that the end-user
> might assemble alongside yours as part of their final program. So
> AFAICT you have two options: either you need a "tree-style" API for
> allocating these streams, or else you need to add some explicit API to
> your library that lets the end-user control in detail which streams
> you use. Both are possible, but the latter is obviously undesireable
> if you can avoid it, since it breaks the abstraction barrier, making
> your library more complicated to use and harder to evolve.

ACK

> >> so whether or not it
> >> turns out to be possible I think we should at least be allowed to have
> >> a discussion about whether there's some way to give it to them.
> >
> > I'm not shutting down discussion of the option. I *implemented* the
option.
> > I think that discussing whether it should be part of the main API is
> > premature. There probably ought to be a paper or three out there
supporting
> > its safety and utility first. Let the utility function version flourish
> > first.
>
> OK -- I guess this particularly makes sense given how
> extra-tightly-constrained we currently are in fixing mistakes in
> np.random. But I feel like in the end the right place for this really
> is inside the RandomState interface, because the person implementing
> RandomState is the one best placed to understand (a) the gnarly
> technical details here, and (b) how those change depending on the
> particular PRNG in use. I don't want to end up with a bunch of
> subtly-buggy utility functions in non-specialist libraries like dask
> -- so we should be trying to help downstream users figure out how to
> actually get this into np.random?

I think this is an open research area. An enterprising grad student could
milk this for a couple of papers analyzing how to do this safely for a
variety of PRNGs. I don't think we can hash this out in an email thread or
PR. So yeah, eventually there might be an API on RandomState for this, but
it's way too premature to do so right now, IMO. Maybe start with a
specialized subclass of RandomState that adds this experimental API. In
ng-numpy-randomstate. ;-)

But if someone has spare time to work on numpy.random, for God's sake, use
it to review @gfyoung's PRs instead.

> >> It's
> >> not even 100% out of the question that we conclude that existing PRNGs
> >> are buggy because they don't take this use case into account -- it
> >> would be far from the first time that numpy found itself going beyond
> >> the limits of older numerical tools that weren't designed to build the
> >> kind of large composable systems that numpy gets used for.
> >>
> >> MT19937's state space is large enough that you could explicitly encode
> >> a "tree seed" into it, even if you don't trust the laws of probability
> >> -- e.g., you start with a RandomState with id [], then its children
> >> have id [0], [1], [2], ..., and their children have ids [0, 0], [0,
> >> 1], ..., [1, 0], ..., and you write these into the state (probably
> >> after sending them through some bit-mixing permutation), to guarantee
> >> non-collision.
> >
> > Sure. Not entirely sure if that can be done without preallocating the
> > branching factor or depth, but I'm sure there's some fancy combinatoric
> > representation of an unbounded tree that could be exploited here. It
seems
> > likely to me that such a thing could be done with the stream IDs of
PRNGs
> > that support that.
>
> I'm pretty sure you do have to preallocate both the branching factor
> and depth, since getting the collision-free guarantee requires that
> across the universe of possible tree addresses, each state id gets
> used at most once -- and there are finitely many state ids. But as a
> practical matter, saying "you can only sprout up to 2**32 new states
> out of any given state, and you can only nest them 600 deep, and
> exceeding these bounds is an error" would still be "composable enough"
> for ~all practical purposes.

To be honest, I'd even be satisfied with taking the treepath breadcrumbs
and hashing them with a suitable cryptographic hash to get a stream ID.
For, say, PCG64 which has 2**127 settable streams, do
sha256(treepath.asbytes()) % (2**127). I'm okay with spray-and-pray if it's
using well-analyzed cryptographic primitives and enough settable streams.
Subject to actual testing, of course.

And that last point concerns me. PRNG quality testing is a dodgy subject as
it is, and the state of the art for testing parallel streams is even worse.
To be honest, making it more convenient to make new streams at a low
library level makes me leery. I do think this is something that needs to be
considered and designed at a high level, irrespective of the available APIs.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/0792dd7a/attachment.html>

From gfyoung17 at gmail.com  Sun May 22 08:35:40 2016
From: gfyoung17 at gmail.com (G Young)
Date: Sun, 22 May 2016 13:35:40 +0100
Subject: [Numpy-discussion] axis parameter for count_nonzero
In-Reply-To: <CABL7CQj3MM891T2zMUGRLyQFws_20pj6JPbim3TV04-fgttQjw@mail.gmail.com>
References: <CAJ1_J5go3wwR2utK4UsVWR7TV72Qj7FfB969HarreNkAzype2Q@mail.gmail.com>
 <CABL7CQj3MM891T2zMUGRLyQFws_20pj6JPbim3TV04-fgttQjw@mail.gmail.com>
Message-ID: <CAJ1_J5hxRuD4=kpc50vWaLvSmW4OFq4GJ-CtEbLUBi2wM32tYg@mail.gmail.com>

1) Correction: The PR was not written with small arrays in mind.  I ran
some new timing tests, and it does perform worse on smaller arrays but
appears to scale better than the current implementation.

2) Let me put it out there that I am not opposed to moving it to C, but
right now, there seems to be a large technical brick wall up against such
an implementation.  So suggestions about how to move the code into C would
be welcome too!

On Sun, May 22, 2016 at 10:32 AM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>
> On Sun, May 22, 2016 at 3:05 AM, G Young <gfyoung17 at gmail.com> wrote:
>
>> Hi,
>>
>> I have had a PR <https://github.com/numpy/numpy/pull/7177> open (first
>> draft can be found here <https://github.com/numpy/numpy/pull/7138>) for
>> quite some time now that adds an 'axis' parameter to *count_nonzero*.
>> While the functionality is fully in-place, very robust, and actually
>> higher-performing than the original *count_nonzero* function, the
>> obstacle at this point is the implementation, as most of the functionality
>> is now surfaced at the Python level instead of at the C level.
>>
>> I have made several attempts to move the code into C to no avail and have
>> not received much feedback from maintainers unfortunately to move this
>> forward, so I'm opening this up to the mailing list to see what you guys
>> think of the changes and whether or not it should be merged in as is or be
>> tabled until a more C-friendly solution can be found.
>>
>
> The discussion is spread over several PRs/issues, so maybe a summary is
> useful:
>
> - adding an axis parameter was a feature request that was generally
> approved of [1]
> - writing the axis selection/validation code in C, like the rest of
> count_nonzero, was preferred by several core devs
> - Writing that C code turns out to be tricky. Jaime had a PR for doing
> this for bincount [2], but closed it with final conclusion "the proper
> approach seems to me to build some intermediate layer over nditer that
> abstracts the complexity away".
> - Julian pointed out that this adds a ufunc-like param, so why not add
> other params like out/keepdims [3]
> - Stephan points out that the current PR has quite a few branches, would
> benefit from reusing a helper function (like _validate_axis, but that may
> not do exactly the right thing), and that he doesn't want to merge it as is
> without further input from other devs [4].
>
> Points previously not raised that I can think of:
> - count_nonzero is also in the C API [5], the axis parameter is now only
> added to the Python API.
> - Part of why the code in this PR is complex is to keep performance for
> small arrays OK, but there's no benchmarks added or result given for the
> existing benchmark [6]. A simple check with:
>   x = np.arange(100)
>   %timeit np.count_nonzero(x)
> shows that that gets about 30x slower (330 ns vs 10.5 us on my machine).
>
> It looks to me like performance is a concern, and if that can be resolved
> there's the broader discussion of whether it's a good idea to merge this PR
> at all. That's a trade-off of adding a useful feature vs. technical debt /
> maintenance burden plus divergence Python/C API. Also, what do we do when
> we merge this and then next week someone else sends a PR adding a keepdims
> or out keyword? For these kinds of additions it would feel better if we
> were sure that the new version is the final/desired one for the foreseeable
> future.
>
> Ralf
>
>
> [1] https://github.com/numpy/numpy/issues/391
> [2] https://github.com/numpy/numpy/pull/4330#issuecomment-77791250
> [3] https://github.com/numpy/numpy/pull/7138#issuecomment-177202894
> [4] https://github.com/numpy/numpy/pull/7177
> [5]
> http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_CountNonzero
> [6]
> https://github.com/numpy/numpy/blob/master/benchmarks/benchmarks/bench_ufunc.py#L70
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/d8a039f9/attachment.html>

From gfyoung17 at gmail.com  Sun May 22 13:36:34 2016
From: gfyoung17 at gmail.com (G Young)
Date: Sun, 22 May 2016 18:36:34 +0100
Subject: [Numpy-discussion] axis parameter for count_nonzero
In-Reply-To: <CAJ1_J5hxRuD4=kpc50vWaLvSmW4OFq4GJ-CtEbLUBi2wM32tYg@mail.gmail.com>
References: <CAJ1_J5go3wwR2utK4UsVWR7TV72Qj7FfB969HarreNkAzype2Q@mail.gmail.com>
 <CABL7CQj3MM891T2zMUGRLyQFws_20pj6JPbim3TV04-fgttQjw@mail.gmail.com>
 <CAJ1_J5hxRuD4=kpc50vWaLvSmW4OFq4GJ-CtEbLUBi2wM32tYg@mail.gmail.com>
Message-ID: <CAJ1_J5ijFXaiHEyDN0xTzkFJwH9n9cSyAaxF2Px70rWxH_+abA@mail.gmail.com>

After some discussion with *@rgommers*, I have simplified the code as
follows:

1) the path to the original count_nonzero in the C API is essentially
unchanged, save some small overhead with Python calling and the
if-statement to check the *axis* parameter

2) All of the complicated validation of the *axis* parameter and acrobatics
for getting the count is handled *only* after we cannot fast-track via a
numerical, boolean, or string *dtype*.

The question still remains whether or not leaving the *axis* parameter in
the Python API for now (given how complicated it is to add in the C API) is
acceptable.  I will say that in response to the concern of adding
parameters such as "out" and "keepdims" (should they be requested), we
could avail ourselves to functions like median
<https://github.com/numpy/numpy/blob/master/numpy/lib/function_base.py#L3528>
for
help as *@juliantaylor* pointed out.  The *scipy* library has dealt with
this problem as well in its *sparse* modules, so that is also a useful
resource.

On Sun, May 22, 2016 at 1:35 PM, G Young <gfyoung17 at gmail.com> wrote:

> 1) Correction: The PR was not written with small arrays in mind.  I ran
> some new timing tests, and it does perform worse on smaller arrays but
> appears to scale better than the current implementation.
>
> 2) Let me put it out there that I am not opposed to moving it to C, but
> right now, there seems to be a large technical brick wall up against such
> an implementation.  So suggestions about how to move the code into C would
> be welcome too!
>
> On Sun, May 22, 2016 at 10:32 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Sun, May 22, 2016 at 3:05 AM, G Young <gfyoung17 at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have had a PR <https://github.com/numpy/numpy/pull/7177> open (first
>>> draft can be found here <https://github.com/numpy/numpy/pull/7138>) for
>>> quite some time now that adds an 'axis' parameter to *count_nonzero*.
>>> While the functionality is fully in-place, very robust, and actually
>>> higher-performing than the original *count_nonzero* function, the
>>> obstacle at this point is the implementation, as most of the functionality
>>> is now surfaced at the Python level instead of at the C level.
>>>
>>> I have made several attempts to move the code into C to no avail and
>>> have not received much feedback from maintainers unfortunately to move this
>>> forward, so I'm opening this up to the mailing list to see what you guys
>>> think of the changes and whether or not it should be merged in as is or be
>>> tabled until a more C-friendly solution can be found.
>>>
>>
>> The discussion is spread over several PRs/issues, so maybe a summary is
>> useful:
>>
>> - adding an axis parameter was a feature request that was generally
>> approved of [1]
>> - writing the axis selection/validation code in C, like the rest of
>> count_nonzero, was preferred by several core devs
>> - Writing that C code turns out to be tricky. Jaime had a PR for doing
>> this for bincount [2], but closed it with final conclusion "the proper
>> approach seems to me to build some intermediate layer over nditer that
>> abstracts the complexity away".
>> - Julian pointed out that this adds a ufunc-like param, so why not add
>> other params like out/keepdims [3]
>> - Stephan points out that the current PR has quite a few branches, would
>> benefit from reusing a helper function (like _validate_axis, but that may
>> not do exactly the right thing), and that he doesn't want to merge it as is
>> without further input from other devs [4].
>>
>> Points previously not raised that I can think of:
>> - count_nonzero is also in the C API [5], the axis parameter is now only
>> added to the Python API.
>> - Part of why the code in this PR is complex is to keep performance for
>> small arrays OK, but there's no benchmarks added or result given for the
>> existing benchmark [6]. A simple check with:
>>   x = np.arange(100)
>>   %timeit np.count_nonzero(x)
>> shows that that gets about 30x slower (330 ns vs 10.5 us on my machine).
>>
>> It looks to me like performance is a concern, and if that can be resolved
>> there's the broader discussion of whether it's a good idea to merge this PR
>> at all. That's a trade-off of adding a useful feature vs. technical debt /
>> maintenance burden plus divergence Python/C API. Also, what do we do when
>> we merge this and then next week someone else sends a PR adding a keepdims
>> or out keyword? For these kinds of additions it would feel better if we
>> were sure that the new version is the final/desired one for the foreseeable
>> future.
>>
>> Ralf
>>
>>
>> [1] https://github.com/numpy/numpy/issues/391
>> [2] https://github.com/numpy/numpy/pull/4330#issuecomment-77791250
>> [3] https://github.com/numpy/numpy/pull/7138#issuecomment-177202894
>> [4] https://github.com/numpy/numpy/pull/7177
>> [5]
>> http://docs.scipy.org/doc/numpy/reference/c-api.array.html#c.PyArray_CountNonzero
>> [6]
>> https://github.com/numpy/numpy/blob/master/benchmarks/benchmarks/bench_ufunc.py#L70
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/5eabb7e5/attachment.html>

From m.h.vankerkwijk at gmail.com  Sun May 22 15:15:29 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Sun, 22 May 2016 15:15:29 -0400
Subject: [Numpy-discussion] Behavior of .reduceat()
In-Reply-To: <CAPOWHWnNqBPEPGwN6VtPsdSA=sCT2jg0_Tsj-WZgmGXzuN-VVQ@mail.gmail.com>
References: <CAPOWHWnNqBPEPGwN6VtPsdSA=sCT2jg0_Tsj-WZgmGXzuN-VVQ@mail.gmail.com>
Message-ID: <CAJNV+9uBYoYziwPkUc90Hk3+MhvOWpXPqpcArcjvrN0-Kwdovg@mail.gmail.com>

Hi Jaime,

Very belated reply, but only with the semester over I seem to have regained
some time to think.

The behaviour of reduceat always has seemed a bit odd to me, logical for
dividing up an array into irregular but contiguous pieces, but illogical
for more random ones (where one effectively passes in pairs of points, only
to remove the unwanted calculations after the fact by slicing with [::2];
indeed, the very first example in the documentation does exactly this [1]).
I'm not sure any of your proposals helps all that much for the latter case,
while it risks breaking existing code in unexpected ways.

For me, for irregular pieces, it would be much nicer to simply pass in
pairs of points. I think this can be quite easily done in the current API,
by expanding it to recognize multidimensional index arrays (with last
dimension of 2; maybe 3 for step as well?). These numbers would just be the
equivalent of start, end (and step?) of `slice`, so I think one can allow
any integer with negative values having the usual meaning and clipping at 0
and length. So, specifically, the first example in the documentation would
change from:

np.add.reduceat(np.arange(8),[0,4, 1,5, 2,6, 3,7])[::2]

to

np.add.reduceat(np.arange(8),[(0, 4), (1, 5), (2, 6), (3,7)])

(Or an equivalent ndarray. Note how horrid the example is: really, you'd
want 4,8 as a pair too, but in the current API, you'd get that by adding a
4.)

What do you think? Would this also be easy to implement?

All the best,

Marten

?[1]
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.reduceat.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160522/d07f5e6e/attachment.html>

From rainwoodman at gmail.com  Sun May 22 22:00:30 2016
From: rainwoodman at gmail.com (Feng Yu)
Date: Sun, 22 May 2016 19:00:30 -0700
Subject: [Numpy-discussion] Behavior of .reduceat()
In-Reply-To: <CAJNV+9uBYoYziwPkUc90Hk3+MhvOWpXPqpcArcjvrN0-Kwdovg@mail.gmail.com>
References: <CAPOWHWnNqBPEPGwN6VtPsdSA=sCT2jg0_Tsj-WZgmGXzuN-VVQ@mail.gmail.com>
 <CAJNV+9uBYoYziwPkUc90Hk3+MhvOWpXPqpcArcjvrN0-Kwdovg@mail.gmail.com>
Message-ID: <CANGhFjYRWOrEErm+=Sj_7NWL1JfRm5r-P7hn0MOcWijtwM4QVw@mail.gmail.com>

Hi Marten,

As a user of reduceat I seriously like your new proposal!

I notice that in your current proposal, each element in the 'at' list
shall be interpreted asif they are parameters to `slice`.

I wonder if it is meaningful to define reduceat on other `fancy` indexing types?

Cheers,

- Yu

On Sun, May 22, 2016 at 12:15 PM, Marten van Kerkwijk
<m.h.vankerkwijk at gmail.com> wrote:
> Hi Jaime,
>
> Very belated reply, but only with the semester over I seem to have regained
> some time to think.
>
> The behaviour of reduceat always has seemed a bit odd to me, logical for
> dividing up an array into irregular but contiguous pieces, but illogical for
> more random ones (where one effectively passes in pairs of points, only to
> remove the unwanted calculations after the fact by slicing with [::2];
> indeed, the very first example in the documentation does exactly this [1]).
> I'm not sure any of your proposals helps all that much for the latter case,
> while it risks breaking existing code in unexpected ways.
>
> For me, for irregular pieces, it would be much nicer to simply pass in pairs
> of points. I think this can be quite easily done in the current API, by
> expanding it to recognize multidimensional index arrays (with last dimension
> of 2; maybe 3 for step as well?). These numbers would just be the equivalent
> of start, end (and step?) of `slice`, so I think one can allow any integer
> with negative values having the usual meaning and clipping at 0 and length.
> So, specifically, the first example in the documentation would change from:
>
> np.add.reduceat(np.arange(8),[0,4, 1,5, 2,6, 3,7])[::2]
>
> to
>
> np.add.reduceat(np.arange(8),[(0, 4), (1, 5), (2, 6), (3,7)])
>
> (Or an equivalent ndarray. Note how horrid the example is: really, you'd
> want 4,8 as a pair too, but in the current API, you'd get that by adding a
> 4.)
>
> What do you think? Would this also be easy to implement?
>
> All the best,
>
> Marten
>
> [1]
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.reduceat.html
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ben.v.root at gmail.com  Mon May 23 11:17:29 2016
From: ben.v.root at gmail.com (Benjamin Root)
Date: Mon, 23 May 2016 11:17:29 -0400
Subject: [Numpy-discussion] A numpy based Entity-Component-System
In-Reply-To: <CAPJVwB=cEW6JT0nRm6_XvO=zO0qnrjmeH-z9z=+wa66X5e-neQ@mail.gmail.com>
References: <CAHedU1SQArspVHzFYCVDnFs7zqUgjGXiKqjC5r+c6b7HJmyYnw@mail.gmail.com>
 <CAPJVwB=cEW6JT0nRm6_XvO=zO0qnrjmeH-z9z=+wa66X5e-neQ@mail.gmail.com>
Message-ID: <CANNq6F=Egcn+XFT9ofUzmKGqbxE1m9Sx+E4nJ45wLHEhNTPpiQ@mail.gmail.com>

As a bit of a real-life example where things can go wrong with naming. The
"pylab" name was accidentally hijacked a couple years ago on pypi, and
caused several bug reports to be filed against matplotlib for failing
scripts.  Some people thought that one should do "pip install pylab" to do
"from pylab import *" -- crazy, I know, right? ;-)

That was at least two years ago, and we are just now getting the person who
uploaded that errant package to fix the problem. We also have not yet been
able to find out the owner of the long defunct matplotlib twitter handle
(we started the process to reclaim it, though). There are a couple other
situations that have caused confusion among users, but these have been
mostly an issue of trademarks (which I believe NumFOCUS holds for
matplotlib?).

What you really have to watch out for is when someone creates a package
that uses some open-sourced library, and then claims that it is "supported"
by it. This can cause an undue burden on the original maintainers in
fielding bug reports and other issues. It also creates a false sense of
association/coordination -- which gets me to the issue at hand. I highly
suggest coming up with a unique name. It benefits you as it becomes
something distinct from numpy (so search results are more relevant), and it
benefits the numpy developers because support requests go exactly where
they are supposed to go.


On Fri, May 20, 2016 at 9:40 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Like naming your child "Human-legsarms"?



You owe me a new monitor!

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160523/44fd8b8e/attachment.html>

From chris.barker at noaa.gov  Mon May 23 12:41:11 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 23 May 2016 09:41:11 -0700
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CAF6FJivq6iH9z+cQdwgJc3TLA2cd6Hu12cix=Su+JfN0j1i57w@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CAPJVwB=T1P27i2LDyXj3xdiZrJ4pEcq8Ac-pZQ7Ryh58YHDOTw@mail.gmail.com>
 <CAF6FJivq6iH9z+cQdwgJc3TLA2cd6Hu12cix=Su+JfN0j1i57w@mail.gmail.com>
Message-ID: <CALGmxEKAkOm4dy5CyQqqBVfn3fd+HRs0xS=Y0XAEBvPqZp5YXQ@mail.gmail.com>

On Sun, May 22, 2016 at 2:35 AM, Robert Kern <robert.kern at gmail.com> wrote:
>
> Well, I mean, engineers want lots of things. I suspect that most engineers
> *really* just want to call `numpy.random.seed(8675309)` at the start and
> never explicitly pass around separate streams. There's an upside to that in
> terms of code simplicity. There are also significant limitations and
> constraints. Ultimately, the upside against the alternative of passing
> around RandomState objects is usually overweighed by the limitations, so
> best practice is to pass around RandomState objects.
>

Could we do something like the logging module, and have numpy.random
"manage" a bunch of stream objects for you -- so you could get the default
single stream easily, and also get access to specific streams without
needing to pass around the objects?

That would allow folks to start off writing code the "easy" way -- then
make it easier to refactor when they realize they need multiple independent
streams.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160523/24ab7377/attachment.html>

From robert.kern at gmail.com  Mon May 23 12:54:02 2016
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 May 2016 17:54:02 +0100
Subject: [Numpy-discussion] Proposal: numpy.random.random_seed
In-Reply-To: <CALGmxEKAkOm4dy5CyQqqBVfn3fd+HRs0xS=Y0XAEBvPqZp5YXQ@mail.gmail.com>
References: <CAEQ_TvfKed1n6bZf0MfGJMmODw95qHDSF3xCOCaBbELzysE_VA@mail.gmail.com>
 <CAF6FJiuktnu6qe1vicMLG5o9cWXO6Ue1coYgFUFTz13tDOsbuw@mail.gmail.com>
 <CAEQ_Tvf2Tv65-P2jxj3iJPkb_KuFOW91N5APLohZ5ViaibKiMA@mail.gmail.com>
 <CAF6FJitXqO7WCjCZPp3+AnfMeUGy-bOeM-qHHUWb36kxBBT31A@mail.gmail.com>
 <CAPJVwBkOYcP=LK3tCQp4VmaJpWyvgKKrhJjWFvkfdt-U2sFFgA@mail.gmail.com>
 <CAF6FJiuOvkQzXexiSoRWytULgFPnHGjU3SdUc0QSAPUv8xn9Tw@mail.gmail.com>
 <CAPJVwB=NP2rXK9s0Wr3Gef+vrqZNHHU=TjHqJtrWFOrvTLbW2w@mail.gmail.com>
 <CAF6FJiv3V5Lve1bvhEHQDKrxrX-Zg6DPSyFb8=v0_GstCA-94g@mail.gmail.com>
 <CAPJVwB=T1P27i2LDyXj3xdiZrJ4pEcq8Ac-pZQ7Ryh58YHDOTw@mail.gmail.com>
 <CAF6FJivq6iH9z+cQdwgJc3TLA2cd6Hu12cix=Su+JfN0j1i57w@mail.gmail.com>
 <CALGmxEKAkOm4dy5CyQqqBVfn3fd+HRs0xS=Y0XAEBvPqZp5YXQ@mail.gmail.com>
Message-ID: <CAF6FJiuaDW82EDQ3=8mngkS4v=NKqfC7tEwAa8+UQ3CJY06vDA@mail.gmail.com>

On Mon, May 23, 2016 at 5:41 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> On Sun, May 22, 2016 at 2:35 AM, Robert Kern <robert.kern at gmail.com>
wrote:
>>
>> Well, I mean, engineers want lots of things. I suspect that most
engineers *really* just want to call `numpy.random.seed(8675309)` at the
start and never explicitly pass around separate streams. There's an upside
to that in terms of code simplicity. There are also significant limitations
and constraints. Ultimately, the upside against the alternative of passing
around RandomState objects is usually overweighed by the limitations, so
best practice is to pass around RandomState objects.
>
> Could we do something like the logging module, and have numpy.random
"manage" a bunch of stream objects for you -- so you could get the default
single stream easily, and also get access to specific streams without
needing to pass around the objects?

No, I don't think so. The logging module's namespacing doesn't really have
an equivalent use case for PRNGs. We would just be making a much more
complicated global state to manage.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160523/7294aab7/attachment.html>

From solipsis at pitrou.net  Tue May 24 07:22:51 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 May 2016 13:22:51 +0200
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
 <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>
 <20160512173810.79fb21b5@fsol>
 <115932182484786870.778106sturla.molden-gmail.com@news.gmane.org>
Message-ID: <20160524132251.1968fcc4@fsol>

On Thu, 12 May 2016 23:14:36 +0000 (UTC)
Sturla Molden <sturla.molden at gmail.com> wrote:
> Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > Can you define "expensive"?
> 
> Slow enough to cause complaints on the Cython mailing list.

What kind of complaints? Please be specific.

> > Buffer acquisition itself only calls a single C callback and uses a
> > stack-allocated C structure. It shouldn't be "expensive".
> 
> I don't know the reason, only that buffer acquisition from NumPy arrays
> with typed memoryviews

Again, what have memoryviews to do with it?  "Acquiring a buffer" is
just asking the buffer provider (the Numpy array) to fill a Py_buffer
structure's contents.  That has nothing to do with memoryviews.

When writing C code to interact with buffer-providing objects, you
usually don't bother with memoryviews at all.  You just use a Py_buffer
structure.

Regards

Antoine.




From solipsis at pitrou.net  Tue May 24 07:30:34 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 24 May 2016 13:30:34 +0200
Subject: [Numpy-discussion] Integers to integer powers
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
Message-ID: <20160524133034.642b60b1@fsol>

On Thu, 19 May 2016 20:30:40 -0700
Nathaniel Smith <njs at pobox.com> wrote:
> 
> Given these immutable and contradictory constraints, the last bad
> option IMHO would be that we make int ** (negative int) an error in
> all cases, and the error message can suggest that instead of writing
> 
>     np.array(2) ** -2
> 
> they should instead write
> 
>     np.array(2) ** -2.0
> 
> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
> 
> Definitely annoying, but all the other options seem even more
> inconsistent and confusing, and likely to encourage the writing of
> subtly buggy code...

That sounds like a good compromise, indeed.

Regards

Antoine.




From m.h.vankerkwijk at gmail.com  Tue May 24 10:03:59 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Tue, 24 May 2016 10:03:59 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <20160524133034.642b60b1@fsol>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
Message-ID: <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>

The error on int ** (-int) is OK with me too (even though I prefer just
returning floats).
Having a `floor_pow` function may still be good with this solution too.
-- Marten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/f61256c4/attachment.html>

From alan.isaac at gmail.com  Tue May 24 12:41:21 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Tue, 24 May 2016 12:41:21 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
Message-ID: <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>

On 5/24/2016 10:03 AM, Marten van Kerkwijk wrote:
> The error on int ** (-int) is OK with me too (even though I prefer just returning floats).


What exactly is the argument against *always* returning float
(even for positive integer exponents)?  Relevant context is:

- int ** int will often ("usually") overflow
- a numpy function cd meet specialized exponentiation needs

Thanks,
Alan Isaac



From shoyer at gmail.com  Tue May 24 13:19:35 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Tue, 24 May 2016 10:19:35 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
Message-ID: <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>

On Tue, May 24, 2016 at 9:41 AM, Alan Isaac <alan.isaac at gmail.com> wrote:

> What exactly is the argument against *always* returning float
> (even for positive integer exponents)?
>

If we were starting over from scratch, I would agree with you, but the int
** 2 example feels quite compelling to me. I would guess there lots of code
out there that expects the result to have integer dtype.

As a contrived example, I might write np.arange(n) ** 2 to produce an
indexer for the diagonal elements of an array.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/eba684c1/attachment.html>

From alan.isaac at gmail.com  Tue May 24 13:31:44 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Tue, 24 May 2016 13:31:44 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
Message-ID: <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>

On 5/24/2016 1:19 PM, Stephan Hoyer wrote:
> the int ** 2 example feels quite compelling to me


Yes, but that one case is trivial: a*a

And at least as compelling is not have a**-2 fail
and not being tricked by say np.arange(10)**10.
The latter is a promise of hidden errors.

Alan



From ewm at redtetrahedron.org  Tue May 24 13:36:33 2016
From: ewm at redtetrahedron.org (Eric Moore)
Date: Tue, 24 May 2016 13:36:33 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
Message-ID: <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>

I'd say the most compelling case for it is that is how it has always
worked.  How much code will break if we make that change? (Or if not break,
at least have a change in dtype?)  Is that worth it?

Eric

On Tue, May 24, 2016 at 1:31 PM, Alan Isaac <alan.isaac at gmail.com> wrote:

> On 5/24/2016 1:19 PM, Stephan Hoyer wrote:
>
>> the int ** 2 example feels quite compelling to me
>>
>
>
> Yes, but that one case is trivial: a*a
>
> And at least as compelling is not have a**-2 fail
> and not being tricked by say np.arange(10)**10.
> The latter is a promise of hidden errors.
>
> Alan
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/c4354b02/attachment.html>

From shoyer at gmail.com  Tue May 24 13:41:54 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Tue, 24 May 2016 10:41:54 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
Message-ID: <CAEQ_TvcTJbEBtNCL=fBJifQrVDR73fHPENLQhPFkNQ-9daHPsQ@mail.gmail.com>

On Tue, May 24, 2016 at 10:31 AM, Alan Isaac <alan.isaac at gmail.com> wrote:

> Yes, but that one case is trivial: a*a


an_explicit_name ** 2 is much better than an_explicit_name *
an_explicit_name, though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/f6a5c613/attachment.html>

From matthew.brett at gmail.com  Tue May 24 13:41:29 2016
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 24 May 2016 13:41:29 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
Message-ID: <CAH6Pt5oYq9RPUx1A0bXNfBWe_ka+QhC3mTXGmywJcKa2S36O9w@mail.gmail.com>

On Tue, May 24, 2016 at 1:31 PM, Alan Isaac <alan.isaac at gmail.com> wrote:
> On 5/24/2016 1:19 PM, Stephan Hoyer wrote:
>>
>> the int ** 2 example feels quite compelling to me
>
>
>
> Yes, but that one case is trivial: a*a

Right, but you'd have to know to change your code when numpy makes
this change.   Your code will suddenly have a new datatype, that you
may be passing into other code, which may generate different results,
that will be hard to track down.

> And at least as compelling is not have a**-2 fail

I imagine that's about an order of magnitude less common in real code,
but an error seems an acceptable result to me, so I know I have to do
a ** -2.0

> and not being tricked by say np.arange(10)**10.
> The latter is a promise of hidden errors.

It's a well-understood promise though - you always have to be careful
of integer overflow.

Best,

Matthew


From alan.isaac at gmail.com  Tue May 24 13:49:21 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Tue, 24 May 2016 13:49:21 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
Message-ID: <da6f21a7-8b24-8eac-b8d5-1eba4db0238c@gmail.com>

On 5/24/2016 1:36 PM, Eric Moore wrote:
> How much code will break if we make that change?


Since even arange(10)**10 is already broken,
there will probably not be much new breakage.

But under any of the new proposals, *something* will break.
So the question is, which shows the most foresight?
Having (**) actually work seems worth quite a lot.

Alan Isaac



From alan.isaac at gmail.com  Tue May 24 13:53:11 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Tue, 24 May 2016 13:53:11 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAH6Pt5oYq9RPUx1A0bXNfBWe_ka+QhC3mTXGmywJcKa2S36O9w@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAH6Pt5oYq9RPUx1A0bXNfBWe_ka+QhC3mTXGmywJcKa2S36O9w@mail.gmail.com>
Message-ID: <dae7cfe8-2582-3529-918a-f27ae7883d04@gmail.com>

On 5/24/2016 1:41 PM, Matthew Brett wrote:
> It's a well-understood promise though - you always have to be careful
> of integer overflow.


Of course.  But **almost all** cases overflow.

And "well understood" assume a certain sophistication
of the user, while `arange` will certainly be used
by beginners.

Alan


From njs at pobox.com  Tue May 24 15:05:26 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 24 May 2016 12:05:26 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
Message-ID: <CAPJVwBkmz2Gsd+QRqCcpWWfhmRE3eH2TWeAO42TOc7i91XWOxg@mail.gmail.com>

On Tue, May 24, 2016 at 10:36 AM, Eric Moore <ewm at redtetrahedron.org> wrote:
> I'd say the most compelling case for it is that is how it has always
worked.
> How much code will break if we make that change? (Or if not break, at
least
> have a change in dtype?)  Is that worth it?

The current behavior for arrays is:

# Returns int
In [2]: np.arange(10) ** 2
Out[2]: array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])

# Returns nonsensical/useless results
In [3]: np.arange(10) ** -1
/home/njs/.user-python3.5-64bit/bin/ipython:1: RuntimeWarning: divide by
zero encountered in power
  #!/home/njs/.user-python3.5-64bit/bin/python3.5
/home/njs/.user-python3.5-64bit/bin/ipython:1: RuntimeWarning: invalid
value encountered in power
  #!/home/njs/.user-python3.5-64bit/bin/python3.5
Out[3]:
array([-9223372036854775808,                    1,                    0,
                          0,                    0,                    0,
                          0,                    0,                    0,
                          0])

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/dc900dbd/attachment.html>

From ewm at redtetrahedron.org  Tue May 24 15:57:53 2016
From: ewm at redtetrahedron.org (Eric Moore)
Date: Tue, 24 May 2016 15:57:53 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAPJVwBkmz2Gsd+QRqCcpWWfhmRE3eH2TWeAO42TOc7i91XWOxg@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
 <CAPJVwBkmz2Gsd+QRqCcpWWfhmRE3eH2TWeAO42TOc7i91XWOxg@mail.gmail.com>
Message-ID: <CAGeA38kTU1Ef4-DQHKr6j4aHrWdw8Ai_aaeeEtFp5JXHYhJPoA@mail.gmail.com>

Yes, I'm fully aware of that.  I'm speaking toward changing the default
result dtype.  Raising an error for negative exponents is a fine idea.
Changing np.arange(10)**3 to have a non-integer dtype seems like a big
change.

Speaking of this, that some of the integer array operation errors can be
controlled via the np.seterr and some cannot should also be addressed
longer term.

Eric

On Tue, May 24, 2016 at 3:05 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, May 24, 2016 at 10:36 AM, Eric Moore <ewm at redtetrahedron.org>
> wrote:
> > I'd say the most compelling case for it is that is how it has always
> worked.
> > How much code will break if we make that change? (Or if not break, at
> least
> > have a change in dtype?)  Is that worth it?
>
> The current behavior for arrays is:
>
> # Returns int
> In [2]: np.arange(10) ** 2
> Out[2]: array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])
>
> # Returns nonsensical/useless results
> In [3]: np.arange(10) ** -1
> /home/njs/.user-python3.5-64bit/bin/ipython:1: RuntimeWarning: divide by
> zero encountered in power
>   #!/home/njs/.user-python3.5-64bit/bin/python3.5
> /home/njs/.user-python3.5-64bit/bin/ipython:1: RuntimeWarning: invalid
> value encountered in power
>   #!/home/njs/.user-python3.5-64bit/bin/python3.5
> Out[3]:
> array([-9223372036854775808,                    1,                    0,
>                           0,                    0,                    0,
>                           0,                    0,                    0,
>                           0])
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/5b98d0b8/attachment.html>

From alan.isaac at gmail.com  Tue May 24 16:15:27 2016
From: alan.isaac at gmail.com (Alan Isaac)
Date: Tue, 24 May 2016 16:15:27 -0400
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <CAGeA38kTU1Ef4-DQHKr6j4aHrWdw8Ai_aaeeEtFp5JXHYhJPoA@mail.gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
 <CAPJVwBkmz2Gsd+QRqCcpWWfhmRE3eH2TWeAO42TOc7i91XWOxg@mail.gmail.com>
 <CAGeA38kTU1Ef4-DQHKr6j4aHrWdw8Ai_aaeeEtFp5JXHYhJPoA@mail.gmail.com>
Message-ID: <020de0da-4ce2-0b0b-f2eb-bc77bbf589ab@gmail.com>

On 5/24/2016 3:57 PM, Eric Moore wrote:
> Changing np.arange(10)**3 to have a non-integer dtype seems like a big change.


What about np.arange(100)**5?

Alan Isaac


From rays at blue-cove.com  Tue May 24 16:33:20 2016
From: rays at blue-cove.com (R Schumacher)
Date: Tue, 24 May 2016 13:33:20 -0700
Subject: [Numpy-discussion] Integers to integer powers
In-Reply-To: <020de0da-4ce2-0b0b-f2eb-bc77bbf589ab@gmail.com>
References: <CAB6mnxKOdm+CUz_joTh5tccwi3NsrvtZynLZ7ZhhAJNu4gPnqA@mail.gmail.com>
 <CAPJVwBkpxW1QXTDz++vnWEuKotnMd8AxZfK1Jkv1W5cMjGGytQ@mail.gmail.com>
 <20160524133034.642b60b1@fsol>
 <CAJNV+9u4LLwcfGj-EsvqLk9NmkAyUj5VQDd8HsDEoKG1=W4tPg@mail.gmail.com>
 <7c0663b6-9d2a-2a0f-e0bf-71e19e30c5b4@gmail.com>
 <CAEQ_Tvfv-pQ2_TETQwxPhv+KaWHj+vhPtSfu=jTZcLc+gjV2-g@mail.gmail.com>
 <8f0576d6-53fc-0fbd-cd2e-3daa72076731@gmail.com>
 <CAGeA38=qvaLL8jRJTAsWyxnPjRgmirjFm8tht_vZgpn0s1iDmg@mail.gmail.com>
 <CAPJVwBkmz2Gsd+QRqCcpWWfhmRE3eH2TWeAO42TOc7i91XWOxg@mail.gmail.com>
 <CAGeA38kTU1Ef4-DQHKr6j4aHrWdw8Ai_aaeeEtFp5JXHYhJPoA@mail.gmail.com>
 <020de0da-4ce2-0b0b-f2eb-bc77bbf589ab@gmail.com>
Message-ID: <201605242033.u4OKXRLf029822@blue-cove.com>

At 01:15 PM 5/24/2016, you wrote:
>On 5/24/2016 3:57 PM, Eric Moore wrote:
>>Changing np.arange(10)**3 to have a non-integer dtype seems like a 
>>big change.
>
>
>What about np.arange(100)**5?

Interesting, one warning per instantiation (Py2.7):

 >>> import numpy
 >>> a=numpy.arange(100)**5
<string>:1: RuntimeWarning: invalid value encountered in power
 >>> a=numpy.arange(100)**5.
 >>> b=numpy.arange(100.)**5
 >>> a==b
array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True, 
True,  True], dtype=bool)
 >>> numpy.arange(100)**5
array([          0,           1,          32,         243,        1024,
               3125,        7776,       16807,       32768,       59049,
             100000,      161051,      248832,      371293,      537824,
             759375,     1048576,     1419857,     1889568,     2476099,
            3200000,     4084101,     5153632,     6436343,     7962624,
            9765625,    11881376,    14348907,    17210368,    20511149,
           24300000,    28629151,    33554432,    39135393,    45435424,
           52521875,    60466176,    69343957,    79235168,    90224199,
          102400000,   115856201,   130691232,   147008443,   164916224,
          184528125,   205962976,   229345007,   254803968,   282475249,
          312500000,   345025251,   380204032,   418195493,   459165024,
          503284375,   550731776,   601692057,   656356768,   714924299,
          777600000,   844596301,   916132832,   992436543,  1073741824,
         1160290625,  1252332576,  1350125107,  1453933568,  1564031349,
         1680700000,  1804229351,  1934917632,  2073071593, -2147483648,
        -2147483648, -2147483648, -2147483648, -2147483648, -2147483648,
        -2147483648, -2147483648, -2147483648, -2147483648, -2147483648,
        -2147483648, -2147483648, -2147483648, -2147483648, -2147483648,
        -2147483648, -2147483648, -2147483648, -2147483648, -2147483648,
        -2147483648, -2147483648, -2147483648, -2147483648, -2147483648])
 >>>
 >>> numpy.arange(100, dtype=numpy.int64)**5
array([         0,          1,         32,        243,       1024,
              3125,       7776,      16807,      32768,      59049,
            100000,     161051,     248832,     371293,     537824,
            759375,    1048576,    1419857,    1889568,    2476099,
           3200000,    4084101,    5153632,    6436343,    7962624,
           9765625,   11881376,   14348907,   17210368,   20511149,
          24300000,   28629151,   33554432,   39135393,   45435424,
          52521875,   60466176,   69343957,   79235168,   90224199,
         102400000,  115856201,  130691232,  147008443,  164916224,
         184528125,  205962976,  229345007,  254803968,  282475249,
         312500000,  345025251,  380204032,  418195493,  459165024,
         503284375,  550731776,  601692057,  656356768,  714924299,
         777600000,  844596301,  916132832,  992436543, 1073741824,
        1160290625, 1252332576, 1350125107, 1453933568, 1564031349,
        1680700000, 1804229351, 1934917632, 2073071593, 2219006624,
        2373046875, 2535525376, 2706784157, 2887174368, 3077056399,
        3276800000, 3486784401, 3707398432, 3939040643, 4182119424,
        4437053125, 4704270176, 4984209207, 5277319168, 5584059449,
        5904900000, 6240321451, 6590815232, 6956883693, 7339040224,
        7737809375, 8153726976, 8587340257, 9039207968, 9509900499], 
dtype=int64) 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160524/57aca92a/attachment.html>

From sturla.molden at gmail.com  Tue May 24 17:03:44 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 24 May 2016 21:03:44 +0000 (UTC)
Subject: [Numpy-discussion] Numpy arrays shareable among related
	processes (PR #7533)
References: <CAHedU1QsUNoCY2qTaf_=tnqEAGMZo1wa2A61ueDS2xKz=dF=hg@mail.gmail.com>
 <274831917484699028.356959sturla.molden-gmail.com@news.gmane.org>
 <5733C086.9020403@gmail.com>
 <732428194484726233.271444sturla.molden-gmail.com@news.gmane.org>
 <20160512173810.79fb21b5@fsol>
 <115932182484786870.778106sturla.molden-gmail.com@news.gmane.org>
 <20160524132251.1968fcc4@fsol>
Message-ID: <666174225485816386.389704sturla.molden-gmail.com@news.gmane.org>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> When writing C code to interact with buffer-providing objects, you
> usually don't bother with memoryviews at all.  You just use a Py_buffer
> structure.

I was taking about "typed memoryviews" which is a Cython abstraction for a
Py_buffer struct. I was not taking about Python memoryviews, which is
something else. When writing Cython code that interacts with a buffer we
usually use typed memoryviews, not Py_buffer structs directly.

Sturla



From allen.welkie at gmail.com  Wed May 25 15:35:02 2016
From: allen.welkie at gmail.com (Allen Welkie)
Date: Wed, 25 May 2016 15:35:02 -0400
Subject: [Numpy-discussion] Fwd: ifft padding
In-Reply-To: <CAMwDPXtimpx3uZ-QFzwVAYEsTQiDms2JJ2Jc3yfvgVnq5xG0NA@mail.gmail.com>
References: <CAMwDPXtimpx3uZ-QFzwVAYEsTQiDms2JJ2Jc3yfvgVnq5xG0NA@mail.gmail.com>
Message-ID: <CAMwDPXuazVaD1KzY1=f+3iSVCY1on7cFnEeW70MX8QhmiPxnLA@mail.gmail.com>

I'd like to get some feedback on my [pull request](
https://github.com/numpy/numpy/pull/7593).

This pull request adds a function `ifftpad` which pads a spectrum by
inserting zeros where the highest frequencies would be. This is necessary
because the padding that `ifft` does simply inserts zeros at the end of the
array. But because of the way the spectrum is laid out, this changes which
bins represent which frequencies and in general messes up the result of
`ifft`. If you pad with the proposed `ifftpad` function, the zeros will be
inserted in the middle of the spectrum and the time signal that results
from `ifft` will be an interpolated version of the unpadded time signal.
See the discussion in [issue #1346](
https://github.com/numpy/numpy/issues/1346).

The following is a script to demonstrate what I mean:

```
import numpy
from numpy import concatenate, zeros
from matplotlib import pyplot

def correct_padding(a, n, scale=True):
    """ A copy of the proposed `ifftpad` function. """
    spectrum = concatenate((a[:len(a) // 2],
                            zeros(n - len(a)),
                            a[len(a) // 2:]))
    if scale:
        spectrum *= (n / len(a))
    return spectrum

def plot_real(signal, label):
    time = numpy.linspace(0, 1, len(signal) + 1)[:-1]
    pyplot.plot(time, signal.real, label=label)

def main():
    spectrum = numpy.zeros(10, dtype=complex)
    spectrum[-1] = 1 + 1j

    signal = numpy.fft.ifft(spectrum)
    signal_bad_padding = numpy.fft.ifft(10 * spectrum, 100)
    signal_good_padding = numpy.fft.ifft(correct_padding(spectrum, 100))

    plot_real(signal, 'No padding')
    plot_real(signal_bad_padding, 'Bad padding')
    plot_real(signal_good_padding, 'Good padding')

    pyplot.legend()
    pyplot.show()


if __name__ == '__main__':
    main()
```
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160525/507e53fd/attachment.html>

From leon.woo at db.com  Thu May 26 08:02:34 2016
From: leon.woo at db.com (Leon Woo)
Date: Thu, 26 May 2016 14:02:34 +0200
Subject: [Numpy-discussion] AUTO: Leon Woo is out of the office (returning
	06/06/2016)
Message-ID: <OFA93746A0.DD96D1DB-ONC1257FBF.0042275D-C1257FBF.0042275D@db.com>



I am out of the office until 06/06/2016.

For standard requests within the scope of EMG PWM Berlin, please write to
EMG PWM Berlin at DBEMEA.
For non-standard requests, please cc Hien Pham-Thu.


Note: This is an automated response to your message  "NumPy-Discussion
Digest, Vol 116, Issue 39" sent on 26.05.2016 14:00:01.

This is the only notification you will receive while this person is away.-- 

Informationen (einschlie?lich Pflichtangaben) zu einzelnen, innerhalb der EU t?tigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enth?lt vertrauliche und/ oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160526/b9bdca6c/attachment.html>

From robbmcleod at gmail.com  Thu May 26 08:28:38 2016
From: robbmcleod at gmail.com (Robert McLeod)
Date: Thu, 26 May 2016 14:28:38 +0200
Subject: [Numpy-discussion] Fwd: ifft padding
In-Reply-To: <CAMwDPXuazVaD1KzY1=f+3iSVCY1on7cFnEeW70MX8QhmiPxnLA@mail.gmail.com>
References: <CAMwDPXtimpx3uZ-QFzwVAYEsTQiDms2JJ2Jc3yfvgVnq5xG0NA@mail.gmail.com>
 <CAMwDPXuazVaD1KzY1=f+3iSVCY1on7cFnEeW70MX8QhmiPxnLA@mail.gmail.com>
Message-ID: <CAEFUWWUkZ_TVeL-+EoJOr7PcbCNRFAJcr8pxU+cY8=EQtfckTg@mail.gmail.com>

Allen,

Probably it needs to work in n-dimensions, like the existing
np.fft.fftshift function does, with an optional axis=tuple parameter. I
recall that fftshift is just an array indexing trick?  It would be helpful
to see what's faster, two fftshifts and a edge padding or your
inter-padding.  Probably it's faster to make a new zeros array of the
appropriate padded size and do 2*ndim copies?

Robert

On Wed, May 25, 2016 at 9:35 PM, Allen Welkie <allen.welkie at gmail.com>
wrote:

> I'd like to get some feedback on my [pull request](
> https://github.com/numpy/numpy/pull/7593).
>
> This pull request adds a function `ifftpad` which pads a spectrum by
> inserting zeros where the highest frequencies would be. This is necessary
> because the padding that `ifft` does simply inserts zeros at the end of the
> array. But because of the way the spectrum is laid out, this changes which
> bins represent which frequencies and in general messes up the result of
> `ifft`. If you pad with the proposed `ifftpad` function, the zeros will be
> inserted in the middle of the spectrum and the time signal that results
> from `ifft` will be an interpolated version of the unpadded time signal.
> See the discussion in [issue #1346](
> https://github.com/numpy/numpy/issues/1346).
>
> The following is a script to demonstrate what I mean:
>
> ```
> import numpy
> from numpy import concatenate, zeros
> from matplotlib import pyplot
>
> def correct_padding(a, n, scale=True):
>     """ A copy of the proposed `ifftpad` function. """
>     spectrum = concatenate((a[:len(a) // 2],
>                             zeros(n - len(a)),
>                             a[len(a) // 2:]))
>     if scale:
>         spectrum *= (n / len(a))
>     return spectrum
>
> def plot_real(signal, label):
>     time = numpy.linspace(0, 1, len(signal) + 1)[:-1]
>     pyplot.plot(time, signal.real, label=label)
>
> def main():
>     spectrum = numpy.zeros(10, dtype=complex)
>     spectrum[-1] = 1 + 1j
>
>     signal = numpy.fft.ifft(spectrum)
>     signal_bad_padding = numpy.fft.ifft(10 * spectrum, 100)
>     signal_good_padding = numpy.fft.ifft(correct_padding(spectrum, 100))
>
>     plot_real(signal, 'No padding')
>     plot_real(signal_bad_padding, 'Bad padding')
>     plot_real(signal_good_padding, 'Good padding')
>
>     pyplot.legend()
>     pyplot.show()
>
>
> if __name__ == '__main__':
>     main()
> ```
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Robert McLeod, Ph.D.
Center for Cellular Imaging and Nano Analytics (C-CINA)
Biozentrum der Universit?t Basel
Mattenstrasse 26, 4058 Basel
Work: +41.061.387.3225
robert.mcleod at unibas.ch
robert.mcleod at bsse.ethz.ch <robert.mcleod at ethz.ch>
robbmcleod at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160526/39512198/attachment.html>

From charlesr.harris at gmail.com  Thu May 26 22:42:02 2016
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 May 2016 20:42:02 -0600
Subject: [Numpy-discussion] Numpy 1.11.1rc1 release
Message-ID: <CAB6mnxLjtKjP4suuSJfZ8=0eXvbvOTFDh8XjbBRqw6wmO1XOAg@mail.gmail.com>

Hi All,

I am pleased to announce the release of Numpy 1.11.1rc1. The sources may be
found on sourceforge
<https://sourceforge.net/projects/numpy/files/NumPy/1.11.1rc1/> and wheels
for OS X, Windows, and Linux will be available on pypi sometime in the next
few days. The pypi release is delayed due to the decision that the wheels
should go up before the sources in order that people not get a source
install when what they want are wheels. The Python versions supported are
2.6-2.7 and 3.2-3.5.

This release has mostly small fixes and build enhancements and should be
good out of the starting gate, but prudence requires a release candidate as
there are a few bits not tested in master. The following fixes have been
applied:

   - #7506 BUG: Make sure numpy imports on python 2.6 when nose is
   unavailable.
   - #7530 BUG: Floating exception with invalid axis in np.lexsort.
   - #7535 BUG: Extend glibc complex trig functions blacklist to glibc <
   2.18.
   - #7551 BUG: Allow graceful recovery for no compiler.
   - #7558 BUG: Constant padding expected wrong type in constant_values.
   - #7578 BUG: Fix OverflowError in Python 3.x. in swig interface.
   - #7590 BLD: Fix configparser.InterpolationSyntaxError.
   - #7597 BUG: Make np.ma.take work on scalars.
   - #7608 BUG: linalg.norm(): Don't convert object arrays to float.
   - #7638 BLD: Correct C compiler customization in system_info.py.
   - #7654 BUG: ma.median of 1d array should return a scalar.
   - #7656 BLD: Remove hardcoded Intel compiler flag -xSSE4.2.
   - #7660 BUG: Temporary fix for str(mvoid) for object field types.
   - #7665 BUG: Fix incorrect printing of 1D masked arrays.
   - #7670 BUG: Correct initial index estimate in histogram.
   - #7671 BUG: Boolean assignment no GIL release when transfer needs API.
   - #7676 BUG: Fix handling of right edge of final histogram bin.
   - #7680 BUG: Fix np.clip bug NaN handling for Visual Studio 2015.


The following people have contributed to this release


   - Allan Haldane
   - Amit Aronovitch
   - Charles Harris
   - Eric Wieser
   - Evgeni Burovski
   - Lo?c Est?ve
   - Mathieu Lamarre
   - Matthew Brett
   - Matthias Geier
   - Nathaniel J. Smith
   - Nikola Forr?
   - Ralf Gommers
   - Robert Kern
   - Sebastian Berg
   - Simon Conseil
   - Simon Gibbons
   - Sorin Sbarnea
   - chiffa

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160526/9a57ee01/attachment.html>

From ralf.gommers at gmail.com  Fri May 27 03:48:51 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 27 May 2016 09:48:51 +0200
Subject: [Numpy-discussion] [SciPy-Dev] Numpy 1.11.1rc1 release
In-Reply-To: <CAB6mnxLjtKjP4suuSJfZ8=0eXvbvOTFDh8XjbBRqw6wmO1XOAg@mail.gmail.com>
References: <CAB6mnxLjtKjP4suuSJfZ8=0eXvbvOTFDh8XjbBRqw6wmO1XOAg@mail.gmail.com>
Message-ID: <CABL7CQgQ9vZpEzA2D8HobzDF7h5eogDuOB5+QLa7m0iHyU3i9g@mail.gmail.com>

On Fri, May 27, 2016 at 4:42 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> Hi All,
>
> I am pleased to announce the release of Numpy 1.11.1rc1.
>

Thanks Chuck!


> The sources may be found on sourceforge
> <https://sourceforge.net/projects/numpy/files/NumPy/1.11.1rc1/> and
> wheels for OS X, Windows, and Linux will be available on pypi sometime in
> the next few days. The pypi release is delayed due to the decision that the
> wheels should go up before the sources in order that people not get a
> source install when what they want are wheels.
>

For pre-releases that delay is maybe not necessary, because the average
user won't see those. Only if you add --pre to your pip install command
will you get them.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160527/13b8910c/attachment.html>

From Permafacture at gmail.com  Fri May 27 15:51:22 2016
From: Permafacture at gmail.com (Elliot Hallmark)
Date: Fri, 27 May 2016 14:51:22 -0500
Subject: [Numpy-discussion] Cross-correlation PR stuck in limbo
In-Reply-To: <5729E60F.1040102@crans.org>
References: <8D8A0029-A28D-4E4E-A0D5-437FDC85AA73@brandeis.edu>
 <5729E60F.1040102@crans.org>
Message-ID: <CAHedU1S=JA5C+X3w=PbBqH1b25ZELj2BhdwpLEJWpf0FBTkdzw@mail.gmail.com>

+1

This would really help with large data sets in certain situations.

Is there still disagreement about whether this should be included? Or are
there some minor details still? Or just lost in the shuffle?

Hopefully,
  Elliot

On Wed, May 4, 2016 at 7:07 AM, Pierre Haessig <pierre.haessig at crans.org>
wrote:

> Hi,
>
> I don't know how to push the PR forward, but all I can say is that this
> maxlag feature would be a major improvement for using Numpy in time
> series analysis! Immediate benefits downstream for Matplotlib and
> statsmodel.
>
> Thanks Honi for having taken the time to implement this!
>
> best,
> Pierre
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160527/4d51d681/attachment.html>

From lion.krischer at gmail.com  Fri May 27 16:51:25 2016
From: lion.krischer at gmail.com (Lion Krischer)
Date: Fri, 27 May 2016 22:51:25 +0200
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
Message-ID: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>

Hi all,

I was told to take this to the mailing list. Relevant pull request:
https://github.com/numpy/numpy/pull/7686


NumPy's FFT implementation caches some form of execution plan for each
encountered input data length. This is currently implemented as a simple
dictionary which can grow without bounds. Calculating lots of different
FFTs thus cause a memory leak from the users' perspective. We
encountered a real world situation where this is an issue.

The PR linked above proposes to replace the simple dictionary with an
LRU (least recently used) cache. It will remove the least recently used
pieces of data if it grows beyond a specified size (currently an
arbitrary limit of 100 MB per cache). Thus almost all users will still
benefit from the caches but their total memory size is now limited.


Things to consider:

* This adds quite some additional complexity but I could not find a
simple way to achieve the same result.
* What is a good limit on cache size? I used 100 MB because it works for
my uses cases.


Cheers!

Lion


From jaime.frio at gmail.com  Fri May 27 19:33:28 2016
From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=)
Date: Fri, 27 May 2016 16:33:28 -0700
Subject: [Numpy-discussion] Cross-correlation PR stuck in limbo
In-Reply-To: <CAHedU1S=JA5C+X3w=PbBqH1b25ZELj2BhdwpLEJWpf0FBTkdzw@mail.gmail.com>
References: <8D8A0029-A28D-4E4E-A0D5-437FDC85AA73@brandeis.edu>
 <5729E60F.1040102@crans.org>
 <CAHedU1S=JA5C+X3w=PbBqH1b25ZELj2BhdwpLEJWpf0FBTkdzw@mail.gmail.com>
Message-ID: <CAPOWHWnxhPhGYbLCRHfN3mq2ZAuQJwZ9Lkwd4SuG39nNUeEk4w@mail.gmail.com>

I did an overall review of the code a couple of weeks ago (see the PR for
details), and there is quite some work to be done before we can merge
Honi's code. But if he can find the time to work on the coding, I'll try to
be more diligent about the reviewing.

Jaime

On Fri, May 27, 2016 at 12:51 PM, Elliot Hallmark <Permafacture at gmail.com>
wrote:

> +1
>
> This would really help with large data sets in certain situations.
>
> Is there still disagreement about whether this should be included? Or are
> there some minor details still? Or just lost in the shuffle?
>
> Hopefully,
>   Elliot
>
> On Wed, May 4, 2016 at 7:07 AM, Pierre Haessig <pierre.haessig at crans.org>
> wrote:
>
>> Hi,
>>
>> I don't know how to push the PR forward, but all I can say is that this
>> maxlag feature would be a major improvement for using Numpy in time
>> series analysis! Immediate benefits downstream for Matplotlib and
>> statsmodel.
>>
>> Thanks Honi for having taken the time to implement this!
>>
>> best,
>> Pierre
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160527/23bd89e8/attachment.html>

From sebastian at sipsolutions.net  Sat May 28 14:19:27 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 28 May 2016 20:19:27 +0200
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
In-Reply-To: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
Message-ID: <1464459567.2690.12.camel@sipsolutions.net>

On Fr, 2016-05-27 at 22:51 +0200, Lion Krischer wrote:
> Hi all,
> 
> I was told to take this to the mailing list. Relevant pull request:
> https://github.com/numpy/numpy/pull/7686
> 
> 
> NumPy's FFT implementation caches some form of execution plan for
> each
> encountered input data length. This is currently implemented as a
> simple
> dictionary which can grow without bounds. Calculating lots of
> different
> FFTs thus cause a memory leak from the users' perspective. We
> encountered a real world situation where this is an issue.
> 
> The PR linked above proposes to replace the simple dictionary with an
> LRU (least recently used) cache. It will remove the least recently
> used
> pieces of data if it grows beyond a specified size (currently an
> arbitrary limit of 100 MB per cache). Thus almost all users will
> still
> benefit from the caches but their total memory size is now limited.
> 
> 
> Things to consider:
> 
> * This adds quite some additional complexity but I could not find a
> simple way to achieve the same result.
> * What is a good limit on cache size? I used 100 MB because it works
> for
> my uses cases.
> 

I am +1 in principle, since I don't like that the cache might just grow
forever and I see that as a bug right now personally. One that rarely
hits maybe, but a bug.

I guess if you have the time, the perfect thing would be if you could
time how big the cache difference is anyway, etc.?
The cache mostly gets a working array I think, so does it even help for
large arrays or is the time spend for the allocation negligible anway
then?
We also have a small array cache in numpy anyway nowadays (not sure how
small small is here). Maybe this already achieves everything that the
fftcache was designed for and we could even just disable it as default?

The complexity addition is a bit annoying I must admit, on python 3
functools.lru_cache could be another option, but only there.

- Sebastian


> 
> Cheers!
> 
> Lion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160528/cceea7d9/attachment.sig>

From shoyer at gmail.com  Sat May 28 22:35:22 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Sat, 28 May 2016 19:35:22 -0700
Subject: [Numpy-discussion] NumPy 1.11 docs
Message-ID: <CAEQ_TveJ3dnNoBbsdbhGGGUzKL4oEraNEmzAW3HD3az9g-LU7A@mail.gmail.com>

These still are missing from the SciPy.org page, several months after the
release.

What do we need to do to keep these updated? Is there someone at Enthought
we should ping? Or do we really just need to transition to different
infrastructure?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160528/8b23929d/attachment.html>

From sievert.scott at gmail.com  Sat May 28 23:53:33 2016
From: sievert.scott at gmail.com (Scott Sievert)
Date: Sat, 28 May 2016 22:53:33 -0500
Subject: [Numpy-discussion] ENH: compute many inner products quickly
Message-ID: <etPan.574a67bd.694f9671.109@Optimization-MacBook.local>

I recently ran into an application where I had to compute many inner products quickly (roughy 50k inner products in less than a second). I wanted a vector of inner products over the 50k vectors, or `[x1.T @ A @ x1, ?, xn.T @ A @ xn]` with A.shape = (1k, 1k).

My first instinct was to look for a NumPy function to quickly compute this, such as np.inner. However, it looks like np.inner has some other behavior and I couldn?t get tensordot/einsum to work for me.

Then a labmate pointed out that I can just do some slick matrix multiplication to compute the same quantity, `(X.T * A @ X.T).sum(axis=0)`. I opened [a PR] with this, and proposed that we define a new function called `inner_prods` for this.

However, in the PR, @shoyer pointed out?

>?The main challenge is to figure out how to transition the behavior of all these operations, while preserving backwards compatibility. Quite likely, we need to pick new names for these functions, though we should try to pick something that doesn't suggest that they are second class alternatives.

Do we choose new function names? Do we add a keyword arg that changes what np.inner returns?

[a PR]:https://github.com/numpy/numpy/pull/7690



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160528/781898f6/attachment.html>

From ralf.gommers at gmail.com  Sun May 29 04:13:06 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 29 May 2016 10:13:06 +0200
Subject: [Numpy-discussion] NumPy 1.11 docs
In-Reply-To: <CAEQ_TveJ3dnNoBbsdbhGGGUzKL4oEraNEmzAW3HD3az9g-LU7A@mail.gmail.com>
References: <CAEQ_TveJ3dnNoBbsdbhGGGUzKL4oEraNEmzAW3HD3az9g-LU7A@mail.gmail.com>
Message-ID: <CABL7CQjP=RvvNDiOaR=hboMubKPYNXXg4SGOHDtDom5kWT+Ctg@mail.gmail.com>

On Sun, May 29, 2016 at 4:35 AM, Stephan Hoyer <shoyer at gmail.com> wrote:

> These still are missing from the SciPy.org page, several months after the
> release.
>

Thanks Stephan, that needs fixing.


>
> What do we need to do to keep these updated?
>

https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt#update-docsscipyorg


> Is there someone at Enthought we should ping? Or do we really just need to
> transition to different infrastructure?
>

No, we just need to not forget:) The release manager normally does this, or
he pings someone else to do it. At the moment Pauli, Julian, Evgeni and me
have access to the server. I'll fix it up today.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160529/baa75a9e/attachment.html>

From m.h.vankerkwijk at gmail.com  Sun May 29 13:12:55 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Sun, 29 May 2016 13:12:55 -0400
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
In-Reply-To: <1464459567.2690.12.camel@sipsolutions.net>
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
Message-ID: <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>

Hi,

I did a few simple timing tests (see comment in PR), which suggests it is
hardly worth having the cache. Indeed, if one really worries about speed,
one should probably use pyFFTW (scipy.fft is a bit faster too, but at least
for me the way real FFT values are stored is just too inconvenient). So, my
suggestion would be to do away with the cache altogether.

If we do keep it, I think the approach in the PR is nice, but I would
advocate setting both a size and number limit (e.g., by default no more
than 8 entries or so, which should cover most repetitive use cases).

All the best,

Marten

p.s. I do like having a quick fft routine in numpy. My main gripe is that
it always casts to float64/complex128 rather than sticking with the input
dtype. Hope to get around to making  a PR for that...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160529/8c4ce97f/attachment.html>

From solipsis at pitrou.net  Mon May 30 04:06:41 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 30 May 2016 10:06:41 +0200
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
Message-ID: <20160530100641.1ff8b898@fsol>

On Sat, 28 May 2016 20:19:27 +0200
Sebastian Berg <sebastian at sipsolutions.net> wrote:
> 
> The complexity addition is a bit annoying I must admit, on python 3
> functools.lru_cache could be another option, but only there.

You can backport the pure Python version of lru_cache for Python 2 (or
vendor the backport done here:
https://pypi.python.org/pypi/backports.functools_lru_cache/).
The advantage is that lru_cache is C-accelerated in Python 3.5 and
upwards...

Regards

Antoine.




From contrebasse at gmail.com  Mon May 30 04:07:41 2016
From: contrebasse at gmail.com (Joseph Martinot-Lagarde)
Date: Mon, 30 May 2016 08:07:41 +0000 (UTC)
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
 <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>
Message-ID: <loom.20160530T100533-785@post.gmane.org>

Marten van Kerkwijk <m.h.vankerkwijk <at> gmail.com> writes:

> I did a few simple timing tests (see comment in PR), which suggests it is
hardly worth having the cache. Indeed, if one really worries about speed,
one should probably use pyFFTW (scipy.fft is a bit faster too, but at least
for me the way real FFT values are stored is just too inconvenient). So, my
suggestion would be to do away with the cache altogether. 

The problem with FFTW is that its license is more restrictive (GPL), and
because of this may not be suitable everywhere numpy.fft is.



From lion.krischer at gmail.com  Mon May 30 05:23:56 2016
From: lion.krischer at gmail.com (Lion Krischer)
Date: Mon, 30 May 2016 11:23:56 +0200
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
In-Reply-To: <20160530100641.1ff8b898@fsol>
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net> <20160530100641.1ff8b898@fsol>
Message-ID: <74c91f5e-2323-7136-1773-750e01e1dc48@gmail.com>


> You can backport the pure Python version of lru_cache for Python 2 (or
> vendor the backport done here:
> https://pypi.python.org/pypi/backports.functools_lru_cache/).
> The advantage is that lru_cache is C-accelerated in Python 3.5 and
> upwards...

That's a pretty big back-port. The speed also does not matter for this
particular use-case: Time for the actual FFT will dominate by far. The
lru_cache decorator can furthermore only limit the cache size by item
count and not size in memory as the proposed solution does. I think the
downsides outweigh the advantages of being able to use functionality
from the stdlib.



From lion.krischer at gmail.com  Mon May 30 05:26:16 2016
From: lion.krischer at gmail.com (Lion Krischer)
Date: Mon, 30 May 2016 11:26:16 +0200
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
In-Reply-To: <loom.20160530T100533-785@post.gmane.org>
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
 <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>
 <loom.20160530T100533-785@post.gmane.org>
Message-ID: <70cb39b4-7bef-3d2f-1388-c1f1f6967f10@gmail.com>



On 30/05/16 10:07, Joseph Martinot-Lagarde wrote:
> Marten van Kerkwijk <m.h.vankerkwijk <at> gmail.com> writes:
> 
>> I did a few simple timing tests (see comment in PR), which suggests it is
> hardly worth having the cache. Indeed, if one really worries about speed,
> one should probably use pyFFTW (scipy.fft is a bit faster too, but at least
> for me the way real FFT values are stored is just too inconvenient). So, my
> suggestion would be to do away with the cache altogether. 


I added a slightly more comprehensive benchmark to the PR. Please have a
look. It tests the total time for 100 FFTs with and without cache. It is
over 30 percent faster with cache which it totally worth it in my
opinion as repeated FFTs of the same size are a very common use case.

Also many people will not have enough knowledge to use FFTW or some
other FFT implementation.


From shoyer at gmail.com  Tue May 31 01:36:28 2016
From: shoyer at gmail.com (Stephan Hoyer)
Date: Tue, 31 May 2016 05:36:28 +0000
Subject: [Numpy-discussion] NumPy 1.11 docs
In-Reply-To: <CABL7CQjP=RvvNDiOaR=hboMubKPYNXXg4SGOHDtDom5kWT+Ctg@mail.gmail.com>
References: <CAEQ_TveJ3dnNoBbsdbhGGGUzKL4oEraNEmzAW3HD3az9g-LU7A@mail.gmail.com>
 <CABL7CQjP=RvvNDiOaR=hboMubKPYNXXg4SGOHDtDom5kWT+Ctg@mail.gmail.com>
Message-ID: <CAEQ_TvfeXavZwuAKCK8Oj7g-qQeQyomvZAGfWeptuNAG6zOW2A@mail.gmail.com>

Awesome, thanks Ralf!
On Sun, May 29, 2016 at 1:13 AM Ralf Gommers <ralf.gommers at gmail.com> wrote:

> On Sun, May 29, 2016 at 4:35 AM, Stephan Hoyer <shoyer at gmail.com> wrote:
>
>> These still are missing from the SciPy.org page, several months after the
>> release.
>>
>
> Thanks Stephan, that needs fixing.
>
>
>>
>> What do we need to do to keep these updated?
>>
>
>
> https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt#update-docsscipyorg
>
>
>> Is there someone at Enthought we should ping? Or do we really just need
>> to transition to different infrastructure?
>>
>
> No, we just need to not forget:) The release manager normally does this,
> or he pings someone else to do it. At the moment Pauli, Julian, Evgeni and
> me have access to the server. I'll fix it up today.
>
> Ralf
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160531/a278e937/attachment.html>

From pierre.debuyl at chem.kuleuven.be  Tue May 31 09:05:23 2016
From: pierre.debuyl at chem.kuleuven.be (Pierre de Buyl)
Date: Tue, 31 May 2016 15:05:23 +0200
Subject: [Numpy-discussion] EuroSciPy 2016
Message-ID: <20160531130523.GN12938@pi-x230>

Dear NumPy and SciPy communities,

No annoucement was made here for EuroSciPy 2016 I believe. The call for
contributions (talks, posters, sprints) is still open for a few days.

EuroSciPy 2016 takes place in Erlangen, Germany, from the 23 to the 27 of August
and consists of two days of tutorials (beginner and advanced tracks) and two
days of conference representing many fields of science, with a focus on Python
tools for science. A day of sprints follows (sprints TBA).

The keynote speakers are Ga?l Varoquaux and Abby Cabunoc Mayes and we can expect
a rich tutorial and scientific program! Videos from previous years are available
at https://www.youtube.com/playlist?list=PLYx7XA2nY5GeQCCugyvtnHMVLdhYlrRxH and
https://www.youtube.com/playlist?list=PLYx7XA2nY5Gcpabmu61kKcToLz0FapmHu

Visit us, register and submit an abstract on our website!
https://www.euroscipy.org/2016/

SciPythonic regards,

The EuroSciPy 2016 team



From sebastian at sipsolutions.net  Tue May 31 11:37:12 2016
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 31 May 2016 17:37:12 +0200
Subject: [Numpy-discussion] Developer Meeting at EuroScipy?
Message-ID: <1464709032.8693.25.camel@sipsolutions.net>

Hi all,

since we had decided to do a regular developers meeting last year, how
would EuroScipy (Aug. 23.-27., Erlangen, Germany) look as a possible
place and time to have one?
I believe EuroScipy would include a few people who were not able to
come to SciPy last year, and it seems SciPy itself already sees some of
the devs quite busy anyway.

Not having checked back for room availability at the conference or
anything, one option may be:

August 24. (parallel to the second/last day of Tutorials)

Does this seem like a good plan or do you have alternative suggestions
or scheduling difficulties with this?

Regards,

Sebastian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160531/2f41afb4/attachment.sig>

From sturla.molden at gmail.com  Tue May 31 17:36:44 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 31 May 2016 21:36:44 +0000 (UTC)
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
 <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>
 <loom.20160530T100533-785@post.gmane.org>
 <70cb39b4-7bef-3d2f-1388-c1f1f6967f10@gmail.com>
Message-ID: <1670833866486423240.853891sturla.molden-gmail.com@news.gmane.org>

Lion Krischer <lion.krischer at gmail.com> wrote:

> I added a slightly more comprehensive benchmark to the PR. Please have a
> look. It tests the total time for 100 FFTs with and without cache. It is
> over 30 percent faster with cache which it totally worth it in my
> opinion as repeated FFTs of the same size are a very common use case.

All the calls to trancendental functions are stored in the cache. Without a
cache, we get excessive calls to sin(x) and cos(x) whenever FFTs of the
same size are repeated. This can indeed matter at lot.

Sturla



From sturla.molden at gmail.com  Tue May 31 17:36:47 2016
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 31 May 2016 21:36:47 +0000 (UTC)
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
 <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>
 <loom.20160530T100533-785@post.gmane.org>
Message-ID: <1393163039486422808.824659sturla.molden-gmail.com@news.gmane.org>

Joseph Martinot-Lagarde <contrebasse at gmail.com> wrote:
 
> The problem with FFTW is that its license is more restrictive (GPL), and
> because of this may not be suitable everywhere numpy.fft is.

A lot of us use NumPy linked with MKL or Accelerate, both of which have
some really nifty FFTs. And the license issue is hardly any worse than
linking with them for BLAS and LAPACK, which we do anyway. We could extend
numpy.fft to use MKL or Accelerate when they are available.

Sturla



From jaime.frio at gmail.com  Tue May 31 21:34:06 2016
From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=)
Date: Tue, 31 May 2016 18:34:06 -0700
Subject: [Numpy-discussion] Developer Meeting at EuroScipy?
In-Reply-To: <1464709032.8693.25.camel@sipsolutions.net>
References: <1464709032.8693.25.camel@sipsolutions.net>
Message-ID: <CAPOWHWk-d0gR=r5GVv0gDc93WC8GDEV2HYVKjx8cy96XCymBhQ@mail.gmail.com>

On Tue, May 31, 2016 at 8:37 AM, Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> Hi all,
>
> since we had decided to do a regular developers meeting last year, how
> would EuroScipy (Aug. 23.-27., Erlangen, Germany) look as a possible
> place and time to have one?
> I believe EuroScipy would include a few people who were not able to
> come to SciPy last year, and it seems SciPy itself already sees some of
> the devs quite busy anyway.
>

I will not be able to attend the full conference, but it's only a 10 hour
train ride from what I now call home, so I am positive I can be there for a
full day meeting, no matter what day of that week it is.

Jaime


>
> Not having checked back for room availability at the conference or
> anything, one option may be:
>
> August 24. (parallel to the second/last day of Tutorials)
>
> Does this seem like a good plan or do you have alternative suggestions
> or scheduling difficulties with this?
>
> Regards,
>
> Sebastian
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160531/67801fc8/attachment.html>

From m.h.vankerkwijk at gmail.com  Tue May 31 22:02:56 2016
From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk)
Date: Tue, 31 May 2016 22:02:56 -0400
Subject: [Numpy-discussion] Changing FFT cache to a bounded LRU cache
In-Reply-To: <1393163039486422808.824659sturla.molden-gmail.com@news.gmane.org>
References: <4f5279db-43d3-4ae2-7a8b-67d1ec7bd802@gmail.com>
 <1464459567.2690.12.camel@sipsolutions.net>
 <CAJNV+9sQqJ3LzDnzdfxvwBHj9TY_ArgxgWVKBYBZD7GkejNnwg@mail.gmail.com>
 <loom.20160530T100533-785@post.gmane.org>
 <1393163039486422808.824659sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAJNV+9sVZELn2m=oqXJN_x7X_j5WSWTYZBC7G7X-Jaen6MnwkA@mail.gmail.com>

>
> A lot of us use NumPy linked with MKL or Accelerate, both of which have
> some really nifty FFTs. And the license issue is hardly any worse than
> linking with them for BLAS and LAPACK, which we do anyway. We could extend
> numpy.fft to use MKL or Accelerate when they are available.
>

That would be wonderful! Especially if one can also remove the automatic
cast to double (as I'm analysing 2-bit VLBI data, getting to 64 bit is
overkill...).

-- Marten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160531/c63634a0/attachment.html>