On Tue, Oct 25, 2016 at 10:22 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:

On Tue, Oct 25, 2016 at 10:41 PM, Robert Kern robert.kern@gmail.com

wrote:

...

On Tue, Oct 25, 2016 at 9:34 PM, Charles R Harris <

charlesr.harris@gmail.com> wrote:

...

...

Hi All,

There is a proposed random number package PR now up on github:

https://github.com/numpy/numpy/pull/8209. It is from

...

...

oleksandr-pavlyk and implements the number random number package using

MKL for increased speed. I think we are definitely interested in the improved speed, but I'm not sure numpy is the best place to put the package. I'd welcome any comments on the PR itself, as well as any thoughts on the best way organize or use of this work. Maybe scikit-random

...

This is what ng-numpy-randomstate is for.

https://github.com/bashtage/ng-numpy-randomstate

Interesting, despite old fashioned original ziggurat implementation of

the normal and gnu c style... Does that project seek to preserve all the bytestreams or is it still in flux?

I would assume some flux for now, but you can ask the author by submitting a corrected ziggurat PR as a trial balloon. ;-)

-- Robert Kern

On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:

On 26.10.2016 06:34, Charles R Harris wrote:

...

Hi All,

There is a proposed random number package PR now up on github: https://github.com/numpy/numpy/pull/8209. It is from oleksandr-pavlyk https://github.com/oleksandr-pavlyk and implements the number random number package using MKL for increased speed. I think we are definitely interested in the improved speed, but I'm not sure numpy is the best place to put the package. I'd welcome any comments on the PR itself, as well as any thoughts on the best way organize or use of this work. Maybe scikit-random

Note that this thread is a continuation of https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html

I'm not a fan of putting code depending on a proprietary library into numpy. This should be a standalone package which may provide the same interface as numpy.

I don't really see a problem with that in principle. Numpy can use Intel MKL (and Accelerate) as well if it's available. It needs some thought put into the API though - a ``numpy.random_intel`` module is certainly not what we want.

Ralf

On 10/26/2016 06:00 PM, Julian Taylor wrote:

On 10/26/2016 10:59 AM, Ralf Gommers wrote:

...

On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor <jtaylor.debian@googlemail.com mailto:jtaylor.debian@googlemail.com> wrote:

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
>
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209
<https://github.com/numpy/numpy/pull/8209>. It is from
> oleksandr-pavlyk <https://github.com/oleksandr-pavlyk
<https://github.com/oleksandr-pavlyk>> and implements
> the number random number package using MKL for increased speed.

I think > we are definitely interested in the improved speed, but I'm not sure > numpy is the best place to put the package. I'd welcome any comments on > the PR itself, as well as any thoughts on the best way organize or use > of this work. Maybe scikit-random

Note that this thread is a continuation of https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html

I'm not a fan of putting code depending on a proprietary library
into numpy.
This should be a standalone package which may provide the same

interface as numpy.

I don't really see a problem with that in principle. Numpy can use Intel MKL (and Accelerate) as well if it's available. It needs some thought put into the API though - a ``numpy.random_intel`` module is certainly not what we want.

For me there is a difference between being able to optionally use a proprietary library as an alternative to free software libraries if the user wishes to do so and offering functionality that only works with non-free software. We are providing a form of advertisement for them by allowing it (hey if you buy this black box that you cannot modify or use freely you get this neat numpy feature!).

I prefer for the full functionality of numpy to stay available with a stack of community owned software, even if it may be less powerful that way.

But then if this is really just the same random numbers numpy already provides just faster, it is probably acceptable in principle. I haven't actually looked at the PR yet.

On Mi, 2016-10-26 at 09:29 -0700, Robert Kern wrote:

On Wed, Oct 26, 2016 at 9:10 AM, Julian Taylor <jtaylor.debian@google mail.com> wrote:

...

On 10/26/2016 06:00 PM, Julian Taylor wrote:

...

...

I prefer for the full functionality of numpy to stay available

with a

...

...

stack of community owned software, even if it may be less powerful

that

...

...

way.

But then if this is really just the same random numbers numpy

already provides just faster, it is probably acceptable in principle. I haven't actually looked at the PR yet.

I think the stream is different in some places, at least. And it's not a silent backend drop-in like np.linalg being built against an optimized BLAS, just a separate module that is inoperative without MKL.

I might be swayed, but my gut feeling would be that a backend change (if the default stream changes, an explicit one, though maybe one could make a "fastest") would be the only reasonable way to provide such a thing in numpy itself.

- Sebastian

-- Robert Kern _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Oct 26, 2016 at 9:10 AM, Julian Taylor jtaylor.debian@googlemail.com wrote:

On 10/26/2016 06:00 PM, Julian Taylor wrote:

...

On 10/26/2016 10:59 AM, Ralf Gommers wrote:

...

On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor <jtaylor.debian@googlemail.com mailto:jtaylor.debian@googlemail.com> wrote:

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
>
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209
<https://github.com/numpy/numpy/pull/8209>. It is from
> oleksandr-pavlyk <https://github.com/oleksandr-pavlyk
<https://github.com/oleksandr-pavlyk>> and implements
> the number random number package using MKL for increased speed.

I think > we are definitely interested in the improved speed, but I'm not sure > numpy is the best place to put the package. I'd welcome any comments on > the PR itself, as well as any thoughts on the best way organize or use > of this work. Maybe scikit-random

Note that this thread is a continuation of https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html

I'm not a fan of putting code depending on a proprietary library
into numpy.
This should be a standalone package which may provide the same

interface as numpy.

I don't really see a problem with that in principle. Numpy can use Intel MKL (and Accelerate) as well if it's available. It needs some thought put into the API though - a ``numpy.random_intel`` module is certainly not what we want.

For me there is a difference between being able to optionally use a proprietary library as an alternative to free software libraries if the user wishes to do so and offering functionality that only works with non-free software. We are providing a form of advertisement for them by allowing it (hey if you buy this black box that you cannot modify or use freely you get this neat numpy feature!).

I prefer for the full functionality of numpy to stay available with a stack of community owned software, even if it may be less powerful that way.

But then if this is really just the same random numbers numpy already provides just faster, it is probably acceptable in principle. I haven't actually looked at the PR yet.

The RNG stream is totally different, so yeah, it can't just be a silent drop-in replacement like BLAS/LAPACK.

The patch also adds ~10,000 lines of code; here's an example of what some of it looks like:

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

I don't see how we can realistically commit to maintaining this.

I'm also not really seeing how shipping it as part of numpy provides extra benefits to maintainers or users? AFAICT right now it's basically structured as a standalone library that's been dropped into the numpy source tree, and it would be just as easy to ship separately (or am I wrong?). And since the public API is that all the functionality comes from importing this specific new module ('numpy.random_intel'), it'd be a one-line change for users to import from a non-numpy namespace, like 'mkl.random' or whatever. If it were more integrated with the rest of numpy then the trade-offs would be more complicated, but in its present form this seems like an easy call.

The other question is whether it could/should change to *become* more integrated... that's more tricky. There's been some work towards supporting swappable backends inside np.random; but the focus has mostly been on allowing new core generators, though, and this code seems to want to take over the whole thing (core generator + distributions), so even once the swappable backends stuff is working I'm not sure it would be relevant here. The one case I can think of that does seem promising is that if we get an API for users to say "I don't care about stream compatibility, just give me un-reproducible variates as fast as you can", then it might make sense for that to silently use MKL if available -- this would be pretty analogous to the use of MKL in np.linalg. But we don't have that API yet, I'm not sure how the MKL fallback could be maintainably implemented given that it would require somehow swapping the entire RandomState implementation, and it's entirely possible that once we figure out solutions to those then it'd still make sense for the actual MKL wrappers to live in a third-party library that numpy imports.

-n

On Wed, Oct 26, 2016 at 12:41 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:

On Wed, Oct 26, 2016 at 3:24 PM, Nathaniel Smith njs@pobox.com wrote:

...

The patch also adds ~10,000 lines of code; here's an example of what some of it looks like:

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

...

I don't see how we can realistically commit to maintaining this.

FYI: numpy already maintains code exactly like that:

https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/distributions...

Perhaps the point should be that the numpy devs won't want to maintain

two nearly identical versions of that code.

Indeed. That's how the algorithm was published. The /* sigh ... */ is my own. ;-)

-- Robert Kern

Hi,

Thanks a lot everybody for the feedback.

The package can certainly be made a stand-alone drop-in replacement for np.random. There are many points raised and unraised in favor of this, and it is easy to accomplish. I will create a stand-alone package on github, but would still appreciate some help in reviewing it and making it available at PyPI.

Interestingly, Nathaniel's link to a representative changes, specifically

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

point at an unused code borrowed directly from mtrand/distributions.c:

https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/distributions...

More representative change would be the implementation of Student's T-distribution:

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

The module under review, similarly to randomstate package, provides alternative basic pseudo-random number generators (BRNGs), like MT2203, MCG31, MRG32K3A, Wichmann-Hill. The scope of support differ, with randomstate implementing some generators absent in MKL and vice-versa.

Thinking about the possibility of providing the functionality of this module within the framework of randomstate, I find that randomstate implements samplers from statistical distributions as functions that take the state of the underlying BRNG, and produce a single variate, e.g.:

https://github.com/bashtage/ng-numpy-randomstate/blob/master/randomstate/dis...

This design stands in a way of efficient use of MKL, which generates a whole vector of variates at a time. This can be done faster than sampling a variate at a time by using vectorized instructions. So I wrote mkl_distributions.cpp to provide functions that return a given size vector of sampled variates from each supported distribution.

mklrand.pyx was then written by modifying mtrand.pyx to work with such vector generators. In particular, this allowed for efficient sampling from product distributions of Poisson distributions with different rate parameters, which is implemented in MKL:

https://software.intel.com/en-us/node/521894

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

Another point already raised by Nathaniel is that for numpy's randomness ideally should provide a way to override default algorithm for sampling from a particular distribution. For example RandomState object that implements PCG may rely on default acceptance-rejection algorithm for sampling from Gamma, while the RandomState object that provides interface to MKL might want to call into MKL directly.

While at this topic, I also would like to point out the need for C-API interface to randomness, particularly felt writing parallel algorithms, where Python's GIL and use of Lock() in RandomState hurt scalability.

Oleksandr

-----Original Message----- From: NumPy-Discussion [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Nathaniel Smith Sent: Wednesday, October 26, 2016 2:25 PM To: Discussion of Numerical Python numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] Intel random number package

On Wed, Oct 26, 2016 at 9:10 AM, Julian Taylor jtaylor.debian@googlemail.com wrote:

On 10/26/2016 06:00 PM, Julian Taylor wrote:

...

On 10/26/2016 10:59 AM, Ralf Gommers wrote:

...

On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor <jtaylor.debian@googlemail.com mailto:jtaylor.debian@googlemail.com> wrote:

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
>
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209
<https://github.com/numpy/numpy/pull/8209>. It is from
> oleksandr-pavlyk <https://github.com/oleksandr-pavlyk
<https://github.com/oleksandr-pavlyk>> and implements
> the number random number package using MKL for increased speed.

I think > we are definitely interested in the improved speed, but I'm not sure > numpy is the best place to put the package. I'd welcome any comments on > the PR itself, as well as any thoughts on the best way organize or use > of this work. Maybe scikit-random

Note that this thread is a continuation of https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.h tml

I'm not a fan of putting code depending on a proprietary library
into numpy.
This should be a standalone package which may provide the same 

interface as numpy.

I don't really see a problem with that in principle. Numpy can use Intel MKL (and Accelerate) as well if it's available. It needs some thought put into the API though - a ``numpy.random_intel`` module is certainly not what we want.

For me there is a difference between being able to optionally use a proprietary library as an alternative to free software libraries if the user wishes to do so and offering functionality that only works with non-free software. We are providing a form of advertisement for them by allowing it (hey if you buy this black box that you cannot modify or use freely you get this neat numpy feature!).

I prefer for the full functionality of numpy to stay available with a stack of community owned software, even if it may be less powerful that way.

But then if this is really just the same random numbers numpy already provides just faster, it is probably acceptable in principle. I haven't actually looked at the PR yet.

The RNG stream is totally different, so yeah, it can't just be a silent drop-in replacement like BLAS/LAPACK.

The patch also adds ~10,000 lines of code; here's an example of what some of it looks like:

https://github.com/oleksandr-pavlyk/numpy/blob/b53880432c19356f4e54b52095827...

I don't see how we can realistically commit to maintaining this.

I'm also not really seeing how shipping it as part of numpy provides extra benefits to maintainers or users? AFAICT right now it's basically structured as a standalone library that's been dropped into the numpy source tree, and it would be just as easy to ship separately (or am I wrong?). And since the public API is that all the functionality comes from importing this specific new module ('numpy.random_intel'), it'd be a one-line change for users to import from a non-numpy namespace, like 'mkl.random' or whatever. If it were more integrated with the rest of numpy then the trade-offs would be more complicated, but in its present form this seems like an easy call.

The other question is whether it could/should change to *become* more integrated... that's more tricky. There's been some work towards supporting swappable backends inside np.random; but the focus has mostly been on allowing new core generators, though, and this code seems to want to take over the whole thing (core generator + distributions), so even once the swappable backends stuff is working I'm not sure it would be relevant here. The one case I can think of that does seem promising is that if we get an API for users to say "I don't care about stream compatibility, just give me un-reproducible variates as fast as you can", then it might make sense for that to silently use MKL if available -- this would be pretty analogous to the use of MKL in np.linalg. But we don't have that API yet, I'm not sure how the MKL fallback could be maintainably implemented given that it would require somehow swapping the entire RandomState implementation, and it's entirely possible that once we figure out solutions to those then it'd still make sense for the actual MKL wrappers to live in a third-party library that numpy imports.

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

Please see responses inline.

From: NumPy-Discussion [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Todd Sent: Wednesday, October 26, 2016 4:04 PM To: Discussion of Numerical Python numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] Intel random number package

On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr <oleksandr.pavlyk@intel.commailto:oleksandr.pavlyk@intel.com> wrote:

The module under review, similarly to randomstate package, provides alternative basic pseudo-random number generators (BRNGs), like MT2203, MCG31, MRG32K3A, Wichmann-Hill. The scope of support differ, with randomstate implementing some generators absent in MKL and vice-versa.

Is there a reason that randomstate shouldn't implement those generators?

No, randomstate certainly can implement all the BRNGs implemented in MKL. It is at developer’s discretion.

Thinking about the possibility of providing the functionality of this module within the framework of randomstate, I find that randomstate implements samplers from statistical distributions as functions that take the state of the underlying BRNG, and produce a single variate, e.g.:

https://github.com/bashtage/ng-numpy-randomstate/blob/master/randomstate/dis...

This design stands in a way of efficient use of MKL, which generates a whole vector of variates at a time. This can be done faster than sampling a variate at a time by using vectorized instructions. So I wrote mkl_distributions.cpp to provide functions that return a given size vector of sampled variates from each supported distribution.

I don't know a huge amount about pseudo-random number generators, but this seems superficially to be something that would benefit random number generation as a whole independently of whether MKL is used. Might it be possible to modify the numpy implementation to support this sort of vectorized approach?

I also think that adopting vectorized mindset would benefit np.random. For example, Gaussians are currently generated using Box-Muller algorithm which produces two variate at a time, so one currently needs to be saved in the random state struct itself, along with an indicator that it should be used on the next iteration. With vectorized approach one could populate the vector two elements at a time with better memory locality, resulting in better performance.

Vectorized approach has merits with or without use of MKL.

Another point already raised by Nathaniel is that for numpy's randomness ideally should provide a way to override default algorithm for sampling from a particular distribution. For example RandomState object that implements PCG may rely on default acceptance-rejection algorithm for sampling from Gamma, while the RandomState object that provides interface to MKL might want to call into MKL directly.

The approach that pyfftw uses at least for scipy, which may also work here, is that you can monkey-patch the scipy.fftpack module at runtime, replacing it with pyfftw's drop-in replacement. scipy then proceeds to use pyfftw instead of its built-in fftpack implementation. Might such an approach work here? Users can either use this alternative randomstate replacement directly, or they can replace numpy's with it at runtime and numpy will then proceed to use the alternative.

I think the monkey-patching approach will work.

RandomState was written with a view to replace numpy.random at some point in the future. It is standalone at the moment, from what I understand, only because it is still being worked on and extended.

One particularly important development is the ability to sample continuous distributions in floats, or to populate a given preallocated buffer with random samples. These features are missing from numpy.random_intel and we thought it providing them.

As I have said earlier, another missing feature in the C-API for randomness in numpy.

Oleksandr

On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers ralf.gommers@gmail.com wrote:

On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr < oleksandr.pavlyk@intel.com> wrote:

...

Please see responses inline.

*From:* NumPy-Discussion [mailto:numpy-discussion-bounces@scipy.org] *On Behalf Of *Todd *Sent:* Wednesday, October 26, 2016 4:04 PM *To:* Discussion of Numerical Python numpy-discussion@scipy.org *Subject:* Re: [Numpy-discussion] Intel random number package

On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr < oleksandr.pavlyk@intel.com> wrote:

Another point already raised by Nathaniel is that for numpy's randomness ideally should provide a way to override default algorithm for sampling from a particular distribution. For example RandomState object that implements PCG may rely on default acceptance-rejection algorithm for sampling from Gamma, while the RandomState object that provides interface to MKL might want to call into MKL directly.

The approach that pyfftw uses at least for scipy, which may also work here, is that you can monkey-patch the scipy.fftpack module at runtime, replacing it with pyfftw's drop-in replacement. scipy then proceeds to use pyfftw instead of its built-in fftpack implementation. Might such an approach work here? Users can either use this alternative randomstate replacement directly, or they can replace numpy's with it at runtime and numpy will then proceed to use the alternative.

The only reason that pyfftw uses monkeypatching is that the better approach is not possible due to license constraints with FFTW (it's GPL).

Yes, that is exactly why I brought it up. Better approaches are also not possible with MKL due to license constraints. It is a very similar situation overall.

On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:

On 10/27/2016 04:30 PM, Todd wrote:

...

On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers <ralf.gommers@gmail.com mailto:ralf.gommers@gmail.com> wrote:

On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
<oleksandr.pavlyk@intel.com <mailto:oleksandr.pavlyk@intel.com>>

wrote:

    Please see responses inline.



    *From:*NumPy-Discussion
    [mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>] *On Behalf Of *Todd
    *Sent:* Wednesday, October 26, 2016 4:04 PM
    *To:* Discussion of Numerical Python <numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>>
    *Subject:* Re: [Numpy-discussion] Intel random number package




    On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
    <oleksandr.pavlyk@intel.com <mailto:oleksandr.pavlyk@intel.com>>
    wrote:

        Another point already raised by Nathaniel is that for
        numpy's randomness ideally should provide a way to override
        default algorithm for sampling from a particular
        distribution.  For example RandomState object that
        implements PCG may rely on default acceptance-rejection
        algorithm for sampling from Gamma, while the RandomState
        object that provides interface to MKL might want to call
        into MKL directly.



    The approach that pyfftw uses at least for scipy, which may also
    work here, is that you can monkey-patch the scipy.fftpack module
    at runtime, replacing it with pyfftw's drop-in replacement.
    scipy then proceeds to use pyfftw instead of its built-in
    fftpack implementation.  Might such an approach work here?
    Users can either use this alternative randomstate replacement
    directly, or they can replace numpy's with it at runtime and
    numpy will then proceed to use the alternative.


The only reason that pyfftw uses monkeypatching is that the better
approach is not possible due to license constraints with FFTW (it's
GPL).

Yes, that is exactly why I brought it up. Better approaches are also not possible with MKL due to license constraints. It is a very similar situation overall.

Its not that similar, the better approach is certainly possible with FFTW, the GPL is compatible with numpys license. It is only a concern users of binary distributions. Nobody provided the code to use fftw yet, but it would certainly be accepted.

Although it is technically compatible, it would make numpy effectively GPL. Suggestions for this have been explicitly rejected on these grounds [1]

[1] https://github.com/numpy/numpy/issues/3485

Releasing NumPy under GPL would make it incompatible with SciPy, which may be _slightly_ inconvenient to the scientific Python community:

https://scipy.github.io/old-wiki/pages/License_Compatibility.html

https://mail.scipy.org/pipermail/scipy-dev/2013-August/019149.html

Robert

On Thu, Oct 27, 2016 at 5:14 PM, Julian Taylor < jtaylor.debian@googlemail.com> wrote:

On 10/27/2016 04:52 PM, Todd wrote:

...

On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor <jtaylor.debian@googlemail.com mailto:jtaylor.debian@googlemail.com> wrote:

On 10/27/2016 04:30 PM, Todd wrote:

    On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers
    <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>
    <mailto:ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>>>
    wrote:


        On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
        <oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>
    <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>>> wrote:

            Please see responses inline.



            *From:*NumPy-Discussion
            [mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>
            <mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>>] *On Behalf Of *Todd
            *Sent:* Wednesday, October 26, 2016 4:04 PM
            *To:* Discussion of Numerical Python
    <numpy-discussion@scipy.org <mailto:numpy-discussion@scipy.org>
            <mailto:numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>>>
            *Subject:* Re: [Numpy-discussion] Intel random number
    package




            On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
            <oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>
    <mailto:oleksandr.pavlyk@intel.com

    <mailto:oleksandr.pavlyk@intel.com>>>
            wrote:

                Another point already raised by Nathaniel is that for
                numpy's randomness ideally should provide a way to
    override
                default algorithm for sampling from a particular
                distribution.  For example RandomState object that
                implements PCG may rely on default

acceptance-rejection algorithm for sampling from Gamma, while the RandomState object that provides interface to MKL might want to call into MKL directly.

            The approach that pyfftw uses at least for scipy, which
    may also
            work here, is that you can monkey-patch the
    scipy.fftpack module
            at runtime, replacing it with pyfftw's drop-in

replacement. scipy then proceeds to use pyfftw instead of its built-in fftpack implementation. Might such an approach work here? Users can either use this alternative randomstate replacement directly, or they can replace numpy's with it at runtime and numpy will then proceed to use the alternative.

        The only reason that pyfftw uses monkeypatching is that the
    better
        approach is not possible due to license constraints with
    FFTW (it's
        GPL).


    Yes, that is exactly why I brought it up.  Better approaches are
    also
    not possible with MKL due to license constraints.  It is a very
    similar
    situation overall.


Its not that similar, the better approach is certainly possible with
FFTW, the GPL is compatible with numpys license. It is only a
concern users of binary distributions. Nobody provided the code to
use fftw yet, but it would certainly be accepted.

Although it is technically compatible, it would make numpy effectively GPL. Suggestions for this have been explicitly rejected on these grounds [1]

[1] https://github.com/numpy/numpy/issues/3485

Yes it would make numpy GPL, but that is not a concern for a lot of users. Users for who it is a problem can still use the non-GPL version. A more interesting debate is whether our binary wheels should then be GPL wheels by default or not. Probably not, but that is something that should be discussed when its an actual issue.

But to clarify what I said, it would be accepted if the value it provides is sufficient compared to the code maintenance it adds. Given that pyfftw already exists the value is probably relatively small, but personally I'd still be interested in code that allows switching the fft backend as that could also allow plugging e.g. gpu based implementations (though again this is already covered by other third party modules).

---

NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

As I understand it the wiki is talking about including code in numpy/scipy itself, all code in numpy and scipy must be permissively licensed so it is easy to reason about when building your binaries.

The license of the binaries produced from the code is a different matter, which at that time didn't really exist as we didn't distribute binaries at all (except for windows).

A GPL licensed binary containing numpy is perfectly compatible with SciPy. It may not be compatible with some other component which has an actually incompatible license (e.g. anything you cannot distribute the source of as required by the GPL). I it is not numpy that is GPL licensed it is the restriction of another component in the binary distribution that makes the full product adhere to the most restrictive license But numpy itself is always permissive, the distributor can always build a permissive numpy binary without the viral component in it.

On 10/27/2016 05:42 PM, Robert McLeod wrote:

Releasing NumPy under GPL would make it incompatible with SciPy, which may be _slightly_ inconvenient to the scientific Python community:

https://scipy.github.io/old-wiki/pages/License_Compatibility.html

https://mail.scipy.org/pipermail/scipy-dev/2013-August/019149.html

Robert

On Thu, Oct 27, 2016 at 5:14 PM, Julian Taylor <jtaylor.debian@googlemail.com mailto:jtaylor.debian@googlemail.com> wrote:

On 10/27/2016 04:52 PM, Todd wrote:

    On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor
    <jtaylor.debian@googlemail.com
    <mailto:jtaylor.debian@googlemail.com>
    <mailto:jtaylor.debian@googlemail.com
    <mailto:jtaylor.debian@googlemail.com>>>
    wrote:

        On 10/27/2016 04:30 PM, Todd wrote:

            On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers
            <ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>
    <mailto:ralf.gommers@gmail.com <mailto:ralf.gommers@gmail.com>>
            <mailto:ralf.gommers@gmail.com
    <mailto:ralf.gommers@gmail.com> <mailto:ralf.gommers@gmail.com
    <mailto:ralf.gommers@gmail.com>>>>
            wrote:


                On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
                <oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>
            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>>
            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>
            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>>>> wrote:

                    Please see responses inline.



                    *From:*NumPy-Discussion
                    [mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>
            <mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>>
                    <mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>
            <mailto:numpy-discussion-bounces@scipy.org
    <mailto:numpy-discussion-bounces@scipy.org>>>] *On Behalf Of *Todd
                    *Sent:* Wednesday, October 26, 2016 4:04 PM
                    *To:* Discussion of Numerical Python
            <numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>
    <mailto:numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>>
                    <mailto:numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>
            <mailto:numpy-discussion@scipy.org
    <mailto:numpy-discussion@scipy.org>>>>
                    *Subject:* Re: [Numpy-discussion] Intel random
    number
            package




                    On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
                    <oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>
            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>>
            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>

            <mailto:oleksandr.pavlyk@intel.com
    <mailto:oleksandr.pavlyk@intel.com>>>>
                    wrote:

                        Another point already raised by Nathaniel is
    that for
                        numpy's randomness ideally should provide a
    way to
            override
                        default algorithm for sampling from a particular
                        distribution.  For example RandomState
    object that
                        implements PCG may rely on default
    acceptance-rejection
                        algorithm for sampling from Gamma, while the
    RandomState
                        object that provides interface to MKL might
    want to call
                        into MKL directly.



                    The approach that pyfftw uses at least for
    scipy, which
            may also
                    work here, is that you can monkey-patch the
            scipy.fftpack module
                    at runtime, replacing it with pyfftw's drop-in
    replacement.
                    scipy then proceeds to use pyfftw instead of its
    built-in
                    fftpack implementation.  Might such an approach
    work here?
                    Users can either use this alternative randomstate
            replacement
                    directly, or they can replace numpy's with it at
    runtime and
                    numpy will then proceed to use the alternative.


                The only reason that pyfftw uses monkeypatching is
    that the
            better
                approach is not possible due to license constraints with
            FFTW (it's
                GPL).


            Yes, that is exactly why I brought it up.  Better
    approaches are
            also
            not possible with MKL due to license constraints.  It is
    a very
            similar
            situation overall.


        Its not that similar, the better approach is certainly
    possible with
        FFTW, the GPL is compatible with numpys license. It is only a
        concern users of binary distributions. Nobody provided the
    code to
        use fftw yet, but it would certainly be accepted.


    Although it is technically compatible, it would make numpy
    effectively
    GPL.  Suggestions for this have been explicitly rejected on these
    grounds [1]

    [1] https://github.com/numpy/numpy/issues/3485
    <https://github.com/numpy/numpy/issues/3485>


Yes it would make numpy GPL, but that is not a concern for a lot of
users. Users for who it is a problem can still use the non-GPL version.
A more interesting debate is whether our binary wheels should then
be GPL wheels by default or not. Probably not, but that is something
that should be discussed when its an actual issue.

But to clarify what I said, it would be accepted if the value it
provides is sufficient compared to the code maintenance it adds.
Given that pyfftw already exists the value is probably relatively
small, but personally I'd still be interested in code that allows
switching the fft backend as that could also allow plugging e.g. gpu
based implementations (though again this is already covered by other
third party modules).

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>
https://mail.scipy.org/mailman/listinfo/numpy-discussion
<https://mail.scipy.org/mailman/listinfo/numpy-discussion>

-- Robert McLeod, Ph.D. Center for Cellular Imaging and Nano Analytics (C-CINA) Biozentrum der Universität Basel Mattenstrasse 26, 4058 Basel Work: +41.061.387.3225 robert.mcleod@unibas.ch mailto:robert.mcleod@unibas.ch robert.mcleod@bsse.ethz.ch mailto:robert.mcleod@ethz.ch robbmcleod@gmail.com mailto:robbmcleod@gmail.com

---

NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

On Thu, Oct 27, 2016 at 12:12 PM, Nathaniel Smith njs@pobox.com wrote:

Ever notice how Anaconda doesn't provide pyfftw? They can't legally ship both MKL and pyfftw, and they picked MKL.

Anaconda does ship GPL code [1]. They even ship GPL code that depends on numpy, such as cvxcanon and pystan, and there doesn't seem to be anything that prevents me from installing them alongside the MKL version of numpy. So I don't see how it would be any different for pyfftw.

[1] https://docs.continuum.io/anaconda/pkg-docs