[Numpy-discussion] New package to speed up ufunc inner loops

[Aaron Meurer]

I hope this isn't too off topic, but this "fair play" NEP reads like
it is a set of additional restrictions on the NumPy license, which if
it is, would make NumPy no longer open source by the OSI definition. I
think the NEP should be much clearer that these are requests but not
requirements.

Aaron Meurer

On Wed, Nov 4, 2020 at 2:44 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Tue, Nov 3, 2020 at 3:54 PM Matti Picus <matti.picus at gmail.com> wrote:
>>
>> Hi. On behalf of Quansight and RTOSHoldings, I would like to introduce
>> "pnumpy", a package to speed up NumPy.
>>
>> https://quansight.github.io/numpy-threading-extensions/stable/index.html
>>
>>
>> What is in it?
>>
>> - use "PyUFunc_ReplaceLoopBySignature" to hook all the UFunc inner loops
>>
>> - When the inner loop is called with a large enough array, chunk the
>> data and perform the iteration via a thread pool
>>
>> - Add a different memory allocator for "ndarray" data (will require an
>> appropriate API from NumPy)
>>
>> - Allow using optimized loops above and beyond what NumPy provides
>>
>> - Allow logging inner loop calls and parameters to learn about the
>> current process and perhaps tune the performance accordingly
>>
>>
>> The first release contains the hooking mechanism and the thread pool,
>> the rest has been prototyped but is not ready for release. The idea
>> behind the package is that a third-party package can try things out and
>> iterate much faster than NumPy. If some of the ideas bear fruit, and do
>> not add an undue maintenance burden to NumPy, the code can be ported to
>> NumPy. I am not sure NumPy wishes to take upon itself the burden of
>> managing threads, but a third-party package may be able to.
>>
>>
>> I am writing to the mailing list both to announce the pre-release under
>> the wrong name, and, in accordance with the fair play rules[1], to
>> request use of the "numpy" name in the package. We had considered many
>> options, in the end would like to propose "pnumpy" (the p is either
>> "parallel" or "performant" or "preliminary", whatever you desire).
>
>
> Thanks Matti!
>
> Obviously as another Quansight employee I have a conflict of interest here, so let me just say I wasn't involved with choosing the `pnumpy` name but did already comment internally on using "numpy" as part of the package name would probably be fine, given that Matti is the main author and the intent is to migrate the useful parts into NumPy itself.
>
> Hopefully someone else can comment, maybe Stéfan as the "fair play" NEP author?
>
> Cheers,
> Ralf
>
>
>>
>>
>> Matti
>>
>>
>> [1] https://numpy.org/neps/nep-0036-fair-play.html#fair-play-rules
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion