From jni at fastmail.com Mon Jul 1 00:08:20 2019 From: jni at fastmail.com (Juan Nunez-Iglesias) Date: Mon, 01 Jul 2019 14:08:20 +1000 Subject: [Numpy-discussion] NumPy 1.17.0rc1 released In-Reply-To: References: Message-ID: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> Hi Chuck, and thanks for putting this together! It seems the release has broken existing uses of dask array with `np.min` (I presume among other functions): https://github.com/dask/dask/issues/5031 Perhaps `__array_function__` should be switched off for one more release cycle? I imagine that scikit-image [1] are not the only ones using this construct, which worked fine before `__array_function__`. Thank goodness (and you!) for pre-releases! ;) Juan. .. [1]: https://github.com/scikit-image/scikit-image/issues/3985 On Mon, 1 Jul 2019, at 8:48 AM, Charles R Harris wrote: > Hi All, > > On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0rc1. The 1.17 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b1 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: > * A new extensible random module along with four selectable random numbe5 generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. > * NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. > * New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. > * Overriding numpy functions is now possible by default > Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . > ** > *Contributors* > > A total of 142 people contributed to this release. People with a "+" by their > names contributed a patch for the first time. > * Aaron Voelker + > * Abdur Rehman + > * Abdur-Rahmaan Janhangeer + > * Abhinav Sagar + > * Adam J. Stewart + > * Adam Orr + > * Albert Thomas + > * Alex Watt + > * Alexander Blinne + > * Alexander Shadchin > * Allan Haldane > * Ander Ustarroz + > * Andras Deak > * Andreas Schwab > * Andrew Naguib + > * Andy Scholand + > * Ankit Shukla + > * Anthony Sottile > * Antoine Pitrou > * Antony Lee > * Arcesio Castaneda Medina + > * Assem + > * Bernardt Duvenhage + > * Bharat Raghunathan + > * Bharat123rox + > * Bran + > * Bruce Merry + > * Charles Harris > * Chirag Nighut + > * Christoph Gohlke > * Christopher Whelan + > * Chuanzhu Xu + > * Daniel Hrisca > * Daniel Lawrence + > * Debsankha Manik + > * Dennis Zollo + > * Dieter Werthm?ller + > * Dominic Jack + > * EelcoPeacs + > * Eric Larson > * Eric Wieser > * Fabrice Fontaine + > * Gary Gurlaskie + > * Gregory Lee + > * Gregory R. Lee > * Hameer Abbasi > * Haoyu Sun + > * He Jia + > * Hunter Damron + > * Ian Sanders + > * Ilja + > * Isaac Virshup + > * Isaiah Norton + > * Jaime Fernandez > * Jakub Wilk > * Jan S. (Milania1) + > * Jarrod Millman > * Javier Dehesa + > * Jeremy Lay + > * Jim Turner + > * Jingbei Li + > * Joachim Hereth + > * John Belmonte + > * John Kirkham > * John Law + > * Jonas Jensen > * Joseph Fox-Rabinovitz > * Joseph Martinot-Lagarde > * Josh Wilson > * Juan Luis Cano Rodr?guez > * Julian Taylor > * J?r?mie du Boisberranger + > * Kai Striega + > * Katharine Hyatt + > * Kevin Sheppard > * Kexuan Sun > * Kiko Correoso + > * Kriti Singh + > * Lars Grueter + > * Maksim Shabunin + > * Manvi07 + > * Mark Harfouche > * Marten van Kerkwijk > * Martin Reinecke + > * Matthew Brett > * Matthias Bussonnier > * Matti Picus > * Michel Fruchart + > * Mike Lui + > * Mike Taves + > * Min ho Kim + > * Mircea Akos Bruma > * Nick Minkyu Lee > * Nick Papior > * Nick R. Papior + > * Nicola Soranzo + > * Nimish Telang + > * OBATA Akio + > * Oleksandr Pavlyk > * Ori Broda + > * Paul Ivanov > * Pauli Virtanen > * Peter Andreas Entschev + > * Peter Bell + > * Pierre de Buyl > * Piyush Jaipuriayar + > * Prithvi MK + > * Raghuveer Devulapalli + > * Ralf Gommers > * Richard Harris + > * Rishabh Chakrabarti + > * Riya Sharma + > * Robert Kern > * Roman Yurchak > * Ryan Levy + > * Sebastian Berg > * Sergei Lebedev + > * Shekhar Prasad Rajak + > * Stefan van der Walt > * Stephan Hoyer > * SuryaChand P + > * S?ren Rasmussen + > * Thibault Hallouin + > * Thomas A Caswell > * Tobias Uelwer + > * Tony LaTorre + > * Toshiki Kataoka > * Tyler Moncur + > * Tyler Reddy > * Valentin Haenel > * Vrinda Narayan + > * Warren Weckesser > * Weitang Li > * Wojtek Ruszczewski > * Yu Feng > * Yu Kobayashi + > * Yury Kirienko + > * @aashuli + > * @euronion + > * @luzpaz > * @parul + > * @spacescientist + > Cheers, > > Charles Harris > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jul 1 00:33:23 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 1 Jul 2019 06:33:23 +0200 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> Message-ID: On Mon, Jul 1, 2019 at 6:08 AM Juan Nunez-Iglesias wrote: > Hi Chuck, and thanks for putting this together! > > It seems the release has broken existing uses of dask array with `np.min` > (I presume among other functions): > > https://github.com/dask/dask/issues/5031 > > Perhaps `__array_function__` should be switched off for one more release > cycle? I imagine that scikit-image [1] are not the only ones using this > construct, which worked fine before `__array_function__`. > Hmm, I'd really like to avoid that, that would be another 6 months where we get close to zero testing. This issue is not very surprising - __array_function__ is going to have a fair bit of backwards compat impact for people who were relying on feeding all sorts of stuff into numpy functions that previously got converted with asarray. At this point Dask is the main worry, followed by CuPy and pydata/sparse. All those libraries have very responsive maintainers. Perhaps we should just try to get these issues fixed asap in those libraries instead? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni at fastmail.com Mon Jul 1 01:37:10 2019 From: jni at fastmail.com (Juan Nunez-Iglesias) Date: Mon, 01 Jul 2019 15:37:10 +1000 Subject: [Numpy-discussion] =?utf-8?q?=5F=5Farray=5Ffunction_related_regr?= =?utf-8?q?ession_for_1=2E17=2E0rc1?= In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> Message-ID: <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> On Mon, 1 Jul 2019, at 2:34 PM, Ralf Gommers wrote: > This issue is not very surprising - __array_function__ is going to have a fair bit of backwards compat impact for people who were relying on feeding all sorts of stuff into numpy functions that previously got converted with asarray. At this point Dask is the main worry, followed by CuPy and pydata/sparse. All those libraries have very responsive maintainers. Perhaps we should just try to get these issues fixed asap in those libraries instead? Fixing them is not sufficient, because many people are still going to end up with broken code unless they are bleeding-edge with everything. It's best to minimise the number of forbidden version combinations. Your suggestion on the issue to switch from typeerror to warning is, imho, much better, as long as the warning contains a link to an issue/webpage explaining what needs to happen. It's only because I've been vaguely aware of the `__array_function__` discussions that I was able to diagnose relatively quickly. The average user would be very confused by this code break or by a warning, and be unsure of what they need to do to get rid of the warning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jul 1 01:42:02 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 1 Jul 2019 07:42:02 +0200 Subject: [Numpy-discussion] PyPI NumPy account changes Message-ID: Hi all, PyPI now has taken 2FA (two-factor authentication) in production, which is a useful security measure I think. Also, Tidelift is able to measure which accounts have 2FA enabled. I had a look at our PyPI account, and there were many owners of it. This isn't great from a security perspective. There are two roles on PyPI: maintainer, and owner. Maintainers can upload, owners can add other people and delete the whole account. The old PyPI added anyone as owner by default, that's why we had so many. I already did some cleanup, removing people who having uploaded a release in 8+ years and/or were never a NumPy maintainer. I propose to clean this up a little further. We don't need more than 3-4 owners (for enough redundancy), converting the rest to maintainer or removing them would be better. Ideally everyone would also enable 2FA. Given who now has access, I propose as owners Charles Harris, Matthew Brett and myself. The other people who have access are fairly unlikely to do another release in the near to medium future (or ever), except probably Matti. So I propose that I make them maintainers now, and then send them an email whether they want to keep access or not. Does that sound okay? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jul 1 01:47:29 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 1 Jul 2019 07:47:29 +0200 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> Message-ID: On Mon, Jul 1, 2019 at 7:37 AM Juan Nunez-Iglesias wrote: > > > On Mon, 1 Jul 2019, at 2:34 PM, Ralf Gommers wrote: > > This issue is not very surprising - __array_function__ is going to have a > fair bit of backwards compat impact for people who were relying on feeding > all sorts of stuff into numpy functions that previously got converted with > asarray. At this point Dask is the main worry, followed by CuPy and > pydata/sparse. All those libraries have very responsive maintainers. > Perhaps we should just try to get these issues fixed asap in those > libraries instead? > > > Fixing them is not sufficient, because many people are still going to end > up with broken code unless they are bleeding-edge with everything. It's > best to minimise the number of forbidden version combinations. > Yes, fair enough. > Your suggestion on the issue to switch from typeerror to warning is, imho, > much better, as long as the warning contains a link to an issue/webpage > explaining what needs to happen. It's only because I've been vaguely aware > of the `__array_function__` discussions that I was able to diagnose > relatively quickly. The average user would be very confused by this code > break or by a warning, and be unsure of what they need to do to get rid of > the warning. > This would work I think. It's not even a band-aid, it's probably the better design option because any sane library that implements __array_function__ will have a much smaller API surface than NumPy - and why forbid users from feeding array-like input to the rest of the NumPy functions? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Jul 2 02:33:26 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 1 Jul 2019 23:33:26 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> Message-ID: > > Your suggestion on the issue to switch from typeerror to warning is, imho, >> much better, as long as the warning contains a link to an issue/webpage >> explaining what needs to happen. It's only because I've been vaguely aware >> of the `__array_function__` discussions that I was able to diagnose >> relatively quickly. The average user would be very confused by this code >> break or by a warning, and be unsure of what they need to do to get rid of >> the warning. >> > > This would work I think. It's not even a band-aid, it's probably the > better design option because any sane library that implements > __array_function__ will have a much smaller API surface than NumPy - and > why forbid users from feeding array-like input to the rest of the NumPy > functions? > This is addressed in the NEP, see bullet 1 under "Partial implementation of NumPy's API": http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api My concern is that fallback coercion behavior makes it difficult to reliably implement "strict" overrides of NumPy's API. Fallback coercion is certainly useful for interactive use, but it isn't really appropriate for libraries. In contrast to putting this into NumPy, if a library like dask prefers to issue warnings or even keep around fallback coercion indefinitely (not that I would recommend it), they can do that by putting it in their __array_function__ implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni at fastmail.com Tue Jul 2 04:43:37 2019 From: jni at fastmail.com (Juan Nunez-Iglesias) Date: Tue, 02 Jul 2019 18:43:37 +1000 Subject: [Numpy-discussion] =?utf-8?q?=5F=5Farray=5Ffunction_related_regr?= =?utf-8?q?ession_for_1=2E17=2E0rc1?= In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> Message-ID: <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote: > This is addressed in the NEP, see bullet 1 under "Partial implementation of NumPy's API": > http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api > > My concern is that fallback coercion behavior makes it difficult to reliably implement "strict" overrides of NumPy's API. Fallback coercion is certainly useful for interactive use, but it isn't really appropriate for libraries. > > In contrast to putting this into NumPy, if a library like dask prefers to issue warnings or even keep around fallback coercion indefinitely (not that I would recommend it), they can do that by putting it in their __array_function__ implementation. I get the above concerns, and thanks for bringing them up, Stephan, as I'd only skimmed the NEP the first time around and missed them. Nevertheless, the fact is that the current behaviour breaks user code that was perfectly valid until NumPy 1.16, which seems, well, insane. So, warning for a few versions followed raising seems like the only way forward to me. The NEP explicitly states ?We would like to gain experience with how `__array_function__` is actually used before making decisions that would be difficult to roll back.? I think that this breakage *is* that experience, and the decision right now should be not to break user code with no warning period. I'm also wondering where the list of functions that must be implemented can be found, so that libraries like dask and CuPy can be sure that they have a complete implementation, and further typeerrors won't be raised with their arrays. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 2 11:15:21 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 2 Jul 2019 08:15:21 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> Message-ID: On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias wrote: > On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote: > > This is addressed in the NEP, see bullet 1 under "Partial implementation > of NumPy's API": > > http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api > > My concern is that fallback coercion behavior makes it difficult to > reliably implement "strict" overrides of NumPy's API. Fallback coercion is > certainly useful for interactive use, but it isn't really appropriate for > libraries. > > Do you mean "fallback coercion in NumPy itself", or "at all"? Right now there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users will keep wanting to do that. Forcing everyone to write `np.median(np.array(some_dask_array))` serves no purpose. So the coercion has to be somewhere. You're arguing that that's up to Dask et al I think? Putting it in Dask right now still doesn't address Juan's backwards compat concern, but perhaps that could be bridged with a Dask bugfix release and some short-lived pain. I'm not convinced that this shouldn't be fixed in NumPy though. Your concern "reliably implement "strict" overrides of NumPy's API" is a bit abstract. Overriding the _whole_ NumPy API is definitely undesirable. If we'd have a reference list somewhere about every function that is handled with __array_function__, then would that address your concern? Such a list could be auto-generated fairly easily. > > In contrast to putting this into NumPy, if a library like dask prefers to > issue warnings or even keep around fallback coercion indefinitely (not that > I would recommend it), they can do that by putting it in their > __array_function__ implementation. > > > I get the above concerns, and thanks for bringing them up, Stephan, as I'd > only skimmed the NEP the first time around and missed them. Nevertheless, > the fact is that the current behaviour breaks user code that was perfectly > valid until NumPy 1.16, which seems, well, insane. So, warning for a few > versions followed raising seems like the only way forward to me. The NEP > explicitly states ?We would like to gain experience with how > __array_function__ is actually used before making decisions that would be > difficult to roll back.? I think that this breakage *is* that experience, > and the decision right now should be not to break user code with no warning > period. > > I'm also wondering where the list of functions that must be implemented > can be found, so that libraries like dask and CuPy can be sure that they > have a complete implementation, and further typeerrors won't be raised with > their arrays. > This is one of the reasons I'm working on https://github.com/Quansight-Labs/rnumpy. It doesn't make sense for any library to copy the whole NumPy API, it's way too large with lots of stuff in there that's only there for backwards compat and has a better alternative or shouldn't be in NumPy in the first place. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Jul 2 11:19:23 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 2 Jul 2019 08:19:23 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> Message-ID: On Tue, Jul 2, 2019 at 1:46 AM Juan Nunez-Iglesias wrote: > I'm also wondering where the list of functions that must be implemented > can be found, so that libraries like dask and CuPy can be sure that they > have a complete implementation, and further typeerrors won't be raised with > their arrays. > This is a good question. We don't have a master list currently. In practice, I would be surprised if there is ever more than exactly one full implementation of NumPy's full API. We added dispatch with __array_function__ even to really obscure corners of NumPy's API, e.g., np.lib.scimath. The short answer right now is "Any publicly exposed function that says it takes array-like arguments, aside from functions specifically for coercing to NumPy arrays and the functions in numpy.testing." -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Jul 2 11:37:53 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 2 Jul 2019 08:37:53 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> Message-ID: On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers wrote: > > > On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias > wrote: > >> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote: >> >> This is addressed in the NEP, see bullet 1 under "Partial implementation >> of NumPy's API": >> >> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api >> >> My concern is that fallback coercion behavior makes it difficult to >> reliably implement "strict" overrides of NumPy's API. Fallback coercion is >> certainly useful for interactive use, but it isn't really appropriate for >> libraries. >> >> > Do you mean "fallback coercion in NumPy itself", or "at all"? Right now > there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users > will keep wanting to do that. Forcing everyone to write > `np.median(np.array(some_dask_array))` serves no purpose. So the coercion > has to be somewhere. You're arguing that that's up to Dask et al I think? > Yes, I'm arguing this is up to dask to maintain backwards compatibility -- or not, as the maintainers see fit. NumPy adding dispatching with __array_function__ did not break any existing code, until the maintainers of other libraries started adding __array_function__ methods. I hope that the risks of implementing such experimental methods were self-evident. > Putting it in Dask right now still doesn't address Juan's backwards compat > concern, but perhaps that could be bridged with a Dask bugfix release and > some short-lived pain. > I really think this is the best (only?) path forward. I'm not convinced that this shouldn't be fixed in NumPy though. Your > concern "reliably implement "strict" overrides of NumPy's API" is a bit > abstract. Overriding the _whole_ NumPy API is definitely undesirable. If > we'd have a reference list somewhere about every function that is handled > with __array_function__, then would that address your concern? Such a list > could be auto-generated fairly easily. > By "reliably implement strict overrides" I mean the ability to ensure that every operation either uses an override or raises an informative error -- making it very clear which operation needs to be implemented or avoided. It's true that we didn't really consider "always issuing warnings" as a long term solution in the NEP. I can see how this would simply a backwards compatibility story for libraries like dask, but in general, I really don't like warnings: Using them like exceptions can easily result in code that is partially broken or that fails later for non-obvious reasons. There's a reason why Python's errors stop execution flow, until errors in languages like PHP or JavaScript. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Tue Jul 2 13:00:33 2019 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 2 Jul 2019 19:00:33 +0200 Subject: [Numpy-discussion] ANN: SfePy 2019.2 Message-ID: I am pleased to announce release 2019.2 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - improved support for time-dependent homogenization problems, - Python 3.7 compatibility For full release notes see [1]. Cheers, Robert Cimrman [1] http://docs.sfepy.org/doc/release_notes.html#id1 --- Contributors to this release in alphabetical order: Robert Cimrman Jan Heczko Lubos Kejzlar Vladimir Lukes From sebastian at sipsolutions.net Tue Jul 2 13:57:50 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 02 Jul 2019 10:57:50 -0700 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, July 3 Message-ID: <0f79979bb1f7f61eb55329aa99a2b2b8010d65f1.camel@sipsolutions.net> Hi all, There will be a NumPy Community meeting Wednesday July 3 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg Best wishes Sebastian -------------- next part -------------- _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Tue Jul 2 16:15:52 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 2 Jul 2019 13:15:52 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> Message-ID: On Tue, Jul 2, 2019 at 8:38 AM Stephan Hoyer wrote: > On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers > wrote: > >> >> >> On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias >> wrote: >> >>> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote: >>> >>> This is addressed in the NEP, see bullet 1 under "Partial implementation >>> of NumPy's API": >>> >>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api >>> >>> My concern is that fallback coercion behavior makes it difficult to >>> reliably implement "strict" overrides of NumPy's API. Fallback coercion is >>> certainly useful for interactive use, but it isn't really appropriate for >>> libraries. >>> >>> >> Do you mean "fallback coercion in NumPy itself", or "at all"? Right now >> there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users >> will keep wanting to do that. Forcing everyone to write >> `np.median(np.array(some_dask_array))` serves no purpose. So the coercion >> has to be somewhere. You're arguing that that's up to Dask et al I think? >> > > Yes, I'm arguing this is up to dask to maintain backwards compatibility -- > or not, as the maintainers see fit. > > NumPy adding dispatching with __array_function__ did not break any > existing code, until the maintainers of other libraries started adding > __array_function__ methods. I hope that the risks of implementing such > experimental methods were self-evident. > Yeah, that's a bit of a chicken-and-egg story though. We add something and try to be "strict". Dask adds something because they like the idea and generally are quick to adopt these types of things. If we make it too hard to be backwards compatible, then neither NumPy nor Dask may try and it ends up breaking scikit-image & co. I for one don't care where the fix lands, but it's pretty to me that breaking scikit-image is the worst of all options. > >> Putting it in Dask right now still doesn't address Juan's backwards >> compat concern, but perhaps that could be bridged with a Dask bugfix >> release and some short-lived pain. >> > > I really think this is the best (only?) path forward. > I think I agree (depending on how easy it is to get the Dask fix landed). > > I'm not convinced that this shouldn't be fixed in NumPy though. Your >> concern "reliably implement "strict" overrides of NumPy's API" is a bit >> abstract. Overriding the _whole_ NumPy API is definitely undesirable. If >> we'd have a reference list somewhere about every function that is handled >> with __array_function__, then would that address your concern? Such a list >> could be auto-generated fairly easily. >> > > By "reliably implement strict overrides" I mean the ability to ensure that > every operation either uses an override or raises an informative error -- > making it very clear which operation needs to be implemented or avoided. > That isn't necessarily a good goal in itself though. In many cases, an `asarray` call still needs to go *somewhere*. If the "reliably implement strict overrides" is to help library authors, then there may be other ways to do that. For end users it can only hurt; those TypeErrors aren't exactly easy to understand. > It's true that we didn't really consider "always issuing warnings" as a > long term solution in the NEP. I can see how this would simply a backwards > compatibility story for libraries like dask, but in general, I really don't > like warnings: > I agree. Cheers, Ralf Using them like exceptions can easily result in code that is partially > broken or that fails later for non-obvious reasons. There's a reason why > Python's errors stop execution flow, until errors in languages like PHP or > JavaScript. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Jul 2 18:18:30 2019 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 2 Jul 2019 15:18:30 -0700 Subject: [Numpy-discussion] Top level release index page Message-ID: In PR 13886 I reworked the way the link to the release notes is generated. The current page is http://www.numpy.org/devdocs/release.html and? the new page is https://8001-908607-gh.circle-artifacts.com/0/home/circleci/repo/doc/build/html/release.html I don't like the "wall of text" in the current page. On the other hand, it is nice to use CTRL-F to search the page itself for answers to questions like "what release deprecated indexing by float", which is why I left the level-of-detail at 3 (1 - release, 2 - sections, 3 - item header). Should the level-of-detail be reduced to only single links to the release document? An alternative would be to render the contents as a collapsible list, which would require some javascript. Matti From ralf.gommers at gmail.com Tue Jul 2 20:20:41 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 2 Jul 2019 17:20:41 -0700 Subject: [Numpy-discussion] Top level release index page In-Reply-To: References: Message-ID: On Tue, Jul 2, 2019 at 3:18 PM Matti Picus wrote: > In PR 13886 I reworked the way the link to the release notes is > generated. The current page is > > > http://www.numpy.org/devdocs/release.html > > > and the new page is > > > > https://8001-908607-gh.circle-artifacts.com/0/home/circleci/repo/doc/build/html/release.html > > > I don't like the "wall of text" in the current page. On the other hand, > it is nice to use CTRL-F to search the page itself for answers to > questions like "what release deprecated indexing by float", which is why > I left the level-of-detail at 3 (1 - release, 2 - sections, 3 - item > header). > > > Should the level-of-detail be reduced to only single links to the > release document? > I like your version, definitely nicer than what we have now, happy to go with your PR and level-of-detail 3 as is. > > An alternative would be to render the contents as a collapsible list, > which would require some javascript. > Hmm, custom JavaScript for navigating Sphinx-navigated docs seems wrong and is going to be a maintenance hassle at some point. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.j.a.cock at googlemail.com Wed Jul 3 05:17:23 2019 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 3 Jul 2019 10:17:23 +0100 Subject: [Numpy-discussion] Top level release index page In-Reply-To: References: Message-ID: On Tue, Jul 2, 2019 at 11:19 PM Matti Picus wrote: > In PR 13886 I reworked the way the link to the release notes is > generated. The current page is > > > http://www.numpy.org/devdocs/release.html > > > and the new page is > > > > https://8001-908607-gh.circle-artifacts.com/0/home/circleci/repo/doc/build/html/release.html > > > I don't like the "wall of text" in the current page. On the other hand, > it is nice to use CTRL-F to search the page itself for answers to > questions like "what release deprecated indexing by float", which is why > I left the level-of-detail at 3 (1 - release, 2 - sections, 3 - item > header). > > That seems a good compromise - certainly I have often used CTRL-F on release notes or change notes while tracing regressions - and having all the releases on one page makes this *so* much easier. Due to the way I typically use release notes, I would perhaps have just left the wall of text as is. > > Should the level-of-detail be reduced to only single links to the > release document? > > I'm unclear what you are suggesting here. > > An alternative would be to render the contents as a collapsible list, > which would require some javascript. > > Collapsible lists (which worked with CTRL-F searching) would be even better for balancing readability with utility, provided it was not too much trouble to implement and maintain of course. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Wed Jul 3 11:03:34 2019 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Wed, 3 Jul 2019 17:03:34 +0200 Subject: [Numpy-discussion] Problem with absolute value Message-ID: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Hi, I?m a bit surprised with the following code: >>> import numpy as np >>> np.seterr(all='warn') >>> Z = np.array([-128], dtype=np.byte) >>> print(np.abs(Z)) [-128] Obviously, it does not return the absolute value and I get no warning. Is it something expected ? (numpy 1.16.4, python 3.7.3) Nicolas From einstein.edison at gmail.com Wed Jul 3 11:07:52 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 3 Jul 2019 15:07:52 +0000 Subject: [Numpy-discussion] Problem with absolute value In-Reply-To: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> References: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Message-ID: Hi, It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well. TL;DR: It's an error with how the integer is stored in memory and how you're running out of space. Regards, Hameer Abbasi ?On 03.07.19, 17:03, "NumPy-Discussion on behalf of Nicolas Rougier" wrote: Hi, I?m a bit surprised with the following code: >>> import numpy as np >>> np.seterr(all='warn') >>> Z = np.array([-128], dtype=np.byte) >>> print(np.abs(Z)) [-128] Obviously, it does not return the absolute value and I get no warning. Is it something expected ? (numpy 1.16.4, python 3.7.3) Nicolas _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Wed Jul 3 12:04:56 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 3 Jul 2019 10:04:56 -0600 Subject: [Numpy-discussion] Problem with absolute value In-Reply-To: References: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Message-ID: On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi wrote: > Hi, > > It turns out you're running into a bit-error. In general, the two's > complement of -2 ** (n-1) with the bit-length being limited to n bits is > itself... No way around that. And integers don't set hardware exceptions so > checking for errors like these is hard as well. > > TL;DR: It's an error with how the integer is stored in memory and how > you're running out of space. > > Regards, > Hameer Abbasi > More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Jul 3 13:38:04 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 3 Jul 2019 10:38:04 -0700 Subject: [Numpy-discussion] Problem with absolute value In-Reply-To: References: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Message-ID: Hi, On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris wrote: > > > > On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi wrote: >> >> Hi, >> >> It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well. >> >> TL;DR: It's an error with how the integer is stored in memory and how you're running out of space. >> >> Regards, >> Hameer Abbasi > > > More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype. I still think this is a major wart in numpy's abs. I'd really like to add a new function, `uabs` which would return an unsigned int for integer inputs. I'm happy to do a pull request if that also seems sensible to y'all, Cheers, Matthew From charlesr.harris at gmail.com Wed Jul 3 14:01:50 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 3 Jul 2019 12:01:50 -0600 Subject: [Numpy-discussion] Problem with absolute value In-Reply-To: References: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Message-ID: On Wed, Jul 3, 2019 at 11:39 AM Matthew Brett wrote: > Hi, > > On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris > wrote: > > > > > > > > On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi > wrote: > >> > >> Hi, > >> > >> It turns out you're running into a bit-error. In general, the two's > complement of -2 ** (n-1) with the bit-length being limited to n bits is > itself... No way around that. And integers don't set hardware exceptions so > checking for errors like these is hard as well. > >> > >> TL;DR: It's an error with how the integer is stored in memory and how > you're running out of space. > >> > >> Regards, > >> Hameer Abbasi > > > > > > More like the eight bit twos complement of -128 is -128, bytes cannot > represent 128. Matlab used to (still does?) solve this problem by returning > 127 instead :) Basically, the data needs more precision. Returning an > unsigned type would lead to it's own problems with unexpected promotions > when the result was used. We could, I suppose, raise a warning, although > that might be considered noisy. If you use `absolute` instead, you can > specify the dtype. > > I still think this is a major wart in numpy's abs. > > I'd really like to add a new function, `uabs` which would return an > unsigned int for integer inputs. I'm happy to do a pull request if > that also seems sensible to y'all, > > Seems reasonable, it would provide something for discussion. Note that the current abs is in the slot provided by Python numeric types. The main problem is that as soon as the unsigned result is combined with a signed type it will get promoted. Which would not be much of a problem except int64 -> double. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Wed Jul 3 14:15:50 2019 From: Nicolas.Rougier at inria.fr (Nicolas Rougier) Date: Wed, 3 Jul 2019 20:15:50 +0200 Subject: [Numpy-discussion] Problem with absolute value In-Reply-To: References: <4FD41D38-9017-49FC-927B-2BBB5FAE634F@inria.fr> Message-ID: Thanks for the quick explanation & answers. I was mostly puzzled by the absence of warning. In C you would get an overflow warning in such a case, no ? Nicolas > On 3 Jul 2019, at 20:01, Charles R Harris wrote: > > > > On Wed, Jul 3, 2019 at 11:39 AM Matthew Brett wrote: > Hi, > > On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris > wrote: > > > > > > > > On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi wrote: > >> > >> Hi, > >> > >> It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well. > >> > >> TL;DR: It's an error with how the integer is stored in memory and how you're running out of space. > >> > >> Regards, > >> Hameer Abbasi > > > > > > More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype. > > I still think this is a major wart in numpy's abs. > > I'd really like to add a new function, `uabs` which would return an > unsigned int for integer inputs. I'm happy to do a pull request if > that also seems sensible to y'all, > > > Seems reasonable, it would provide something for discussion. Note that the current abs is in the slot provided by Python numeric types. The main problem is that as soon as the unsigned result is combined with a signed type it will get promoted. Which would not be much of a problem except int64 -> double. > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion From einstein.edison at gmail.com Wed Jul 3 14:57:31 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Wed, 3 Jul 2019 18:57:31 +0000 Subject: [Numpy-discussion] uarray update: API changes, overhead and comparison to __array_function__ Message-ID: Hello all, I just wrote a blog post on uarray and how it compares against NumPy?s __array_function__ protocol. Give it a read here: https://labs.quansight.org/blog/2019/07/uarray-update-api-changes-overhead-and-comparison-to-__array_function__/ Best Regards, Hameer Abbasi -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at kianasun.com Thu Jul 4 03:04:49 2019 From: me at kianasun.com (Kexuan Sun) Date: Thu, 4 Jul 2019 00:04:49 -0700 Subject: [Numpy-discussion] Code review for adding axis argument to permutation and shuffle function Message-ID: Hi, I would like to request a code review. The random.permutation and random.shuffle functions now can only shuffle along the first axis of a multi-dimensional array. I propose to add an axis argument for the functions and allow them to shuffle along a given axis. Here is the link to the PR (https://github.com/numpy/numpy/pull/13829). Thanks! From matti.picus at gmail.com Thu Jul 4 13:56:28 2019 From: matti.picus at gmail.com (Matti Picus) Date: Thu, 4 Jul 2019 10:56:28 -0700 Subject: [Numpy-discussion] Shippable builds are broken - I filed an issue with them Message-ID: <3a793bd1-c271-1206-295c-5349c5f97f4e@gmail.com> For the past few days shippable (arm builds) CI has been failing. I opened an issue with them https://github.com/Shippable/support/issues/4882 From ralf.gommers at gmail.com Fri Jul 5 13:59:54 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 5 Jul 2019 10:59:54 -0700 Subject: [Numpy-discussion] __array_function related regression for 1.17.0rc1 In-Reply-To: References: <1c1cee81-131a-4c96-9e1d-73a38c6f9a9d@www.fastmail.com> <76bbb235-fd83-45fa-b30b-be674cc8e0b8@www.fastmail.com> <963d6b85-3948-4b8d-828d-2c270b9a6bbe@www.fastmail.com> Message-ID: On Tue, Jul 2, 2019 at 1:15 PM Ralf Gommers wrote: > > > On Tue, Jul 2, 2019 at 8:38 AM Stephan Hoyer wrote: > >> On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers >> wrote: >> >>> >>> >>> On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias >>> wrote: >>> >>>> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote: >>>> >>>> This is addressed in the NEP, see bullet 1 under "Partial >>>> implementation of NumPy's API": >>>> >>>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api >>>> >>>> My concern is that fallback coercion behavior makes it difficult to >>>> reliably implement "strict" overrides of NumPy's API. Fallback coercion is >>>> certainly useful for interactive use, but it isn't really appropriate for >>>> libraries. >>>> >>>> >>> Do you mean "fallback coercion in NumPy itself", or "at all"? Right now >>> there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users >>> will keep wanting to do that. Forcing everyone to write >>> `np.median(np.array(some_dask_array))` serves no purpose. So the coercion >>> has to be somewhere. You're arguing that that's up to Dask et al I think? >>> >> >> Yes, I'm arguing this is up to dask to maintain backwards compatibility >> -- or not, as the maintainers see fit. >> >> NumPy adding dispatching with __array_function__ did not break any >> existing code, until the maintainers of other libraries started adding >> __array_function__ methods. I hope that the risks of implementing such >> experimental methods were self-evident. >> > > Yeah, that's a bit of a chicken-and-egg story though. We add something and > try to be "strict". Dask adds something because they like the idea and > generally are quick to adopt these types of things. If we make it too hard > to be backwards compatible, then neither NumPy nor Dask may try and it ends > up breaking scikit-image & co. I for one don't care where the fix lands, > but it's pretty to me that breaking scikit-image is the worst of all > options. > > >> >>> Putting it in Dask right now still doesn't address Juan's backwards >>> compat concern, but perhaps that could be bridged with a Dask bugfix >>> release and some short-lived pain. >>> >> >> I really think this is the best (only?) path forward. >> > > I think I agree (depending on how easy it is to get the Dask fix landed). > That's landed, and Dask is planning a bugfix release in 2 days, so before the NumPy 1.17.0 release. So this is not a release blocker anymore for us I think. Cheers, Ralf >> I'm not convinced that this shouldn't be fixed in NumPy though. Your >>> concern "reliably implement "strict" overrides of NumPy's API" is a bit >>> abstract. Overriding the _whole_ NumPy API is definitely undesirable. If >>> we'd have a reference list somewhere about every function that is handled >>> with __array_function__, then would that address your concern? Such a list >>> could be auto-generated fairly easily. >>> >> >> By "reliably implement strict overrides" I mean the ability to ensure >> that every operation either uses an override or raises an informative error >> -- making it very clear which operation needs to be implemented or avoided. >> > > That isn't necessarily a good goal in itself though. In many cases, an > `asarray` call still needs to go *somewhere*. If the "reliably implement > strict overrides" is to help library authors, then there may be other ways > to do that. For end users it can only hurt; those TypeErrors aren't exactly > easy to understand. > > >> It's true that we didn't really consider "always issuing warnings" as a >> long term solution in the NEP. I can see how this would simply a backwards >> compatibility story for libraries like dask, but in general, I really don't >> like warnings: >> > > I agree. > > Cheers, > Ralf > > Using them like exceptions can easily result in code that is partially >> broken or that fails later for non-obvious reasons. There's a reason why >> Python's errors stop execution flow, until errors in languages like PHP or >> JavaScript. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anurag at ph.iitr.ac.in Mon Jul 8 02:40:19 2019 From: anurag at ph.iitr.ac.in (ANURAG ANURAG) Date: Mon, 8 Jul 2019 14:40:19 +0800 Subject: [Numpy-discussion] Regarding adding new function in Numpy Message-ID: Sir, I was recently converting some Matlab codes into python, where I recently stumble upon bitget(n,x) function of Matlab, b = bitget(A,bit) returns the bit value at the position bit in integer array A. I was unable to find any Numpy equivalent of this, so should I create a pull request in Numpy for the same! Regards, Anurag -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jul 8 05:55:01 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 8 Jul 2019 02:55:01 -0700 Subject: [Numpy-discussion] Regarding adding new function in Numpy In-Reply-To: References: Message-ID: Hi, On Sun, Jul 7, 2019 at 11:41 PM ANURAG ANURAG wrote: > > Sir, > I was recently converting some Matlab codes into python, where I recently stumble upon bitget(n,x) function of Matlab, b = bitget(A,bit) returns the bit value at the position bit in integer array A. I was unable to find any Numpy equivalent of this, so should I create a pull request in Numpy for the same! Thanks for the suggestion. I think Eric Wieser is right in his comment on the related pull request: https://github.com/numpy/numpy/pull/13937#issuecomment-509091992 The function is a one-liner in Python - albeit a slightly fancy one-liner. Also - I have never had need of this function in Matlab or Numpy, and had no idea it existed. So, I'd vote against, unless it turned out it was much more common in use than my experience would suggest, Cheers, Matthew From sebastian at sipsolutions.net Mon Jul 8 13:38:29 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 08 Jul 2019 10:38:29 -0700 Subject: [Numpy-discussion] No Community Meeting this Week Message-ID: <84c1b7121f9a12f04d34fdb128212b8323b5bbe0.camel@sipsolutions.net> Hi all, due to the SciPy conference we will _not_ have a Community meeting this week. We will have a developer meeting at the SciPy conference on Friday at 1pm in room 108 and take part in the Sprints. I will send a tentative Agenda/things to discuss in a separate email. Best, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Mon Jul 8 15:56:01 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 08 Jul 2019 12:56:01 -0700 Subject: [Numpy-discussion] Developer Meeting @SciPy Conference, Friday 1pm Message-ID: <65689a66a0884f8c41964a11831b9993f7171219.camel@sipsolutions.net> Hi all, we have reserved a room for a Developer meeting at SciPy. The meeting is planned for Friday at 1pm in room 108. I have created a tentative Agenda with some broad discussion points at: https://hackmd.io/rg0DI0JASKeZ_G5bTJ3YSA Please feel free to edit or add the Agenda. All the Best, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From matti.picus at gmail.com Tue Jul 9 20:31:24 2019 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 9 Jul 2019 19:31:24 -0500 Subject: [Numpy-discussion] Adding a "suffle=True" kwarg to numpy.random.Generator.choice Message-ID: <3c186b17-b05b-4ea6-9c3d-0d10d88409fc@gmail.com> In PR https://github.com/numpy/numpy/pull/13812, Thrasibule rewrote the algorithm used with a faster alternative branch for some cases. The faster algorithm does not necessarily shuffle the results, so for instance gen.choice(2000, 2000, replace=False) may simply return arange(2000). In the old code the result is always shuffled. We propose adding a new kwarg "shuffle" that defaults to True. Users looking for maximum performance may choose to use shuffle=False. Since this is a behavioral change (although only in the new Generator class, the new code will not be used in RandomState), we are proposing it to the mailing list Any thoughts? Matti From shoyer at gmail.com Tue Jul 9 21:20:08 2019 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 9 Jul 2019 18:20:08 -0700 Subject: [Numpy-discussion] Adding a "suffle=True" kwarg to numpy.random.Generator.choice In-Reply-To: <3c186b17-b05b-4ea6-9c3d-0d10d88409fc@gmail.com> References: <3c186b17-b05b-4ea6-9c3d-0d10d88409fc@gmail.com> Message-ID: This sounds like a welcome backwards compatible option for more performance. I imagine there are plenty of applications (e.g., sets) where shuffled order doesn't matter. +1 from me. On Tue, Jul 9, 2019 at 5:32 PM Matti Picus wrote: > In PR https://github.com/numpy/numpy/pull/13812, Thrasibule rewrote the > algorithm used with a faster alternative branch for some cases. The > faster algorithm does not necessarily shuffle the results, so for > instance gen.choice(2000, 2000, replace=False) may simply return > arange(2000). In the old code the result is always shuffled. We propose > adding a new kwarg "shuffle" that defaults to True. Users looking for > maximum performance may choose to use shuffle=False. > > > Since this is a behavioral change (although only in the new Generator > class, the new code will not be used in RandomState), we are proposing > it to the mailing list > > Any thoughts? > > > Matti > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Jul 11 07:45:36 2019 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 11 Jul 2019 07:45:36 -0400 Subject: [Numpy-discussion] float16 but no complex32? Message-ID: I see a float16 dtype but not complex32. Is this an oversight? Thanks, Neal -- Those who don't understand recursion are doomed to repeat it From cameronjblocker at gmail.com Thu Jul 11 15:59:52 2019 From: cameronjblocker at gmail.com (Cameron Blocker) Date: Thu, 11 Jul 2019 15:59:52 -0400 Subject: [Numpy-discussion] float16 but no complex32? In-Reply-To: References: Message-ID: I would use a complex32 dtype if it existed, whether provided by numpy or another library. My guess would be that there was not much demand for a complex32 datatype since float16s are slow and are generally used as a storage format [1] and you could easily store a complex array as two float16 arrays and get the same space savings. That said, I am occasionally storing complex-valued validation data in memory, and the datatype would make it more convenient. I just don't know how common my use case is. Maybe there are more compelling use cases? I know some GPUs natively support float16, I'm not sure how common complex32 support is though. [1] https://stackoverflow.com/a/24590380/5026175 On Thu, Jul 11, 2019 at 7:47 AM Neal Becker wrote: > I see a float16 dtype but not complex32. Is this an oversight? > > Thanks, > Neal > > -- > Those who don't understand recursion are doomed to repeat it > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Jul 12 13:05:00 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 12 Jul 2019 12:05:00 -0500 Subject: [Numpy-discussion] Developer Meeting @SciPy Conference, Friday 1pm In-Reply-To: <65689a66a0884f8c41964a11831b9993f7171219.camel@sipsolutions.net> References: <65689a66a0884f8c41964a11831b9993f7171219.camel@sipsolutions.net> Message-ID: Hi all, just a quick reminder that we have the meeting in about an hour. If you wish to attend/chat from externally, please shoot me an email, I am sure we can set up something. Best, Sebastian On Mon, 2019-07-08 at 12:56 -0700, Sebastian Berg wrote: > Hi all, > > we have reserved a room for a Developer meeting at SciPy. The meeting > is planned for Friday at 1pm in room 108. > > I have created a tentative Agenda with some broad discussion points > at: > https://hackmd.io/rg0DI0JASKeZ_G5bTJ3YSA > > Please feel free to edit or add the Agenda. > > All the Best, > > Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From mikofski at berkeley.edu Sat Jul 13 03:47:32 2019 From: mikofski at berkeley.edu (Mark Mikofski) Date: Sat, 13 Jul 2019 00:47:32 -0700 Subject: [Numpy-discussion] defining a NumPy API standard? In-Reply-To: References: <631ce03d-0b01-c491-1b23-a1469a87d47e@gmail.com> <5b80b21127a82dade2539bf3ecdffb011b9695d5.camel@sipsolutions.net> Message-ID: This slide deck from Matthew Rocklin at SciPy 2019 might be relevant: https://matthewrocklin.com/slides/scipy-2019#/ On Tue, Jun 4, 2019 at 12:06 AM Ralf Gommers wrote: > > > On Mon, Jun 3, 2019 at 7:56 PM Sebastian Berg > wrote: > >> On Sun, 2019-06-02 at 08:42 +0200, Ralf Gommers wrote: >> > >> > >> >> > > > >> > > >> > > This sounds like a restructuring or factorization of the API, in >> > > order to make it smaller, and thus easier to learn and use. >> > > It may start with the docs, by paying more attention to the "core" >> > > or important functions and methods, and noting the deprecated, or >> > > not frequently used, or not important functions. This could also >> > > help the satellite projects, which use NumPy API as an example, and >> > > may also be influenced by them and their decisions. >> > > >> > >> > Indeed. It will help restructure our docs. Perhaps not the reference >> > guide (not sure yet), but definitely the user guide and other high- >> > level docs we (or third parties) may want to create. >> > >> >> Trying to follow the discussion, there seems to be various ideas? Do I >> understand it right that the original proposal was much like doing a >> list of: >> >> * np.ndarray.cumprod: low importance -> prefer np.multiply.accumulate >> * np.ravel_multi_index: low importance, but distinct feature >> > > Indeed. Certainly no more than that was my idea. > > >> Maybe with added groups such as "transpose-like" and "reshape-like" >> functions? >> This would be based on 1. "Experience" and 2. usage statistics. This >> seems mostly a task for 2-3 people to then throw out there for >> discussion. >> There will be some very difficult/impossible calls, since in the end >> Nathaniel is right, we do not quite know the question we want to >> answer. But for a huge part of the API it may not be problematic? >> > > Agreed, won't be problematic. > > >> >> Then there is an idea of providing better mixins (and tests). >> This could be made easier by the first idea, for prioritization. >> Although, the first idea is probably not really necessary to kick this >> off at all. The interesting parts to me seem likely how to best solve >> testing of the mixins and numpy-api-duplicators in general. >> >> Implementing a growing set of mixin seems likely fairly straight >> forwrad (although maybe much easier to approach if there is a list from >> the first project)? > > > Indeed. I think there's actually 3 levels here (at least): > 1. function name: high/low importance or some such simple classification > 2. function signature and behavior: is the behavior optimal, what would be > change, etc. > 3. making duck arrays and subclasses that rely on all those functions and > their behavior easier to implemement/use > > Mixins are a specific answer to (3). And it's unclear if they're the best > answer (could be, I don't know - please don't start a discussion on that > here). Either way, working on (3) will be helped by having a better sense > of (1) and (2). > > Also think about effort: (2) is at least an order of magnitude more work > than (1), and (3) likely even more work than (2). > > >> And, once we have a start, maybe we can rely on the >> array-like implementors to be the main developers (limiting us mostly >> to review). >> >> >> The last part would be probably for users and consumers of array-likes. >> This largely overlaps, but comes closer to the problem of "standard". >> If we have a list of functions that we tend to see as more or less >> important, it may be interesting for downstream projects to restrict >> themselves to simplify interoperability e.g. with dask. >> >> Maybe we do not have to draw a strict line though? How plausible would >> it be to set up a list (best auto-updating) saying nothing but: >> >> `np.concatenate` supported by: dask, jax, cupy >> > > That's probably not that hard, and I agree it would be quite useful. The > namespaces of each of those libraries is probably not the same, but with > dir() and some strings and lists you'll get a long way here I think. > > >> >> I am not sure if this is helpful, but it feels to me that the first >> part is what Ralf was thinking of? Just to kick of such a a "living >> document". > > > Indeed. > > I could maybe help with providing the second pair of eyes >> for a first iteration there, Ralf. > > > Awesome, thanks Sebastian. > > Cheers, > Ralf > > > The last list I would actually find >> interesting myself, but not sure how easy it would be to approach it? >> >> Best, >> >> Sebastian >> >> >> > Ralf >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at python.org >> > https://mail.python.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Sun Jul 14 10:46:47 2019 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 14 Jul 2019 09:46:47 -0500 Subject: [Numpy-discussion] Using a pyproject.toml file Message-ID: <17f87fc1-64ac-1e5c-4116-2f58552278c9@gmail.com> An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Sun Jul 14 12:05:45 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Sun, 14 Jul 2019 10:05:45 -0600 Subject: [Numpy-discussion] Using a pyproject.toml file In-Reply-To: <17f87fc1-64ac-1e5c-4116-2f58552278c9@gmail.com> References: <17f87fc1-64ac-1e5c-4116-2f58552278c9@gmail.com> Message-ID: For what it's worth, pyproject.toml seems to have been adopted without much friction in SciPy. In some CI runs in SciPy, we use something like "pip wheel --no-build-isolation -v -v -v" to disable the isolation in any case. I suppose NumPy is indeed even closer to the base of the ecosystem, but the ease of disabling may mitigate concerns related to switching from system-installed dependencies to an isolated environment for a naive "pip install" Tyler On Sun, 14 Jul 2019 at 08:47, Matti Picus wrote: > In PR #13908 I implemented the previously-discussed new method of creating > the release notes: writing separate fragments and then combining them at > release time via towncrier. Towncrier requires a PEP-508/PEP-517/PEP-518 > pyproject.toml file for configuration, and does not currently support a > command line option to specify a different location for this file. That > means that we now must ship a pyproject.toml file, which subtlely changes > the way "pip install ." builds NumPy: it does an "isolated build" > https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support > > > Questions: > > - Is the pain of adding a pyproject.toml worth it for using towncrier or > should we o for another release-note solution > > - Is the addition of pyproject.toml problematic enough that I should break > it out into a separate pull request for evaluation? > > > Matti > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Mon Jul 15 12:10:29 2019 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 15 Jul 2019 11:10:29 -0500 Subject: [Numpy-discussion] Kriti (our Outreachy intern) has published a blog post about being accepted to the program Message-ID: <1fb98e49-71c4-3f95-0a87-afbfa5b9745c@gmail.com> https://github.com/kritisingh1/numpy/wiki/Now-You-Know-It! Worth a read, see what applicants go through to get accepted. Feel free to re-publish it. There will be more coming as she progresses through the summer. From omrylevy at gmail.com Mon Jul 15 13:43:50 2019 From: omrylevy at gmail.com (Omry Levy) Date: Mon, 15 Jul 2019 20:43:50 +0300 Subject: [Numpy-discussion] converting 1 dimensional array to 2 dimensional array Message-ID: Hi All, I know how to reshape arrays, my problem is a little more complicated than that. I am looking for the most efficient way to do the following and an example will help: 1) I have a an array of bytes b = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] This bytes array represents a width of 2 and a height of 3. Each 2 bytes in that bytes array makes an unsigned 16 bit integer. 2) I want to get the following 2 dimensional array from the bytes array where each item in the two dimensional array is an unsigned 16 bit integer something like: [ [(1,2) , (3,4)], [(5,6) , (7,8)], [(9,10) , (11,12)] ] in the first row there are 2 elements each one is made of 2 bytes from the bytes array etc... meaning that (1, 2) = b[0] + b[1] << 8 (3, 4) = b[2] + b[3] << 8 ... ... What is the most efficient way to achieve that with numpy ? Thanks 0L [1] << 8 ,[2], [3, 4] -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni at fastmail.com Mon Jul 15 14:13:48 2019 From: jni at fastmail.com (Juan Nunez-Iglesias) Date: Mon, 15 Jul 2019 13:13:48 -0500 Subject: [Numpy-discussion] =?utf-8?q?converting_1_dimensional_array_to_2?= =?utf-8?q?_dimensional_array?= In-Reply-To: References: Message-ID: Hi Omry! You're looking for `.view()`: In [1]: import numpy as np In [2]: b = np.arange(1, 13).astype(np.uint8) In [4]: y = b.view(np.uint16).reshape((3, 2)) In [5]: y Out[5]: array([[ 513, 1027], [1541, 2055], [2569, 3083]], dtype=uint16) You can also change the endianness by replacing `np.uint16` with `'>u2'`. In [6]: z = b.view('>u2') In [7]: z Out[7]: array([ 258, 772, 1286, 1800, 2314, 2828], dtype=uint16) Hope this helps! Juan. On Mon, 15 Jul 2019, at 12:45 PM, Omry Levy wrote: > Hi All, > > I know how to reshape arrays, my problem is a little more complicated than that. > I am looking for the most efficient way to do the following and an example will help: > > 1) I have a an array of bytes b = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] > This bytes array represents a width of 2 and a height of 3. > Each 2 bytes in that bytes array makes an unsigned 16 bit integer. > > 2) I want to get the following 2 dimensional array from the bytes array where each item in the two dimensional array is an unsigned 16 bit integer > something like: > > [ > [(1,2) , (3,4)], > [(5,6) , (7,8)], > [(9,10) , (11,12)] > ] > > in the first row there are 2 elements each one is made of 2 bytes from the bytes array etc... > meaning that > (1, 2) = b[0] + b[1] << 8 > (3, 4) = b[2] + b[3] << 8 > ... > ... > > What is the most efficient way to achieve that with numpy ? > Thanks > 0L > > > > > > > [1] << 8 ,[2], [3, 4] > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Mon Jul 15 18:48:37 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Tue, 16 Jul 2019 00:48:37 +0200 Subject: [Numpy-discussion] Helping with website improvements Message-ID: Hi, With respect to this call for contributions: https://github.com/numpy/numpy/pull/13988/files I would like to help with improving the website of numpy (and maybe scipy as well). I have also applied for the Google Season of Docs 2019, and if accepted, I will be starting by the beginning of August. The improvements that I would like to make include: - Making the website responsive (so that it looks nice on small screens as well). There are responsive themes for sphinx and I may use one of them. - Improving the main page (or the landing page) so that it looks a bit more modern and attractive. - Reorganizing the structure of the information on the website, so that people from different backgrounds (students, professionals, etc.) can find more easily the relevant information that they are looking for. Including references to the external tutorials or courses about NumPy/SciPy. Other tentative improvements may be these: - Reorganize the docs so that each major release has its own version of docs. Major releases are those that may introduce new features, or in general, changes in the API (minor releases are the maintenance releases, which fix bugs, or make small changes, for example to improve the efficiency, but do not change the API). For example major releases may be v1.15, v1.16, v1.17 (however I am not sure about this). - Reorganize the docs so that the core API functionality is shown more prominently than the rest, and so that functions that may be deprecated in the future can be marked so (in order to discourage users from using them) and alternative solutions are suggested instead of them, etc. - Allow the users to add comments for each function or package. These may be usage examples for the benefit of other users, or pitfall alerts, or even bug reports. Reporting bugs on GitHub is better of course, but this may be a bit easier for the users. Since these are not incremental changes, if may be better if I work on a fork of the website repository, until they are finished and the new website is ready. Regards, Dashamir Hoxha -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jul 15 18:54:54 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 15 Jul 2019 15:54:54 -0700 Subject: [Numpy-discussion] defining a NumPy API standard? In-Reply-To: References: <631ce03d-0b01-c491-1b23-a1469a87d47e@gmail.com> <5b80b21127a82dade2539bf3ecdffb011b9695d5.camel@sipsolutions.net> Message-ID: On Sat, Jul 13, 2019 at 12:48 AM Mark Mikofski wrote: > This slide deck from Matthew Rocklin at SciPy 2019 might be relevant: > https://matthewrocklin.com/slides/scipy-2019#/ > That was a very nice talk indeed. It's also up on Youtube, worth watching: https://www.youtube.com/watch?v=Q0DsdiY-jiw I've also put up an 0.0.1 version of RNumPy "restricted NumPy" up on PyPI (mostly to reserve the name, but it's usable). The README and __init__.py docstring plus the package itself (https://github.com/Quansight-Labs/rnumpy) should give a better idea of the ideas we were discussing in this thread. Cheers, Ralf > On Tue, Jun 4, 2019 at 12:06 AM Ralf Gommers > wrote: > >> >> >> On Mon, Jun 3, 2019 at 7:56 PM Sebastian Berg >> wrote: >> >>> On Sun, 2019-06-02 at 08:42 +0200, Ralf Gommers wrote: >>> > >>> > >>> >>> > > > >>> > > >>> > > This sounds like a restructuring or factorization of the API, in >>> > > order to make it smaller, and thus easier to learn and use. >>> > > It may start with the docs, by paying more attention to the "core" >>> > > or important functions and methods, and noting the deprecated, or >>> > > not frequently used, or not important functions. This could also >>> > > help the satellite projects, which use NumPy API as an example, and >>> > > may also be influenced by them and their decisions. >>> > > >>> > >>> > Indeed. It will help restructure our docs. Perhaps not the reference >>> > guide (not sure yet), but definitely the user guide and other high- >>> > level docs we (or third parties) may want to create. >>> > >>> >>> Trying to follow the discussion, there seems to be various ideas? Do I >>> understand it right that the original proposal was much like doing a >>> list of: >>> >>> * np.ndarray.cumprod: low importance -> prefer np.multiply.accumulate >>> * np.ravel_multi_index: low importance, but distinct feature >>> >> >> Indeed. Certainly no more than that was my idea. >> >> >>> Maybe with added groups such as "transpose-like" and "reshape-like" >>> functions? >>> This would be based on 1. "Experience" and 2. usage statistics. This >>> seems mostly a task for 2-3 people to then throw out there for >>> discussion. >>> There will be some very difficult/impossible calls, since in the end >>> Nathaniel is right, we do not quite know the question we want to >>> answer. But for a huge part of the API it may not be problematic? >>> >> >> Agreed, won't be problematic. >> >> >>> >>> Then there is an idea of providing better mixins (and tests). >>> This could be made easier by the first idea, for prioritization. >>> Although, the first idea is probably not really necessary to kick this >>> off at all. The interesting parts to me seem likely how to best solve >>> testing of the mixins and numpy-api-duplicators in general. >>> >>> Implementing a growing set of mixin seems likely fairly straight >>> forwrad (although maybe much easier to approach if there is a list from >>> the first project)? >> >> >> Indeed. I think there's actually 3 levels here (at least): >> 1. function name: high/low importance or some such simple classification >> 2. function signature and behavior: is the behavior optimal, what would >> be change, etc. >> 3. making duck arrays and subclasses that rely on all those functions and >> their behavior easier to implemement/use >> >> Mixins are a specific answer to (3). And it's unclear if they're the best >> answer (could be, I don't know - please don't start a discussion on that >> here). Either way, working on (3) will be helped by having a better sense >> of (1) and (2). >> >> Also think about effort: (2) is at least an order of magnitude more work >> than (1), and (3) likely even more work than (2). >> >> >>> And, once we have a start, maybe we can rely on the >>> array-like implementors to be the main developers (limiting us mostly >>> to review). >>> >>> >>> The last part would be probably for users and consumers of array-likes. >>> This largely overlaps, but comes closer to the problem of "standard". >>> If we have a list of functions that we tend to see as more or less >>> important, it may be interesting for downstream projects to restrict >>> themselves to simplify interoperability e.g. with dask. >>> >>> Maybe we do not have to draw a strict line though? How plausible would >>> it be to set up a list (best auto-updating) saying nothing but: >>> >>> `np.concatenate` supported by: dask, jax, cupy >>> >> >> That's probably not that hard, and I agree it would be quite useful. The >> namespaces of each of those libraries is probably not the same, but with >> dir() and some strings and lists you'll get a long way here I think. >> >> >>> >>> I am not sure if this is helpful, but it feels to me that the first >>> part is what Ralf was thinking of? Just to kick of such a a "living >>> document". >> >> >> Indeed. >> >> I could maybe help with providing the second pair of eyes >>> for a first iteration there, Ralf. >> >> >> Awesome, thanks Sebastian. >> >> Cheers, >> Ralf >> >> >> The last list I would actually find >>> interesting myself, but not sure how easy it would be to approach it? >>> >>> Best, >>> >>> Sebastian >>> >>> >>> > Ralf >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at python.org >>> > https://mail.python.org/mailman/listinfo/numpy-discussion >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > -- > Mark Mikofski, PhD (2005) > *Fiat Lux* > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Tue Jul 16 03:22:46 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Tue, 16 Jul 2019 09:22:46 +0200 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 12:48 AM Dashamir Hoxha wrote: > Hi, > > With respect to this call for contributions: > https://github.com/numpy/numpy/pull/13988/files > I would like to help with improving the website of numpy (and maybe scipy > as well). > I have also applied for the Google Season of Docs 2019, and if accepted, I > will be starting by the beginning of August. > > The improvements that I would like to make include: > - Making the website responsive (so that it looks nice on small screens as > well). There are responsive themes for sphinx and I may use one of them. > - Improving the main page (or the landing page) so that it looks a bit > more modern and attractive. > - Reorganizing the structure of the information on the website, so that > people from different backgrounds (students, professionals, etc.) can find > more easily the relevant information that they are looking for. Including > references to the external tutorials or courses about NumPy/SciPy. > Has anybody tried Katacoda before: https://www.katacoda.com/courses/python/playground ? It can be used to develop interactive tutorials, which can also be embedded on a web page. I may also write a couple of beginners' tutorials (based on the existing tutorials), if this seems to be useful. > > Other tentative improvements may be these: > - Reorganize the docs so that each major release has its own version of > docs. Major releases are those that may introduce new features, or in > general, changes in the API (minor releases are the maintenance releases, > which fix bugs, or make small changes, for example to improve the > efficiency, but do not change the API). For example major releases may be > v1.15, v1.16, v1.17 (however I am not sure about this). > - Reorganize the docs so that the core API functionality is shown more > prominently than the rest, and so that functions that may be deprecated in > the future can be marked so (in order to discourage users from using them) > and alternative solutions are suggested instead of them, etc. > - Allow the users to add comments for each function or package. These may > be usage examples for the benefit of other users, or pitfall alerts, or > even bug reports. Reporting bugs on GitHub is better of course, but this > may be a bit easier for the users. > > Since these are not incremental changes, if may be better if I work on a > fork of the website repository, until they are finished and the new website > is ready. > > Regards, > Dashamir Hoxha > -------------- next part -------------- An HTML attachment was scrubbed... URL: From omrylevy at gmail.com Tue Jul 16 03:30:22 2019 From: omrylevy at gmail.com (Omry Levy) Date: Tue, 16 Jul 2019 10:30:22 +0300 Subject: [Numpy-discussion] converting a C bytes array to two dimensional numpy array Message-ID: Hi, I have a question, regarding conversion of C (unsigned char *) buffer to a two dimensional numpy array this is what i am doing: 1) I get a C network buffer of unsigned char * let's call it the source buffer the size of the source buffer is: W * H * 2 bytes 2) I am using PyByteArray_FromStringAndSize() to convert the source buffer (a C unsigned char *) to python bytes array. a = PyByteArray_FromStringAndSize(source buffer, W * H * 2) 3) i am using numpy.frombuffer to convert the python bytes array to a 1 dimensional numpy array of size W *H *2 bytes b = numpy.frombuffer(a, dtype = np.uint8) 4) i am creating a 2 dimensional numpy array from (3) when each element in that array is made of 2 bytes from the python bytes array c = b.view(np.uint16).reshape((H, W)) Is there a way to optimize this some how ? Can you suggest a faster and better solution ? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.k.sheppard at gmail.com Tue Jul 16 05:43:31 2019 From: kevin.k.sheppard at gmail.com (Kevin Sheppard) Date: Tue, 16 Jul 2019 10:43:31 +0100 Subject: [Numpy-discussion] Creating a subclass that never propagates Message-ID: I am trying to make a subclass that never propagates so that when interacted with another ndarray, or even itself so that the return type is always ndarray. Is this possible? I got pretty far with def __array_wrap__(self, out_arr, context=None): if out_arr.shape == (): return out_arr.item() # if ufunc output is scalar, return it else: out = super(ArrayLike, self).__array_wrap__(out_arr, context) # Never return ArrayLike if isinstance(out, ArrayLike): out = out.view(np.ndarray) return out Which works well for ufuncs. However, when I try other functions like `dot` I get my subclass type returned. If there a reasonable way to ensure that my subclass doesn't propagate? I think I would need some way to override the behavior when .view(MySubClass) is called. Thanks, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From garcia.marc at gmail.com Tue Jul 16 06:54:38 2019 From: garcia.marc at gmail.com (Marc Garcia) Date: Tue, 16 Jul 2019 11:54:38 +0100 Subject: [Numpy-discussion] Stricter numpydoc validation Message-ID: In pandas we've been, for more than a year now, enforcing a stricter numpydoc standard. And we've got a script to help with it, which validates things like capitalization and punctuation of paragraphs, the documented parameters (they must match the ones in the signature, have both a type and a description...), PEP-8 of the standards, and many more things, so all our docstrings are consistent. I saw that there is an issue with a discussion on having a more strict standard for numpydoc, I added a comment there on whether would make sense to move the pandas standard and validation code to numpydoc: https://github.com/numpy/numpydoc/issues/213#issuecomment-511760580 I think it's worth opening the discussion here too. Is there interest in the rest of the community on having a stricter standard, and move the pandas validation (with the required updates) to numpydoc? Of course we can discuss also the exact standard, but probably worth finding out first if a stricter numpydoc standard would make sense for everyone. You can find the documentation of our standard at: https://dev.pandas.io/development/contributing_docstring.html And the script that we use to validate, as well as the exact list of errors we detect in: https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py#L76 Cheers! -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Tue Jul 16 07:16:47 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Tue, 16 Jul 2019 07:16:47 -0400 Subject: [Numpy-discussion] Stricter numpydoc validation In-Reply-To: References: Message-ID: I have written and/or used something like this (though not nearly as complete!) in several other projects. It would be great to have one maintained source for such a checker, and numpydoc seems like a reasonable place for it. The one thing I worry about is maintenance burden, where numpydoc is already spread a little bit thin -- would any of the Pandas developers be willing to maintain it? Eric On Tue, Jul 16, 2019 at 6:56 AM Marc Garcia wrote: > In pandas we've been, for more than a year now, enforcing a stricter > numpydoc standard. And we've got a script to help with it, which validates > things like capitalization and punctuation of paragraphs, the documented > parameters (they must match the ones in the signature, have both a type and > a description...), PEP-8 of the standards, and many more things, so all our > docstrings are consistent. > > I saw that there is an issue with a discussion on having a more strict > standard for numpydoc, I added a comment there on whether would make sense > to move the pandas standard and validation code to numpydoc: > https://github.com/numpy/numpydoc/issues/213#issuecomment-511760580 > > I think it's worth opening the discussion here too. Is there interest in > the rest of the community on having a stricter standard, and move the > pandas validation (with the required updates) to numpydoc? Of course we can > discuss also the exact standard, but probably worth finding out first if a > stricter numpydoc standard would make sense for everyone. > > You can find the documentation of our standard at: > https://dev.pandas.io/development/contributing_docstring.html > > And the script that we use to validate, as well as the exact list of > errors we detect in: > https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py#L76 > > Cheers! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Jul 16 07:22:58 2019 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 16 Jul 2019 13:22:58 +0200 Subject: [Numpy-discussion] Stricter numpydoc validation In-Reply-To: References: Message-ID: <20190716112258.42q5szerivvprawd@phare.normalesup.org> > The one thing I worry about is maintenance burden, where numpydoc is already > spread a little bit thin -- would any of the Pandas developers be willing to > maintain it? Any reason that this is not done in sphinx, with the napoleon extension? https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html I would hope that this can increase the number of people likely to help maintaining. My 2 cents, Ga?l From charlesr.harris at gmail.com Tue Jul 16 08:57:31 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Jul 2019 06:57:31 -0600 Subject: [Numpy-discussion] Creating a subclass that never propagates In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 3:44 AM Kevin Sheppard wrote: > I am trying to make a subclass that never propagates so that when > interacted with another ndarray, or even itself so that the return type is > always ndarray. Is this possible? > > I got pretty far with > > def __array_wrap__(self, out_arr, context=None): > if out_arr.shape == (): > return out_arr.item() # if ufunc output is scalar, return it > else: > out = super(ArrayLike, self).__array_wrap__(out_arr, context) > # Never return ArrayLike > if isinstance(out, ArrayLike): > out = out.view(np.ndarray) > return out > > Which works well for ufuncs. However, when I try other functions like > `dot` I get my subclass type returned. > > If there a reasonable way to ensure that my subclass doesn't propagate? I > think I would need some way to override the behavior when .view(MySubClass) > is called. > > I think you will be able to do that with `__array_function__` in the upcoming 1.17 release. It is also in 1.16, but you need an environmental variable to activate it. Some documentation can be found at https://www.numpy.org/devdocs/reference/arrays.classes.html#special-attributes-and-methods . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jul 16 09:06:27 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Jul 2019 07:06:27 -0600 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: Hi Dashamir, On Mon, Jul 15, 2019 at 4:49 PM Dashamir Hoxha wrote: > Hi, > > With respect to this call for contributions: > https://github.com/numpy/numpy/pull/13988/files > I would like to help with improving the website of numpy (and maybe scipy > as well). > I have also applied for the Google Season of Docs 2019, and if accepted, I > will be starting by the beginning of August. > > The improvements that I would like to make include: > - Making the website responsive (so that it looks nice on small screens as > well). There are responsive themes for sphinx and I may use one of them. > - Improving the main page (or the landing page) so that it looks a bit > more modern and attractive. > - Reorganizing the structure of the information on the website, so that > people from different backgrounds (students, professionals, etc.) can find > more easily the relevant information that they are looking for. Including > references to the external tutorials or courses about NumPy/SciPy. > > Other tentative improvements may be these: > - Reorganize the docs so that each major release has its own version of > docs. Major releases are those that may introduce new features, or in > general, changes in the API (minor releases are the maintenance releases, > which fix bugs, or make small changes, for example to improve the > efficiency, but do not change the API). For example major releases may be > v1.15, v1.16, v1.17 (however I am not sure about this). > - Reorganize the docs so that the core API functionality is shown more > prominently than the rest, and so that functions that may be deprecated in > the future can be marked so (in order to discourage users from using them) > and alternative solutions are suggested instead of them, etc. > - Allow the users to add comments for each function or package. These may > be usage examples for the benefit of other users, or pitfall alerts, or > even bug reports. Reporting bugs on GitHub is better of course, but this > may be a bit easier for the users. > > Since these are not incremental changes, if may be better if I work on a > fork of the website repository, until they are finished and the new website > is ready. > > That sounds interesting and ambitious. I'll let others offer suggestions, we might want to host the site at a different provider which will offer easier access to developers. The repo is at at https://github.com/numpy/numpy.org. There is also a scipy.org website that could also use some work. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Tue Jul 16 08:48:02 2019 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 16 Jul 2019 14:48:02 +0200 Subject: [Numpy-discussion] converting a C bytes array to two dimensional numpy array In-Reply-To: References: Message-ID: On 16 Jul 2019, at 9:30 am, Omry Levy wrote: > > I have a question, regarding conversion of C (unsigned char *) buffer to a two dimensional numpy array > > this is what i am doing: > 1) I get a C network buffer of unsigned char * let's call it the source buffer > the size of the source buffer is: > W * H * 2 bytes > > 2) I am using PyByteArray_FromStringAndSize() to convert the source buffer (a C unsigned char *) to python bytes array. > a = PyByteArray_FromStringAndSize(source buffer, W * H * 2) > > 3) i am using numpy.frombuffer to convert the python bytes array to a 1 dimensional numpy array of size W *H *2 bytes > b = numpy.frombuffer(a, dtype = np.uint8) > > 4) i am creating a 2 dimensional numpy array from (3) when each element in that array is made of 2 bytes from the python bytes array > c = b.view(np.uint16).reshape((H, W)) > > Is there a way to optimize this some how ? > Can you suggest a faster and better solution ? > The PyByteArray conversion seems unnecessary - if you can access your input as a buffer, calling np.frombuffer on it directly with the correct dtype should work just as well, and you can reshape it on the fly: c = np.frombuffer(source_buffer, dtype=np.uint16, [count=W*H]).reshape((H, W)) The optional ?count? argument would only be required if you cannot simply read the buffer to its end. HTH, Derek From ralf.gommers at gmail.com Tue Jul 16 11:06:40 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jul 2019 08:06:40 -0700 Subject: [Numpy-discussion] Creating a subclass that never propagates In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 5:58 AM Charles R Harris wrote: > > > On Tue, Jul 16, 2019 at 3:44 AM Kevin Sheppard > wrote: > >> I am trying to make a subclass that never propagates so that when >> interacted with another ndarray, or even itself so that the return type is >> always ndarray. Is this possible? >> >> I got pretty far with >> >> def __array_wrap__(self, out_arr, context=None): >> if out_arr.shape == (): >> return out_arr.item() # if ufunc output is scalar, return it >> else: >> out = super(ArrayLike, self).__array_wrap__(out_arr, context) >> # Never return ArrayLike >> if isinstance(out, ArrayLike): >> out = out.view(np.ndarray) >> return out >> >> Which works well for ufuncs. However, when I try other functions like >> `dot` I get my subclass type returned. >> >> If there a reasonable way to ensure that my subclass doesn't propagate? I >> think I would need some way to override the behavior when .view(MySubClass) >> is called. >> > I think you need to implement __array_finalize__ for this (see e.g. https://docs.scipy.org/doc/numpy-1.13.0/user/basics.subclassing.html#implications-for-subclassing ) >> > I think you will be able to do that with `__array_function__` in the > upcoming 1.17 release. It is also in 1.16, but you need an environmental > variable to activate it. Some documentation can be found at > https://www.numpy.org/devdocs/reference/arrays.classes.html#special-attributes-and-methods > . > That's kind of an orthogonal thing: __array_function__ is for providing your own implementation of functions, which you don't necessarily want to do if you're just building a small subclass. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 16 11:12:13 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jul 2019 08:12:13 -0700 Subject: [Numpy-discussion] Stricter numpydoc validation In-Reply-To: <20190716112258.42q5szerivvprawd@phare.normalesup.org> References: <20190716112258.42q5szerivvprawd@phare.normalesup.org> Message-ID: On Tue, Jul 16, 2019 at 4:23 AM Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > > The one thing I worry about is maintenance burden, where numpydoc is > already > > spread a little bit thin -- would any of the Pandas developers be > willing to > > maintain it? > > Any reason that this is not done in sphinx, with the napoleon extension? > https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html > > I would hope that this can increase the number of people likely to help > maintaining. > Just history. Numpydoc came first, most projects rely on it. Napoleon came way later and then did its own numpy docstring support rather than contribute to numpydoc or propose a merge. If someone wants to figure out how compatible these two implementations are and whether they can be merged, that would be really welcome. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jul 16 11:15:15 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 16 Jul 2019 09:15:15 -0600 Subject: [Numpy-discussion] NumPy 1.17.0rc2 released In-Reply-To: References: Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0rc2. The 1.17 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b2 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: - A new extensible random module along with four selectable random number generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. - NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. - New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. - Overriding numpy functions is now possible by default Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 146 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Voelker + - Abdur Rehman + - Abdur-Rahmaan Janhangeer + - Abhinav Sagar + - Adam J. Stewart + - Adam Orr + - Albert Thomas + - Alex Watt + - Alexander Blinne + - Alexander Shadchin - Allan Haldane - Ander Ustarroz + - Andras Deak - Andrea Pattori + - Andreas Schwab - Andrew Naguib + - Andy Scholand + - Ankit Shukla + - Anthony Sottile - Antoine Pitrou - Antony Lee - Arcesio Castaneda Medina + - Assem + - Bernardt Duvenhage + - Bharat Raghunathan + - Bharat123rox + - Bran + - Bruce Merry + - Charles Harris - Chirag Nighut + - Christoph Gohlke - Christopher Whelan + - Chuanzhu Xu + - Dan Allan + - Daniel Hrisca - Daniel Lawrence + - Debsankha Manik + - Dennis Zollo + - Dieter Werthm?ller + - Dominic Jack + - EelcoPeacs + - Eric Larson - Eric Wieser - Fabrice Fontaine + - Gary Gurlaskie + - Gregory Lee + - Gregory R. Lee - Guillaume Horel + - Hameer Abbasi - Haoyu Sun + - He Jia + - Hunter Damron + - Ian Sanders + - Ilja + - Isaac Virshup + - Isaiah Norton + - Jaime Fernandez - Jakub Wilk - Jan S. (Milania1) + - Jarrod Millman - Javier Dehesa + - Jeremy Lay + - Jim Turner + - Jingbei Li + - Joachim Hereth + - Johannes Hampp + - John Belmonte + - John Kirkham - John Law + - Jonas Jensen - Joseph Fox-Rabinovitz - Joseph Martinot-Lagarde - Josh Wilson - Juan Luis Cano Rodr?guez - Julian Taylor - J?r?mie du Boisberranger + - Kai Striega + - Katharine Hyatt + - Kevin Sheppard - Kexuan Sun - Kiko Correoso + - Kriti Singh + - Lars Grueter + - Maksim Shabunin + - Manvi07 + - Mark Harfouche - Marten van Kerkwijk - Martin Reinecke + - Matthew Brett - Matthias Bussonnier - Matti Picus - Michel Fruchart + - Mike Lui + - Mike Taves + - Min ho Kim + - Mircea Akos Bruma - Nick Minkyu Lee - Nick Papior - Nick R. Papior + - Nicola Soranzo + - Nimish Telang + - OBATA Akio + - Oleksandr Pavlyk - Ori Broda + - Paul Ivanov - Pauli Virtanen - Peter Andreas Entschev + - Peter Bell + - Pierre de Buyl - Piyush Jaipuriayar + - Prithvi MK + - Raghuveer Devulapalli + - Ralf Gommers - Richard Harris + - Rishabh Chakrabarti + - Riya Sharma + - Robert Kern - Roman Yurchak - Ryan Levy + - Sebastian Berg - Sergei Lebedev + - Shekhar Prasad Rajak + - Stefan van der Walt - Stephan Hoyer - Steve Stagg + - SuryaChand P + - S?ren Rasmussen + - Thibault Hallouin + - Thomas A Caswell - Tobias Uelwer + - Tony LaTorre + - Toshiki Kataoka - Tyler Moncur + - Tyler Reddy - Valentin Haenel - Vrinda Narayan + - Warren Weckesser - Weitang Li - Wojtek Ruszczewski - Yu Feng - Yu Kobayashi + - Yury Kirienko + - @aashuli + - @luzpaz - @parul + - @spacescientist + Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jul 16 12:22:41 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 16 Jul 2019 09:22:41 -0700 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, July 17 Message-ID: <6731d1a81cf80eca338e460007da14306c171df1.camel@sipsolutions.net> Hi all, There will be a NumPy Community meeting Wednesday July 17 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg Best wishes Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Tue Jul 16 12:57:35 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 16 Jul 2019 09:57:35 -0700 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: <3bd9179ee1015fe6df608901113a14287f130305.camel@sipsolutions.net> On Tue, 2019-07-16 at 07:06 -0600, Charles R Harris wrote: > Hi Dashamir, > > On Mon, Jul 15, 2019 at 4:49 PM Dashamir Hoxha > wrote: > > Hi, > > > > With respect to this call for contributions: > > https://github.com/numpy/numpy/pull/13988/files > > I would like to help with improving the website of numpy (and maybe > > scipy as well). > > I have also applied for the Google Season of Docs 2019, and if > > accepted, I will be starting by the beginning of August. > > > > The improvements that I would like to make include: > > - Making the website responsive (so that it looks nice on small > > screens as well). There are responsive themes for sphinx and I may > > use one of them. > > - Improving the main page (or the landing page) so that it looks a > > bit more modern and attractive. > > - Reorganizing the structure of the information on the website, so > > that people from different backgrounds (students, professionals, > > etc.) can find more easily the relevant information that they are > > looking for. Including references to the external tutorials or > > courses about NumPy/SciPy. > > > > Other tentative improvements may be these: > > - Reorganize the docs so that each major release has its own > > version of docs. Major releases are those that may introduce new > > features, or in general, changes in the API (minor releases are the > > maintenance releases, which fix bugs, or make small changes, for > > example to improve the efficiency, but do not change the API). For > > example major releases may be v1.15, v1.16, v1.17 (however I am not > > sure about this). > > - Reorganize the docs so that the core API functionality is shown > > more prominently than the rest, and so that functions that may be > > deprecated in the future can be marked so (in order to discourage > > users from using them) and alternative solutions are suggested > > instead of them, etc. > > - Allow the users to add comments for each function or package. > > These may be usage examples for the benefit of other users, or > > pitfall alerts, or even bug reports. Reporting bugs on GitHub is > > better of course, but this may be a bit easier for the users. > > > > Since these are not incremental changes, if may be better if I work > > on a fork of the website repository, until they are finished and > > the new website is ready. > > > > > > That sounds interesting and ambitious. I'll let others offer > suggestions, we might want to host the site at a different provider > which will offer easier access to developers. The repo is at at > https://github.com/numpy/numpy.org. There is also a scipy.org website > that could also use some work. > Indeed, great to see interest, it sounds very nice! I believe you are already in contact with Ralf directly (just in case someone wonders if there might be little immediate followup on the mailing list). Best, Sebastian > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From pav at iki.fi Tue Jul 16 13:59:38 2019 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 16 Jul 2019 20:59:38 +0300 Subject: [Numpy-discussion] Using a pyproject.toml file In-Reply-To: <17f87fc1-64ac-1e5c-4116-2f58552278c9@gmail.com> References: <17f87fc1-64ac-1e5c-4116-2f58552278c9@gmail.com> Message-ID: <766ed754dc981186ea4261a2544731c1bbed4a56.camel@iki.fi> su, 2019-07-14 kello 09:46 -0500, Matti Picus kirjoitti: [clip] > Questions: > - Is the pain of adding a pyproject.toml worth it for using towncrier > or should we o for another release-note solution > - Is the addition of pyproject.toml problematic enough that I should > break it out into a separate pull request for evaluation? This decision is probably better to make separately, without towncrier as a motivation, since there is some end-user impact (would e.g. have to be mentioned in release notes). You probably also need to disable build isolation in numpy-wheels, to manage the dependency versions manually. It probably also should be mentioned in install instructions somewhere. The main thing to watch out with introducing pyproject.toml is that build-time dependencies for non-pure-Python packages work properly only with pip 19 (unless they have wheels), which is fairly recent. Currently, numpy doesn't appear to have any build deps, so this probably does not matter. However, if you introduce Cython as a build- dep, it would solve the problem of installing numpy from without Cython pre-installed, and allows you to get rid of cythonize.py and hacks associated with setup_requires. Scipy 1.3.0 has pyproject.toml, and so far it seems the number of problems associated with it on the issue tracker is very small. Since it's becoming more common, downstream is forced to adjust infrastructure anyway, so Numpy hopping onto the wagon probably won't be a big impact for them. Pauli From p.j.a.cock at googlemail.com Tue Jul 16 14:02:28 2019 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 16 Jul 2019 19:02:28 +0100 Subject: [Numpy-discussion] Stricter numpydoc validation In-Reply-To: References: <20190716112258.42q5szerivvprawd@phare.normalesup.org> Message-ID: I?d find this sort of (stricter) numpydoc validation tool very useful, especially if the different codes can be selectively enforced while bringing a large code base into compliance (as pandas seems to have used this). A stand alone tool would be fine, a flake8 plug-in perhaps even better - see also my https://github.com/peterjc/flake8-rst-docstrings for doing basic RST validation of docstrings (but not their contents as we are discussing here). The nice thing with writing a flake8 plugin is that handles include/excluding of codes and file names for you. Regards, Peter On Tue, 16 Jul 2019 at 16:13, Ralf Gommers wrote: > > > On Tue, Jul 16, 2019 at 4:23 AM Gael Varoquaux < > gael.varoquaux at normalesup.org> wrote: > >> > The one thing I worry about is maintenance burden, where numpydoc is >> already >> > spread a little bit thin -- would any of the Pandas developers be >> willing to >> > maintain it? >> >> Any reason that this is not done in sphinx, with the napoleon extension? >> https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html >> >> I would hope that this can increase the number of people likely to help >> maintaining. >> > > Just history. Numpydoc came first, most projects rely on it. Napoleon came > way later and then did its own numpy docstring support rather than > contribute to numpydoc or propose a merge. > > If someone wants to figure out how compatible these two implementations > are and whether they can be merged, that would be really welcome. > > Cheers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Jul 16 23:59:15 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jul 2019 20:59:15 -0700 Subject: [Numpy-discussion] numpy.org website redesign proposal Message-ID: Hi all, I just opened https://github.com/numpy/numpy/pull/14032, which contains a NEP for a redesign of the NumPy website (just the top-level site, not the docs). The part that most warrants discussion is probably the translation part. Additions to the NEP from anyone who has useful experiences with managing translations like that would be quite welcome. There's a couple of motivations for writing this as a NEP: 1. Ensure that we're all on the same page before doing quite a bit of work 2. Empowering people who have offered help with the website to go ahead and make changes. We got a few offers of help from people who are new to the project, which is awesome. I want to make sure they feel comfortable making some decisions. 3. Pushing forward with the website redesign now prepares the ground for GSoD, where we have tech writers and people with marketing skills that I hope will generate much better content than we have now. Integrating that content is much easier in a well-designed site than into the current Sphinx site. I also will start inviting our new volunteers to the NumPy web team, so they have the permissions needed to work on the website repo. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 17 00:06:52 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jul 2019 21:06:52 -0700 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: <3bd9179ee1015fe6df608901113a14287f130305.camel@sipsolutions.net> References: <3bd9179ee1015fe6df608901113a14287f130305.camel@sipsolutions.net> Message-ID: On Tue, Jul 16, 2019 at 9:57 AM Sebastian Berg wrote: > On Tue, 2019-07-16 at 07:06 -0600, Charles R Harris wrote: > > Hi Dashamir, > > > > On Mon, Jul 15, 2019 at 4:49 PM Dashamir Hoxha > > wrote: > > > Hi, > > > > > > With respect to this call for contributions: > > > https://github.com/numpy/numpy/pull/13988/files > > > I would like to help with improving the website of numpy (and maybe > > > scipy as well). > Thanks for offering to help Dashamir! > > I have also applied for the Google Season of Docs 2019, and if > > > accepted, I will be starting by the beginning of August. > > > > > > The improvements that I would like to make include: > > > - Making the website responsive (so that it looks nice on small > > > screens as well). There are responsive themes for sphinx and I may > > > use one of them. > > > - Improving the main page (or the landing page) so that it looks a > > > bit more modern and attractive. > > > - Reorganizing the structure of the information on the website, so > > > that people from different backgrounds (students, professionals, > > > etc.) can find more easily the relevant information that they are > > > looking for. Including references to the external tutorials or > > > courses about NumPy/SciPy. > This all sounds good. I just sent another email about the numpy.org redesign, please feel free to jump in on the proposal ( https://github.com/numpy/numpy/pull/14032). > > > > > > Other tentative improvements may be these: > > > - Reorganize the docs so that each major release has its own > > > version of docs. Major releases are those that may introduce new > > > features, or in general, changes in the API (minor releases are the > > > maintenance releases, which fix bugs, or make small changes, for > > > example to improve the efficiency, but do not change the API). For > > > example major releases may be v1.15, v1.16, v1.17 (however I am not > > > sure about this). > Note that the docs are kept separate from the numpy.org site, and when we move numpy.org away from Sphinx that separation will be clearer. Matti is already working on this multi-version reshuffle in https://github.com/numpy/numpy/pull/13886. > > > - Reorganize the docs so that the core API functionality is shown > > > more prominently than the rest, and so that functions that may be > > > deprecated in the future can be marked so (in order to discourage > > > users from using them) and alternative solutions are suggested > > > instead of them, etc. > This is a good idea. That sounds more like a GSoD topic - it's quite some work to do, and to agree on the choices of what's important. > > - Allow the users to add comments for each function or package. > > > These may be usage examples for the benefit of other users, or > > > pitfall alerts, or even bug reports. Reporting bugs on GitHub is > > > better of course, but this may be a bit easier for the users. > This one I'm not so sure about. Could result in a lot more work for us...... > > > > > Since these are not incremental changes, if may be better if I work > > > on a fork of the website repository, until they are finished and > > > the new website is ready. > Yes, would be good to start in a clean repo. I'd like to get the NEP accepted first, and then put together a bit of a plan so multiple people can work together on this. Cheers, Ralf > > > > > > > > > That sounds interesting and ambitious. I'll let others offer > > suggestions, we might want to host the site at a different provider > > which will offer easier access to developers. The repo is at at > > https://github.com/numpy/numpy.org. There is also a scipy.org website > > that could also use some work. > > > > > Indeed, great to see interest, it sounds very nice! I believe you are > already in contact with Ralf directly (just in case someone wonders if > there might be little immediate followup on the mailing list). > > Best, > > Sebastian > > > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 17 00:10:25 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 16 Jul 2019 21:10:25 -0700 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: On Tue, Jul 16, 2019 at 12:23 AM Dashamir Hoxha wrote: > On Tue, Jul 16, 2019 at 12:48 AM Dashamir Hoxha > wrote: > >> Hi, >> >> With respect to this call for contributions: >> https://github.com/numpy/numpy/pull/13988/files >> I would like to help with improving the website of numpy (and maybe scipy >> as well). >> I have also applied for the Google Season of Docs 2019, and if accepted, >> I will be starting by the beginning of August. >> >> The improvements that I would like to make include: >> - Making the website responsive (so that it looks nice on small screens >> as well). There are responsive themes for sphinx and I may use one of them. >> - Improving the main page (or the landing page) so that it looks a bit >> more modern and attractive. >> - Reorganizing the structure of the information on the website, so that >> people from different backgrounds (students, professionals, etc.) can find >> more easily the relevant information that they are looking for. Including >> references to the external tutorials or courses about NumPy/SciPy. >> > > Has anybody tried Katacoda before: > https://www.katacoda.com/courses/python/playground ? > I haven't heard of it before. Looks cool. Seems to run on Docker, so would require some infrastructure. If it's something we could get from a static site, then it may be worth considering. Could you summarize what it would require? Cheers, Ralf It can be used to develop interactive tutorials, which can also be embedded > on a web page. > I may also write a couple of beginners' tutorials (based on the existing > tutorials), if this seems to be useful. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Wed Jul 17 08:28:40 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Wed, 17 Jul 2019 14:28:40 +0200 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 6:11 AM Ralf Gommers wrote: > >> Has anybody tried Katacoda before: >> https://www.katacoda.com/courses/python/playground ? >> > > I haven't heard of it before. Looks cool. Seems to run on Docker, so would > require some infrastructure. If it's something we could get from a static > site, then it may be worth considering. Could you summarize what it would > require? > The infrastructure is supplied by Katacoda, we don't need to run or maintain any server (although I don't think this would be a big problem). And it is free. The only limitation is that the duration of the training session for a student is 1 hour (and then he can reload the browser and start a fresh session). If enterprises or training companies would like to lift this limitation, they have to contact them. Building an interactive tutorial is very easy and everybody can learn it in a short time. The code of the tutorial is just a bunch of json and markdown files (for the configuration of the environment and for the steps of the tutorial). It can be saved on your preferred git repository (GitHub, GitLab, Bitbacket, etc.) and katacoda pulls it from there and builds the tutorial environment (docker image). Here is a minimal (hello world) example: https://github.com/dashohoxha/katacoda-scenarios/tree/master/hello-world You also need to set a webhook on the repository, so that whenever you make some changes (commits) katacoda is notified to refresh the content of the tutorial. Regards, Dashamir -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Wed Jul 17 09:43:22 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Wed, 17 Jul 2019 15:43:22 +0200 Subject: [Numpy-discussion] numpy.org website redesign proposal In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 6:00 AM Ralf Gommers wrote: > Hi all, > > I just opened https://github.com/numpy/numpy/pull/14032, which contains a > NEP for a redesign of the NumPy website (just the top-level site, not the > docs). The part that most warrants discussion is probably the translation > part. Additions to the NEP from anyone who has useful experiences with > managing translations like that would be quite welcome. > > There's a couple of motivations for writing this as a NEP: > 1. Ensure that we're all on the same page before doing quite a bit of work > 2. Empowering people who have offered help with the website to go ahead > and make changes. We got a few offers of help from people who are new to > the project, which is awesome. I want to make sure they feel comfortable > making some decisions. > 3. Pushing forward with the website redesign now prepares the ground for > GSoD, where we have tech writers and people with marketing skills that I > hope will generate much better content than we have now. Integrating that > content is much easier in a well-designed site than into the current Sphinx > site. > > I also will start inviting our new volunteers to the NumPy web team, so > they have the permissions needed to work on the website repo. > First of all, choosing a few existing sites as examples to follow is the best thing to do. I like all the three sites selected by Ralf, especially that of Jupiter and Julia. Having such examples makes our job much easier (and also reduces our choices and discussions about possible choices). Indeed, if we want to make our site similar to Jupiter, we just fork it ( https://github.com/jupyter/jupyter.github.io) and customize its content. If we want to make it similar to Julia, we just fork it: https://github.com/JuliaLang/www.julialang.org Both of them are in Jekyll and markdown, so this is the only choice (unless we want to make our site similar to that of pandas, which is built with sphinx). We admit that none of us is good as a web designer, so following the example of a nice website is the best way to go. I don't understand the need/requirement for translation or i18n. None of the sites that we are considering as examples have it, and I have not seen any other sites that have it. Maybe I am missing something. Can someone show an example of a website with translation? By the way, I know that the docs are not the topic of this NEP, but I think that we should follow the good examples for the documentation of the code as well. For example Jupiter and Julia: - https://jupyter.org/documentation - https://docs.julialang.org/en/v1/ Regards, Dashamir -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 17 17:43:53 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 17 Jul 2019 16:43:53 -0500 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 7:29 AM Dashamir Hoxha wrote: > > > On Wed, Jul 17, 2019 at 6:11 AM Ralf Gommers > wrote: > >> >>> Has anybody tried Katacoda before: >>> https://www.katacoda.com/courses/python/playground ? >>> >> >> I haven't heard of it before. Looks cool. Seems to run on Docker, so >> would require some infrastructure. If it's something we could get from a >> static site, then it may be worth considering. Could you summarize what it >> would require? >> > > The infrastructure is supplied by Katacoda, we don't need to run or > maintain any server (although I don't think this would be a big problem). > And it is free. The only limitation is that the duration of the training > session for a student is 1 hour (and then he can reload the browser and > start a fresh session). If enterprises or training companies would like to > lift this limitation, they have to contact them. > > Building an interactive tutorial is very easy and everybody can learn it > in a short time. The code of the tutorial is just a bunch of json and > markdown files (for the configuration of the environment and for the steps > of the tutorial). It can be saved on your preferred git repository (GitHub, > GitLab, Bitbacket, etc.) and katacoda pulls it from there and builds the > tutorial environment (docker image). Here is a minimal (hello world) > example: > https://github.com/dashohoxha/katacoda-scenarios/tree/master/hello-world You > also need to set a webhook on the repository, so that whenever you make > some changes (commits) katacoda is notified to refresh the content of the > tutorial. > That sounds good to me. I believe SymPy is also pretty happy with their interactive terminal embedded in their website. Could you create a new issue on https://github.com/numpy/numpy.org/issues? It would be good to start keeping track of website improvement ideas - ideas in mailing list threads tend to get lost eventually. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 17 18:40:25 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 17 Jul 2019 17:40:25 -0500 Subject: [Numpy-discussion] numpy.org website redesign proposal In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 8:43 AM Dashamir Hoxha wrote: > > On Wed, Jul 17, 2019 at 6:00 AM Ralf Gommers > wrote: > >> Hi all, >> >> I just opened https://github.com/numpy/numpy/pull/14032, which contains >> a NEP for a redesign of the NumPy website (just the top-level site, not the >> docs). The part that most warrants discussion is probably the translation >> part. Additions to the NEP from anyone who has useful experiences with >> managing translations like that would be quite welcome. >> >> There's a couple of motivations for writing this as a NEP: >> 1. Ensure that we're all on the same page before doing quite a bit of work >> 2. Empowering people who have offered help with the website to go ahead >> and make changes. We got a few offers of help from people who are new to >> the project, which is awesome. I want to make sure they feel comfortable >> making some decisions. >> 3. Pushing forward with the website redesign now prepares the ground for >> GSoD, where we have tech writers and people with marketing skills that I >> hope will generate much better content than we have now. Integrating that >> content is much easier in a well-designed site than into the current Sphinx >> site. >> >> I also will start inviting our new volunteers to the NumPy web team, so >> they have the permissions needed to work on the website repo. >> > > First of all, choosing a few existing sites as examples to follow is the > best thing to do. I like all the three sites selected by Ralf, especially > that of Jupiter and Julia. Having such examples makes our job much easier > (and also reduces our choices and discussions about possible choices). > > Indeed, if we want to make our site similar to Jupiter, we just fork it ( > https://github.com/jupyter/jupyter.github.io) and customize its content. > If we want to make it similar to Julia, we just fork it: > https://github.com/JuliaLang/www.julialang.org Both of them are in Jekyll > and markdown, so this is the only choice (unless we want to make our site > similar to that of pandas, which is built with sphinx). We admit that none > of us is good as a web designer, so following the example of a nice website > is the best way to go. > That could be an idea. Not sure it's much less work (we'd have to change all stying, graphics and text anyway), but it may be. I think > I don't understand the need/requirement for translation or i18n. > The need is simple: for many of our users (the majority I think), English is not the first language. Many people are prevented from learning about NumPy effectively. A few pages in their own language goes a long way to helping them. Also, we're likely missing out on contributions due to this. I've had a number of people tell me that this is a significant barrier. None of the sites that we are considering as examples have it, and I have > not seen any other sites that have it. Maybe I am missing something. Can > someone show an example of a website with translation? > AstroPy is wanting to do this too IIRC. Most widely used commercial software/platforms have this (because they can measure the impact of translations in sales - it matters). Examples: https://hortonworks.com/, https://aws.amazon.com/. An example from an open source project: https://www.tensorflow.org/guide/extend/bindings ("language button" at the top, changes to Chinese). I'd like to try it and see if we can make it work with low maintenance overhead. If we can't figure out the workflow to make the overhead low enough, then that's a no-go. But I think with today's tools it's possible. Cheers, Ralf > By the way, I know that the docs are not the topic of this NEP, but I > think that we should follow the good examples for the documentation of the > code as well. For example Jupiter and Julia: > - https://jupyter.org/documentation > - https://docs.julialang.org/en/v1/ > > Regards, > Dashamir > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 17 20:12:00 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 17 Jul 2019 19:12:00 -0500 Subject: [Numpy-discussion] Season of Docs applications Message-ID: Hi GSoD applicants, To start with, I want to thank all of you who applied! There's a lot of interest in helping improve NumPy's documentation and web presence, which is awesome to see. During the application phase, we have interacted with some of you already, and I will reach out to others over the next few days. I'm just writing this email to say thank you and to confirm that yes we did receive your application. Due to the volume of applications, it will be difficult to reach out personally to all of you (I'll do my best though!), hence this email. Really looking forward to what will happen next! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From dashohoxha at gmail.com Thu Jul 18 05:17:41 2019 From: dashohoxha at gmail.com (Dashamir Hoxha) Date: Thu, 18 Jul 2019 11:17:41 +0200 Subject: [Numpy-discussion] Helping with website improvements In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 11:45 PM Ralf Gommers wrote: > > That sounds good to me. I believe SymPy is also pretty happy with their > interactive terminal embedded in their website. > > Could you create a new issue on https://github.com/numpy/numpy.org/issues? > It would be good to start keeping track of website improvement ideas - > ideas in mailing list threads tend to get lost eventually. > I have created it: https://github.com/numpy/numpy.org/issues/26 As a side note, I think that it would be nice if documentation issues are kept on this repo as well (or on their on repo), instead of discussing them on numpy/numpy. This would clearly distinguish them from the rest of the issues (bugs, features, enhancements, etc.). For example I am interested on the website and documentation issues, but I have to subscribe (watch) to numpy/numpy. As a result I get more than 100 notification messages a day, 90% of which I have to delete because I am not interested in them (actually I don't understand what is being discussed there). This is like a kind of innocent spam. A better classification of issues to different repositories may help to reduce this. Maybe the same should be done for scipy too. I have created another issue for discussing this: https://github.com/numpy/numpy.org/issues/27 Regards, Dashamir -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Jul 19 20:12:49 2019 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 19 Jul 2019 17:12:49 -0700 Subject: [Numpy-discussion] converting a C bytes array to two dimensional numpy array In-Reply-To: References: Message-ID: You can also directly build a numpy array from a pointer with the numpy API. And I recommend Cython as an interface to make these things easy. This does mean you?d need to have the numpy lib at build time, .which may be a downside. -CHB Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception On Jul 16, 2019, at 5:48 AM, Derek Homeier < derek at astro.physik.uni-goettingen.de> wrote: On 16 Jul 2019, at 9:30 am, Omry Levy wrote: I have a question, regarding conversion of C (unsigned char *) buffer to a two dimensional numpy array this is what i am doing: 1) I get a C network buffer of unsigned char * let's call it the source buffer the size of the source buffer is: W * H * 2 bytes 2) I am using PyByteArray_FromStringAndSize() to convert the source buffer (a C unsigned char *) to python bytes array. a = PyByteArray_FromStringAndSize(source buffer, W * H * 2) 3) i am using numpy.frombuffer to convert the python bytes array to a 1 dimensional numpy array of size W *H *2 bytes b = numpy.frombuffer(a, dtype = np.uint8) 4) i am creating a 2 dimensional numpy array from (3) when each element in that array is made of 2 bytes from the python bytes array c = b.view(np.uint16).reshape((H, W)) Is there a way to optimize this some how ? Can you suggest a faster and better solution ? The PyByteArray conversion seems unnecessary - if you can access your input as a buffer, calling np.frombuffer on it directly with the correct dtype should work just as well, and you can reshape it on the fly: c = np.frombuffer(source_buffer, dtype=np.uint16, [count=W*H]).reshape((H, W)) The optional ?count? argument would only be required if you cannot simply read the buffer to its end. HTH, Derek _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at python.org https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jul 19 21:45:40 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 19 Jul 2019 20:45:40 -0500 Subject: [Numpy-discussion] creating a .github repo Message-ID: Hi all, For the GitHub "community health" files like CoC, CONTRIBUTING, FUNDING, SECURITY, etc., there's now the option to put them in a .github repository so they show up for all other repos in the organization. See https://help.github.com/en/articles/creating-a-default-community-health-file-for-your-organization I propose to do this in the next couple of days. If anyone sees an issue with this, please reply. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Tue Jul 23 10:28:03 2019 From: ram at rachum.com (Ram Rachum) Date: Tue, 23 Jul 2019 17:28:03 +0300 Subject: [Numpy-discussion] Creating a sine wave with exponential decay Message-ID: Hi everyone! Total Numpy newbie here. I'd like to create an array with a million numbers, that has a sine wave with exponential decay on the amplitude. In other words, I want the value of each cell n to be sin(n) * 2 ** (-n * factor). What would be the most efficient way to do that? Someone suggested I do something like this: y = np.sin(x) * np.exp(newfactor * x) But this would create 2 arrays, wouldn't it? Isn't that wasteful? Does Numpy provide an efficient way of doing that without creating a redundant array? Thanks for your help, Ram Rachum. -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Tue Jul 23 11:37:01 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Tue, 23 Jul 2019 15:37:01 +0000 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: Message-ID: Hi Ram, No, NumPy doesn?t have a way. And it newer versions, it probably won?t create two arrays if all the dtypes match, it?ll do some magic to re use the existing ones, although it will use multiple loops instead of just one. You might want to look into NumExpr or Numba if you want an efficient implementation. Get Outlook for iOS ________________________________ From: NumPy-Discussion on behalf of Ram Rachum Sent: Tuesday, July 23, 2019 7:29 pm To: numpy-discussion at python.org Subject: [Numpy-discussion] Creating a sine wave with exponential decay Hi everyone! Total Numpy newbie here. I'd like to create an array with a million numbers, that has a sine wave with exponential decay on the amplitude. In other words, I want the value of each cell n to be sin(n) * 2 ** (-n * factor). What would be the most efficient way to do that? Someone suggested I do something like this: y = np.sin(x) * np.exp(newfactor * x) But this would create 2 arrays, wouldn't it? Isn't that wasteful? Does Numpy provide an efficient way of doing that without creating a redundant array? Thanks for your help, Ram Rachum. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jul 23 12:46:44 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 23 Jul 2019 09:46:44 -0700 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, July 24 Message-ID: <782f4ba7403520624e14cb2c7296d7862482870c.camel@sipsolutions.net> Hi all, There will be a NumPy Community meeting Wednesday July 24 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg Best wishes Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sseibert at anaconda.com Tue Jul 23 14:38:01 2019 From: sseibert at anaconda.com (Stanley Seibert) Date: Tue, 23 Jul 2019 13:38:01 -0500 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: Message-ID: (Full disclosure: I work on Numba...) Just to note, the NumPy implementation will allocate (and free) more than 2 arrays to compute that expression. It has to allocate the result array for each operation as Python executes. That expression is equivalent to: s1 = newfactor * x s2 = np.exp(s1) s3 = np.sin(x) y = s3 * s2 However, memory allocation is still pretty fast compared to special math functions (exp and sin), which dominate that calculation. I find this expression takes around 20 milliseconds for a million elements on my older laptop, so that might be negligible in your program execution time unless you need to recreate this decaying exponential thousands of times. Tools like Numba or numexpr will be useful to fuse loops so you only do one allocation, but they aren't necessary unless this becomes the bottleneck in your code. If you are getting started with NumPy, I would suggest not worrying about these issues too much, and focus on making good use of arrays, NumPy array functions, and array expressions in your code. If you have to write for loops (if there is no good way to do the operation with existing NumPy functions), I would reach for something like Numba, and if you want to speed up complex array expressions, both Numba and Numexpr will do a good job. On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi wrote: > Hi Ram, > > No, NumPy doesn?t have a way. And it newer versions, it probably won?t > create two arrays if all the dtypes match, it?ll do some magic to re use > the existing ones, although it will use multiple loops instead of just one. > > You might want to look into NumExpr or Numba if you want an efficient > implementation. > > Get Outlook for iOS > > ------------------------------ > *From:* NumPy-Discussion gmail.com at python.org> on behalf of Ram Rachum > *Sent:* Tuesday, July 23, 2019 7:29 pm > *To:* numpy-discussion at python.org > *Subject:* [Numpy-discussion] Creating a sine wave with exponential decay > > > Hi everyone! Total Numpy newbie here. > > I'd like to create an array with a million numbers, that has a sine wave > with exponential decay on the amplitude. > > In other words, I want the value of each cell n to be sin(n) * 2 ** (-n * > factor). > > What would be the most efficient way to do that? > > Someone suggested I do something like this: > > y = np.sin(x) * np.exp(newfactor * x) > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? Does > Numpy provide an efficient way of doing that without creating a redundant > array? > > > Thanks for your help, > > Ram Rachum. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jul 23 19:31:32 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 23 Jul 2019 16:31:32 -0700 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: Message-ID: On Tue, 2019-07-23 at 13:38 -0500, Stanley Seibert wrote: > (Full disclosure: I work on Numba...) > > Just to note, the NumPy implementation will allocate (and free) more > than 2 arrays to compute that expression. It has to allocate the > result array for each operation as Python executes. That expression > is equivalent to: > That is mostly true, although ? as Hameer mentioned ? on many platforms (gcc compiler is needed I think) a bit of magic happens. If an array is temporary, the operation is replaced with an in-place operation for most python operators calls. For example: `-abs(arr1 * arr2 / arr3 - arr4)` should only create a single new array and keep reusing it in many cases[0]. You would achieve similar things with `arr1 *= arr2` manually. Another thing is that numpy will cache some arrays, so that the allocation cost itself may be avoided in many cases. NumPy does no "loop fusing", i.e. each operation is finished before the next is started. In many cases, with simple math loop fusing can give a very good speedup (which is wher Numba or numexpr come in). Larger speedups are likely if you have large arrays and very simple math (addition). [1] As Stanley noted, you probably should not worry too much about it. You have `exp`/`sin` in there, which are slow by nature. You can try, but it is likely that you simply cannot gain much speed there. Best, Sebastian [0] It requires that the shapes all match and that the result arrays are obviously temporary. [1] For small arrays overheads may be avoided using tools such as numba, which can help a lot as well. If you want to use multiple threads for a specific function that may also be worth a look. > s1 = newfactor * x > s2 = np.exp(s1) > s3 = np.sin(x) > y = s3 * s2 > > However, memory allocation is still pretty fast compared to special > math functions (exp and sin), which dominate that calculation. I > find this expression takes around 20 milliseconds for a million > elements on my older laptop, so that might be negligible in your > program execution time unless you need to recreate this decaying > exponential thousands of times. Tools like Numba or numexpr will be > useful to fuse loops so you only do one allocation, but they aren't > necessary unless this becomes the bottleneck in your code. > > If you are getting started with NumPy, I would suggest not worrying > about these issues too much, and focus on making good use of arrays, > NumPy array functions, and array expressions in your code. If you > have to write for loops (if there is no good way to do the operation > with existing NumPy functions), I would reach for something like > Numba, and if you want to speed up complex array expressions, both > Numba and Numexpr will do a good job. > > > On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi < > einstein.edison at gmail.com> wrote: > > Hi Ram, > > > > No, NumPy doesn?t have a way. And it newer versions, it probably > > won?t create two arrays if all the dtypes match, it?ll do some > > magic to re use the existing ones, although it will use multiple > > loops instead of just one. > > > > You might want to look into NumExpr or Numba if you want an > > efficient implementation. > > > > Get Outlook for iOS > > > > From: NumPy-Discussion < > > numpy-discussion-bounces+einstein.edison=gmail.com at python.org> on > > behalf of Ram Rachum > > Sent: Tuesday, July 23, 2019 7:29 pm > > To: numpy-discussion at python.org > > Subject: [Numpy-discussion] Creating a sine wave with exponential > > decay > > > > Hi everyone! Total Numpy newbie here. > > > > I'd like to create an array with a million numbers, that has a sine > > wave with exponential decay on the amplitude. > > > > In other words, I want the value of each cell n to be sin(n) > > * 2 ** (-n * factor). > > > > What would be the most efficient way to do that? > > > > Someone suggested I do something like this: > > > > y = np.sin(x) * np.exp(newfactor * x) > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? > > Does Numpy provide an efficient way of doing that without creating > > a redundant array? > > > > > > > > Thanks for your help, > > > > Ram Rachum. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From tcaswell at gmail.com Tue Jul 23 19:51:55 2019 From: tcaswell at gmail.com (Thomas Caswell) Date: Tue, 23 Jul 2019 19:51:55 -0400 Subject: [Numpy-discussion] =?utf-8?q?NEP_28_=E2=80=94_A_standard_communi?= =?utf-8?q?ty_policy_for_dropping_support_of_old_Python_and_NumPy_v?= =?utf-8?q?ersions?= Message-ID: Folks, This NEP is a proposing a standard policy for the community to determine when we age-out support for old versions of Python. This came out of in-person discussions at SciPy earlier in July and scattered discussion across github. This is being proposed by maintainers from Matplotlib, scikit-learn, IPython, Jupyter, yt, SciPy, NumPy, and scikit-image. TL;DR: We propose only supporting versions of CPython initially released in the preceding 42 months of a major or minor release of any of our projects. Please see https://github.com/numpy/numpy/pull/14086 and keep the discussion there as there are many interested parties (from the other projects) that may not be subscribed to the numpy mailing list. Tom -- Thomas Caswell tcaswell at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Wed Jul 24 04:39:37 2019 From: faltet at gmail.com (Francesc Alted) Date: Wed, 24 Jul 2019 10:39:37 +0200 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: Message-ID: (disclosure: I have been a long time maintainer of numexpr) An important thing to be noted is that, besides avoiding creating big temporaries, numexpr has support for multi-threading out of the box and Intel VML for optimal evaluation times on Intel CPUs. For choosing your best bet, there is no replacement for experimentation: https://gist.github.com/FrancescAlted/203be8a44d02566f31dae11a22c179f3 I have no time now to check for memory consumption, but you can expect numexpr and Numba consuming barely the same amount of memory. Performance wise things are quite different, but this is probably due to my inexperience with Numba (in particular, paralellism does not seem to work for this example in Numba 0.45, but I am not sure why). Cheers! Probably numba has this too, but my attempts to use parallelism failed Missatge de Sebastian Berg del dia dc., 24 de jul. 2019 a les 1:32: > On Tue, 2019-07-23 at 13:38 -0500, Stanley Seibert wrote: > > (Full disclosure: I work on Numba...) > > > > Just to note, the NumPy implementation will allocate (and free) more > > than 2 arrays to compute that expression. It has to allocate the > > result array for each operation as Python executes. That expression > > is equivalent to: > > > > That is mostly true, although ? as Hameer mentioned ? on many platforms > (gcc compiler is needed I think) a bit of magic happens. > > If an array is temporary, the operation is replaced with an in-place > operation for most python operators calls. For example: > `-abs(arr1 * arr2 / arr3 - arr4)` > should only create a single new array and keep reusing it in many > cases[0]. You would achieve similar things with `arr1 *= arr2` > manually. > > Another thing is that numpy will cache some arrays, so that the > allocation cost itself may be avoided in many cases. > > NumPy does no "loop fusing", i.e. each operation is finished before the > next is started. In many cases, with simple math loop fusing can give a > very good speedup (which is wher Numba or numexpr come in). Larger > speedups are likely if you have large arrays and very simple math > (addition). [1] > > As Stanley noted, you probably should not worry too much about it. You > have `exp`/`sin` in there, which are slow by nature. You can try, but > it is likely that you simply cannot gain much speed there. > > Best, > > Sebastian > > > [0] It requires that the shapes all match and that the result arrays > are obviously temporary. > [1] For small arrays overheads may be avoided using tools such as > numba, which can help a lot as well. If you want to use multiple > threads for a specific function that may also be worth a look. > > > s1 = newfactor * x > > s2 = np.exp(s1) > > s3 = np.sin(x) > > y = s3 * s2 > > > > However, memory allocation is still pretty fast compared to special > > math functions (exp and sin), which dominate that calculation. I > > find this expression takes around 20 milliseconds for a million > > elements on my older laptop, so that might be negligible in your > > program execution time unless you need to recreate this decaying > > exponential thousands of times. Tools like Numba or numexpr will be > > useful to fuse loops so you only do one allocation, but they aren't > > necessary unless this becomes the bottleneck in your code. > > > > If you are getting started with NumPy, I would suggest not worrying > > about these issues too much, and focus on making good use of arrays, > > NumPy array functions, and array expressions in your code. If you > > have to write for loops (if there is no good way to do the operation > > with existing NumPy functions), I would reach for something like > > Numba, and if you want to speed up complex array expressions, both > > Numba and Numexpr will do a good job. > > > > > > On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi < > > einstein.edison at gmail.com> wrote: > > > Hi Ram, > > > > > > No, NumPy doesn?t have a way. And it newer versions, it probably > > > won?t create two arrays if all the dtypes match, it?ll do some > > > magic to re use the existing ones, although it will use multiple > > > loops instead of just one. > > > > > > You might want to look into NumExpr or Numba if you want an > > > efficient implementation. > > > > > > Get Outlook for iOS > > > > > > From: NumPy-Discussion < > > > numpy-discussion-bounces+einstein.edison=gmail.com at python.org> on > > > behalf of Ram Rachum > > > Sent: Tuesday, July 23, 2019 7:29 pm > > > To: numpy-discussion at python.org > > > Subject: [Numpy-discussion] Creating a sine wave with exponential > > > decay > > > > > > Hi everyone! Total Numpy newbie here. > > > > > > I'd like to create an array with a million numbers, that has a sine > > > wave with exponential decay on the amplitude. > > > > > > In other words, I want the value of each cell n to be sin(n) > > > * 2 ** (-n * factor). > > > > > > What would be the most efficient way to do that? > > > > > > Someone suggested I do something like this: > > > > > > y = np.sin(x) * np.exp(newfactor * x) > > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? > > > Does Numpy provide an efficient way of doing that without creating > > > a redundant array? > > > > > > > > > > > > Thanks for your help, > > > > > > Ram Rachum. > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.elnajami at thisismetis.com Wed Jul 24 19:55:46 2019 From: michael.elnajami at thisismetis.com (Michael Elnajami) Date: Wed, 24 Jul 2019 18:55:46 -0500 Subject: [Numpy-discussion] The FREE Demystifying Data Science Conference is happening again! July 30-31!!! Message-ID: Metis is hosting our 3rd annual *FREE *Demystifying Data Science Conference July 30-31 from 10am - 5pm ET! There will be 16 interactive data science talks and 6 workshops led by industry-leading speakers. Experience interactivity via real-time chat with world-wide audience, polling, and social sharing. Plus, get access to recordings of every presentation and workshop. I know many of you NumPy users also are data scientists, so please check this out! This is a great opportunity to learn and connect with like minded people! Day 1 - July 30 will be for aspiring data scientists. Day 2 - July 31 will be for business leaders and practitioners Registration page: https://www.thisismetis.com/demystifying-data-science -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 24 22:19:00 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 24 Jul 2019 19:19:00 -0700 Subject: [Numpy-discussion] Season of Docs applications In-Reply-To: References: Message-ID: Hi all, Here is a short update on Season of Docs. We are very happy and impressed with the number and quality of tech writers who applied to NumPy and SciPy. We decided we would really like to accept more people than the number of slots Google is likely to give us. We have found some more funding and mentoring bandwidth to be able to do so. After Google announces which candidates have been accepted, we will reach out to the next people on our shortlist and offer them the same mentoring and funding for their proposed projects as the accepted GSoD writers. I'm really looking forward to an interesting and productive fall working with several of you. Stay tuned! Cheers, Ralf On Wed, Jul 17, 2019 at 5:12 PM Ralf Gommers wrote: > Hi GSoD applicants, > > To start with, I want to thank all of you who applied! There's a lot of > interest in helping improve NumPy's documentation and web presence, which > is awesome to see. During the application phase, we have interacted with > some of you already, and I will reach out to others over the next few days. > I'm just writing this email to say thank you and to confirm that yes we did > receive your application. Due to the volume of applications, it will be > difficult to reach out personally to all of you (I'll do my best though!), > hence this email. > > Really looking forward to what will happen next! > > Cheers, > Ralf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni at fastmail.com Wed Jul 24 22:29:13 2019 From: jni at fastmail.com (Juan Nunez-Iglesias) Date: Thu, 25 Jul 2019 12:29:13 +1000 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: Message-ID: <5E0AE8B8-34E6-47CC-8A33-6915F7ED41F2@fastmail.com> Hi Francesc, Those numbers are really eye-popping! But the formatting of the code as a string still bugs me a lot. Asking this as a totally naive user, do you know whether PEP523 (adding a frame evaluation API) would allow numexpr to have a more Pythonic syntax? eg. with numexpr: y = np.sin(x) * np.exp(newfactor * x) ? Juan. > On 24 Jul 2019, at 6:39 pm, Francesc Alted wrote: > > (disclosure: I have been a long time maintainer of numexpr) > > An important thing to be noted is that, besides avoiding creating big temporaries, numexpr has support for multi-threading out of the box and Intel VML for optimal evaluation times on Intel CPUs. For choosing your best bet, there is no replacement for experimentation: > > https://gist.github.com/FrancescAlted/203be8a44d02566f31dae11a22c179f3 > > I have no time now to check for memory consumption, but you can expect numexpr and Numba consuming barely the same amount of memory. Performance wise things are quite different, but this is probably due to my inexperience with Numba (in particular, paralellism does not seem to work for this example in Numba 0.45, but I am not sure why). > > Cheers! > > > Probably numba has this too, but my attempts to use parallelism failed > > Missatge de Sebastian Berg > del dia dc., 24 de jul. 2019 a les 1:32: > On Tue, 2019-07-23 at 13:38 -0500, Stanley Seibert wrote: > > (Full disclosure: I work on Numba...) > > > > Just to note, the NumPy implementation will allocate (and free) more > > than 2 arrays to compute that expression. It has to allocate the > > result array for each operation as Python executes. That expression > > is equivalent to: > > > > That is mostly true, although ? as Hameer mentioned ? on many platforms > (gcc compiler is needed I think) a bit of magic happens. > > If an array is temporary, the operation is replaced with an in-place > operation for most python operators calls. For example: > `-abs(arr1 * arr2 / arr3 - arr4)` > should only create a single new array and keep reusing it in many > cases[0]. You would achieve similar things with `arr1 *= arr2` > manually. > > Another thing is that numpy will cache some arrays, so that the > allocation cost itself may be avoided in many cases. > > NumPy does no "loop fusing", i.e. each operation is finished before the > next is started. In many cases, with simple math loop fusing can give a > very good speedup (which is wher Numba or numexpr come in). Larger > speedups are likely if you have large arrays and very simple math > (addition). [1] > > As Stanley noted, you probably should not worry too much about it. You > have `exp`/`sin` in there, which are slow by nature. You can try, but > it is likely that you simply cannot gain much speed there. > > Best, > > Sebastian > > > [0] It requires that the shapes all match and that the result arrays > are obviously temporary. > [1] For small arrays overheads may be avoided using tools such as > numba, which can help a lot as well. If you want to use multiple > threads for a specific function that may also be worth a look. > > > s1 = newfactor * x > > s2 = np.exp(s1) > > s3 = np.sin(x) > > y = s3 * s2 > > > > However, memory allocation is still pretty fast compared to special > > math functions (exp and sin), which dominate that calculation. I > > find this expression takes around 20 milliseconds for a million > > elements on my older laptop, so that might be negligible in your > > program execution time unless you need to recreate this decaying > > exponential thousands of times. Tools like Numba or numexpr will be > > useful to fuse loops so you only do one allocation, but they aren't > > necessary unless this becomes the bottleneck in your code. > > > > If you are getting started with NumPy, I would suggest not worrying > > about these issues too much, and focus on making good use of arrays, > > NumPy array functions, and array expressions in your code. If you > > have to write for loops (if there is no good way to do the operation > > with existing NumPy functions), I would reach for something like > > Numba, and if you want to speed up complex array expressions, both > > Numba and Numexpr will do a good job. > > > > > > On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi < > > einstein.edison at gmail.com > wrote: > > > Hi Ram, > > > > > > No, NumPy doesn?t have a way. And it newer versions, it probably > > > won?t create two arrays if all the dtypes match, it?ll do some > > > magic to re use the existing ones, although it will use multiple > > > loops instead of just one. > > > > > > You might want to look into NumExpr or Numba if you want an > > > efficient implementation. > > > > > > Get Outlook for iOS > > > > > > From: NumPy-Discussion < > > > numpy-discussion-bounces+einstein.edison=gmail.com at python.org > on > > > behalf of Ram Rachum > > > > Sent: Tuesday, July 23, 2019 7:29 pm > > > To: numpy-discussion at python.org > > > Subject: [Numpy-discussion] Creating a sine wave with exponential > > > decay > > > > > > Hi everyone! Total Numpy newbie here. > > > > > > I'd like to create an array with a million numbers, that has a sine > > > wave with exponential decay on the amplitude. > > > > > > In other words, I want the value of each cell n to be sin(n) > > > * 2 ** (-n * factor). > > > > > > What would be the most efficient way to do that? > > > > > > Someone suggested I do something like this: > > > > > > y = np.sin(x) * np.exp(newfactor * x) > > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? > > > Does Numpy provide an efficient way of doing that without creating > > > a redundant array? > > > > > > > > > > > > Thanks for your help, > > > > > > Ram Rachum. > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at python.org > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > > -- > Francesc Alted > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jul 25 13:35:13 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Jul 2019 11:35:13 -0600 Subject: [Numpy-discussion] Adding generic dtypes Message-ID: Hi All, The possibility of disabling default creation of object arrays has come up again. I'm wondering if one way to get there is to allow generic dtypes. The `numpy/core/numerictypes.py` module defines a hierarchy, and if we could allow things like `dtype=integer` or `dtype=no_object` and such we would gain more control over the resulting array types. At some point we could make `no_object` the default if we so desired. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Jul 25 14:23:06 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 25 Jul 2019 11:23:06 -0700 Subject: [Numpy-discussion] Adding generic dtypes In-Reply-To: References: Message-ID: <24ecc85b4ae0f423ca0ff0e9bed260663a101bf9.camel@sipsolutions.net> Hey, On Thu, 2019-07-25 at 11:35 -0600, Charles R Harris wrote: > Hi All, > > The possibility of disabling default creation of object arrays has > come up again. I'm wondering if one way to get there is to allow > generic dtypes. The `numpy/core/numerictypes.py` module defines a > hierarchy, and if we could allow things like `dtype=integer` or > `dtype=no_object` and such we would gain more control over the > resulting array types. At some point we could make `no_object` the > default if we so desired. > > Thoughts? personally, that is what I am currently thinking along. I was calling them AbstractDTypes. These would also (probably mainly) be used for type promotion during ufunc calls (ufunc type resolution). OTOH, for `no_object` itself, I feel it would be OK with just special case it, since the I do believe the plan should be to deprecate it. I have to think a bit about how "flexible" dtypes come in (and value based promotion, which is a bit of a funny beast). I would like to plan a zoom meeting beginning of next week to discuss this type of thing a bit more. Maybe Monday at 11 California time, but right now I am very flexible, so whoever would be interested, feel free to shoot me a mail to move things. Not sure how concrete things will be, but without talking about it, it is difficult for me to settle design ideas a bit more. Best, Sebastian > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From einstein.edison at gmail.com Thu Jul 25 14:34:14 2019 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Thu, 25 Jul 2019 18:34:14 +0000 Subject: [Numpy-discussion] Adding generic dtypes In-Reply-To: <24ecc85b4ae0f423ca0ff0e9bed260663a101bf9.camel@sipsolutions.net> References: , <24ecc85b4ae0f423ca0ff0e9bed260663a101bf9.camel@sipsolutions.net> Message-ID: Hi Sebastian; all: I plan to attend the dtype meeting. Monday 11 AM California time is fine for me. Best regards, Hameer Abbasi Get Outlook for iOS ________________________________ From: NumPy-Discussion on behalf of Sebastian Berg Sent: Thursday, July 25, 2019 11:23 pm To: numpy-discussion at python.org Subject: Re: [Numpy-discussion] Adding generic dtypes Hey, On Thu, 2019-07-25 at 11:35 -0600, Charles R Harris wrote: > Hi All, > > The possibility of disabling default creation of object arrays has > come up again. I'm wondering if one way to get there is to allow > generic dtypes. The `numpy/core/numerictypes.py` module defines a > hierarchy, and if we could allow things like `dtype=integer` or > `dtype=no_object` and such we would gain more control over the > resulting array types. At some point we could make `no_object` the > default if we so desired. > > Thoughts? personally, that is what I am currently thinking along. I was calling them AbstractDTypes. These would also (probably mainly) be used for type promotion during ufunc calls (ufunc type resolution). OTOH, for `no_object` itself, I feel it would be OK with just special case it, since the I do believe the plan should be to deprecate it. I have to think a bit about how "flexible" dtypes come in (and value based promotion, which is a bit of a funny beast). I would like to plan a zoom meeting beginning of next week to discuss this type of thing a bit more. Maybe Monday at 11 California time, but right now I am very flexible, so whoever would be interested, feel free to shoot me a mail to move things. Not sure how concrete things will be, but without talking about it, it is difficult for me to settle design ideas a bit more. Best, Sebastian > > Chuck > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at gmail.com Fri Jul 26 02:06:36 2019 From: faltet at gmail.com (Francesc Alted) Date: Fri, 26 Jul 2019 08:06:36 +0200 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: <5E0AE8B8-34E6-47CC-8A33-6915F7ED41F2@fastmail.com> References: <5E0AE8B8-34E6-47CC-8A33-6915F7ED41F2@fastmail.com> Message-ID: Hi Juan, With the time I have grown to appreciate the simplicity of strings for representing the expressions that numexpr is designed to tackle. Having said that, PEP 523 looks intriguing indeed. As always, PRs are welcome! Francesc El dj., 25 jul. 2019, 4.29, Juan Nunez-Iglesias va escriure: > Hi Francesc, > > Those numbers are really eye-popping! But the formatting of the code as a > string still bugs me a lot. Asking this as a totally naive user, do you > know whether PEP523 (adding a > frame evaluation API) would allow numexpr to have a more Pythonic syntax? > eg. > > with numexpr: > y = np.sin(x) * np.exp(newfactor * x) > > ? > > Juan. > > > On 24 Jul 2019, at 6:39 pm, Francesc Alted wrote: > > (disclosure: I have been a long time maintainer of numexpr) > > An important thing to be noted is that, besides avoiding creating big > temporaries, numexpr has support for multi-threading out of the box and > Intel VML for optimal evaluation times on Intel CPUs. For choosing your > best bet, there is no replacement for experimentation: > > https://gist.github.com/FrancescAlted/203be8a44d02566f31dae11a22c179f3 > > I have no time now to check for memory consumption, but you can expect > numexpr and Numba consuming barely the same amount of memory. Performance > wise things are quite different, but this is probably due to my > inexperience with Numba (in particular, paralellism does not seem to work > for this example in Numba 0.45, but I am not sure why). > > Cheers! > > > Probably numba has this too, but my attempts to use parallelism failed > > Missatge de Sebastian Berg del dia dc., 24 > de jul. 2019 a les 1:32: > >> On Tue, 2019-07-23 at 13:38 -0500, Stanley Seibert wrote: >> > (Full disclosure: I work on Numba...) >> > >> > Just to note, the NumPy implementation will allocate (and free) more >> > than 2 arrays to compute that expression. It has to allocate the >> > result array for each operation as Python executes. That expression >> > is equivalent to: >> > >> >> That is mostly true, although ? as Hameer mentioned ? on many platforms >> (gcc compiler is needed I think) a bit of magic happens. >> >> If an array is temporary, the operation is replaced with an in-place >> operation for most python operators calls. For example: >> `-abs(arr1 * arr2 / arr3 - arr4)` >> should only create a single new array and keep reusing it in many >> cases[0]. You would achieve similar things with `arr1 *= arr2` >> manually. >> >> Another thing is that numpy will cache some arrays, so that the >> allocation cost itself may be avoided in many cases. >> >> NumPy does no "loop fusing", i.e. each operation is finished before the >> next is started. In many cases, with simple math loop fusing can give a >> very good speedup (which is wher Numba or numexpr come in). Larger >> speedups are likely if you have large arrays and very simple math >> (addition). [1] >> >> As Stanley noted, you probably should not worry too much about it. You >> have `exp`/`sin` in there, which are slow by nature. You can try, but >> it is likely that you simply cannot gain much speed there. >> >> Best, >> >> Sebastian >> >> >> [0] It requires that the shapes all match and that the result arrays >> are obviously temporary. >> [1] For small arrays overheads may be avoided using tools such as >> numba, which can help a lot as well. If you want to use multiple >> threads for a specific function that may also be worth a look. >> >> > s1 = newfactor * x >> > s2 = np.exp(s1) >> > s3 = np.sin(x) >> > y = s3 * s2 >> > >> > However, memory allocation is still pretty fast compared to special >> > math functions (exp and sin), which dominate that calculation. I >> > find this expression takes around 20 milliseconds for a million >> > elements on my older laptop, so that might be negligible in your >> > program execution time unless you need to recreate this decaying >> > exponential thousands of times. Tools like Numba or numexpr will be >> > useful to fuse loops so you only do one allocation, but they aren't >> > necessary unless this becomes the bottleneck in your code. >> > >> > If you are getting started with NumPy, I would suggest not worrying >> > about these issues too much, and focus on making good use of arrays, >> > NumPy array functions, and array expressions in your code. If you >> > have to write for loops (if there is no good way to do the operation >> > with existing NumPy functions), I would reach for something like >> > Numba, and if you want to speed up complex array expressions, both >> > Numba and Numexpr will do a good job. >> > >> > >> > On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi < >> > einstein.edison at gmail.com> wrote: >> > > Hi Ram, >> > > >> > > No, NumPy doesn?t have a way. And it newer versions, it probably >> > > won?t create two arrays if all the dtypes match, it?ll do some >> > > magic to re use the existing ones, although it will use multiple >> > > loops instead of just one. >> > > >> > > You might want to look into NumExpr or Numba if you want an >> > > efficient implementation. >> > > >> > > Get Outlook for iOS >> > > >> > > From: NumPy-Discussion < >> > > numpy-discussion-bounces+einstein.edison=gmail.com at python.org> on >> > > behalf of Ram Rachum >> > > Sent: Tuesday, July 23, 2019 7:29 pm >> > > To: numpy-discussion at python.org >> > > Subject: [Numpy-discussion] Creating a sine wave with exponential >> > > decay >> > > >> > > Hi everyone! Total Numpy newbie here. >> > > >> > > I'd like to create an array with a million numbers, that has a sine >> > > wave with exponential decay on the amplitude. >> > > >> > > In other words, I want the value of each cell n to be sin(n) >> > > * 2 ** (-n * factor). >> > > >> > > What would be the most efficient way to do that? >> > > >> > > Someone suggested I do something like this: >> > > >> > > y = np.sin(x) * np.exp(newfactor * x) >> > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? >> > > Does Numpy provide an efficient way of doing that without creating >> > > a redundant array? >> > > >> > > >> > > >> > > Thanks for your help, >> > > >> > > Ram Rachum. >> > > >> > > _______________________________________________ >> > > NumPy-Discussion mailing list >> > > NumPy-Discussion at python.org >> > > https://mail.python.org/mailman/listinfo/numpy-discussion >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at python.org >> > https://mail.python.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > > > -- > Francesc Alted > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Fri Jul 26 07:09:13 2019 From: ram at rachum.com (Ram Rachum) Date: Fri, 26 Jul 2019 14:09:13 +0300 Subject: [Numpy-discussion] Creating a sine wave with exponential decay In-Reply-To: References: <5E0AE8B8-34E6-47CC-8A33-6915F7ED41F2@fastmail.com> Message-ID: Thanks for your answers and insight, everybody! On Fri, Jul 26, 2019 at 9:07 AM Francesc Alted wrote: > Hi Juan, > > With the time I have grown to appreciate the simplicity of strings for > representing the expressions that numexpr is designed to tackle. Having > said that, PEP 523 looks intriguing indeed. As always, PRs are welcome! > > Francesc > > El dj., 25 jul. 2019, 4.29, Juan Nunez-Iglesias va > escriure: > >> Hi Francesc, >> >> Those numbers are really eye-popping! But the formatting of the code as a >> string still bugs me a lot. Asking this as a totally naive user, do you >> know whether PEP523 (adding >> a frame evaluation API) would allow numexpr to have a more Pythonic syntax? >> eg. >> >> with numexpr: >> y = np.sin(x) * np.exp(newfactor * x) >> >> ? >> >> Juan. >> >> >> On 24 Jul 2019, at 6:39 pm, Francesc Alted wrote: >> >> (disclosure: I have been a long time maintainer of numexpr) >> >> An important thing to be noted is that, besides avoiding creating big >> temporaries, numexpr has support for multi-threading out of the box and >> Intel VML for optimal evaluation times on Intel CPUs. For choosing your >> best bet, there is no replacement for experimentation: >> >> https://gist.github.com/FrancescAlted/203be8a44d02566f31dae11a22c179f3 >> >> I have no time now to check for memory consumption, but you can expect >> numexpr and Numba consuming barely the same amount of memory. Performance >> wise things are quite different, but this is probably due to my >> inexperience with Numba (in particular, paralellism does not seem to work >> for this example in Numba 0.45, but I am not sure why). >> >> Cheers! >> >> >> Probably numba has this too, but my attempts to use parallelism failed >> >> Missatge de Sebastian Berg del dia dc., 24 >> de jul. 2019 a les 1:32: >> >>> On Tue, 2019-07-23 at 13:38 -0500, Stanley Seibert wrote: >>> > (Full disclosure: I work on Numba...) >>> > >>> > Just to note, the NumPy implementation will allocate (and free) more >>> > than 2 arrays to compute that expression. It has to allocate the >>> > result array for each operation as Python executes. That expression >>> > is equivalent to: >>> > >>> >>> That is mostly true, although ? as Hameer mentioned ? on many platforms >>> (gcc compiler is needed I think) a bit of magic happens. >>> >>> If an array is temporary, the operation is replaced with an in-place >>> operation for most python operators calls. For example: >>> `-abs(arr1 * arr2 / arr3 - arr4)` >>> should only create a single new array and keep reusing it in many >>> cases[0]. You would achieve similar things with `arr1 *= arr2` >>> manually. >>> >>> Another thing is that numpy will cache some arrays, so that the >>> allocation cost itself may be avoided in many cases. >>> >>> NumPy does no "loop fusing", i.e. each operation is finished before the >>> next is started. In many cases, with simple math loop fusing can give a >>> very good speedup (which is wher Numba or numexpr come in). Larger >>> speedups are likely if you have large arrays and very simple math >>> (addition). [1] >>> >>> As Stanley noted, you probably should not worry too much about it. You >>> have `exp`/`sin` in there, which are slow by nature. You can try, but >>> it is likely that you simply cannot gain much speed there. >>> >>> Best, >>> >>> Sebastian >>> >>> >>> [0] It requires that the shapes all match and that the result arrays >>> are obviously temporary. >>> [1] For small arrays overheads may be avoided using tools such as >>> numba, which can help a lot as well. If you want to use multiple >>> threads for a specific function that may also be worth a look. >>> >>> > s1 = newfactor * x >>> > s2 = np.exp(s1) >>> > s3 = np.sin(x) >>> > y = s3 * s2 >>> > >>> > However, memory allocation is still pretty fast compared to special >>> > math functions (exp and sin), which dominate that calculation. I >>> > find this expression takes around 20 milliseconds for a million >>> > elements on my older laptop, so that might be negligible in your >>> > program execution time unless you need to recreate this decaying >>> > exponential thousands of times. Tools like Numba or numexpr will be >>> > useful to fuse loops so you only do one allocation, but they aren't >>> > necessary unless this becomes the bottleneck in your code. >>> > >>> > If you are getting started with NumPy, I would suggest not worrying >>> > about these issues too much, and focus on making good use of arrays, >>> > NumPy array functions, and array expressions in your code. If you >>> > have to write for loops (if there is no good way to do the operation >>> > with existing NumPy functions), I would reach for something like >>> > Numba, and if you want to speed up complex array expressions, both >>> > Numba and Numexpr will do a good job. >>> > >>> > >>> > On Tue, Jul 23, 2019 at 10:38 AM Hameer Abbasi < >>> > einstein.edison at gmail.com> wrote: >>> > > Hi Ram, >>> > > >>> > > No, NumPy doesn?t have a way. And it newer versions, it probably >>> > > won?t create two arrays if all the dtypes match, it?ll do some >>> > > magic to re use the existing ones, although it will use multiple >>> > > loops instead of just one. >>> > > >>> > > You might want to look into NumExpr or Numba if you want an >>> > > efficient implementation. >>> > > >>> > > Get Outlook for iOS >>> > > >>> > > From: NumPy-Discussion < >>> > > numpy-discussion-bounces+einstein.edison=gmail.com at python.org> on >>> > > behalf of Ram Rachum >>> > > Sent: Tuesday, July 23, 2019 7:29 pm >>> > > To: numpy-discussion at python.org >>> > > Subject: [Numpy-discussion] Creating a sine wave with exponential >>> > > decay >>> > > >>> > > Hi everyone! Total Numpy newbie here. >>> > > >>> > > I'd like to create an array with a million numbers, that has a sine >>> > > wave with exponential decay on the amplitude. >>> > > >>> > > In other words, I want the value of each cell n to be sin(n) >>> > > * 2 ** (-n * factor). >>> > > >>> > > What would be the most efficient way to do that? >>> > > >>> > > Someone suggested I do something like this: >>> > > >>> > > y = np.sin(x) * np.exp(newfactor * x) >>> > > But this would create 2 arrays, wouldn't it? Isn't that wasteful? >>> > > Does Numpy provide an efficient way of doing that without creating >>> > > a redundant array? >>> > > >>> > > >>> > > >>> > > Thanks for your help, >>> > > >>> > > Ram Rachum. >>> > > >>> > > _______________________________________________ >>> > > NumPy-Discussion mailing list >>> > > NumPy-Discussion at python.org >>> > > https://mail.python.org/mailman/listinfo/numpy-discussion >>> > >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at python.org >>> > https://mail.python.org/mailman/listinfo/numpy-discussion >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >> >> >> -- >> Francesc Alted >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jul 26 15:02:06 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 26 Jul 2019 13:02:06 -0600 Subject: [Numpy-discussion] NumPy 1.17.0 released In-Reply-To: References: Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0. The 1.17.0 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b2 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: - A new extensible random module along with four selectable random number generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. - NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. - New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. - Overriding numpy functions is now possible by default Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Contributors* A total of 150 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Voelker + - Abdur Rehman + - Abdur-Rahmaan Janhangeer + - Abhinav Sagar + - Adam J. Stewart + - Adam Orr + - Albert Thomas + - Alex Watt + - Alexander Blinne + - Alexander Shadchin - Allan Haldane - Ander Ustarroz + - Andras Deak - Andrea Pattori + - Andreas Schwab - Andrew Naguib + - Andy Scholand + - Ankit Shukla + - Anthony Sottile - Antoine Pitrou - Antony Lee - Arcesio Castaneda Medina + - Assem + - Bernardt Duvenhage + - Bharat Raghunathan + - Bharat123rox + - Bran + - Bruce Merry + - Charles Harris - Chirag Nighut + - Christoph Gohlke - Christopher Whelan + - Chuanzhu Xu + - Colin Snyder + - Dan Allan + - Daniel Hrisca - Daniel Lawrence + - Debsankha Manik + - Dennis Zollo + - Dieter Werthm?ller + - Dominic Jack + - EelcoPeacs + - Eric Larson - Eric Wieser - Fabrice Fontaine + - Gary Gurlaskie + - Gregory Lee + - Gregory R. Lee - Guillaume Horel + - Hameer Abbasi - Haoyu Sun + - Harmon + - He Jia + - Hunter Damron + - Ian Sanders + - Ilja + - Isaac Virshup + - Isaiah Norton + - Jackie Leng + - Jaime Fernandez - Jakub Wilk - Jan S. (Milania1) + - Jarrod Millman - Javier Dehesa + - Jeremy Lay + - Jim Turner + - Jingbei Li + - Joachim Hereth + - Johannes Hampp + - John Belmonte + - John Kirkham - John Law + - Jonas Jensen - Joseph Fox-Rabinovitz - Joseph Martinot-Lagarde - Josh Wilson - Juan Luis Cano Rodr?guez - Julian Taylor - J?r?mie du Boisberranger + - Kai Striega + - Katharine Hyatt + - Kevin Sheppard - Kexuan Sun - Kiko Correoso + - Kriti Singh + - Lars Grueter + - Luis Pedro Coelho - Maksim Shabunin + - Manvi07 + - Mark Harfouche - Marten van Kerkwijk - Martin Reinecke + - Matthew Brett - Matthias Bussonnier - Matti Picus - Michel Fruchart + - Mike Lui + - Mike Taves + - Min ho Kim + - Mircea Akos Bruma - Nick Minkyu Lee - Nick Papior - Nick R. Papior + - Nicola Soranzo + - Nimish Telang + - OBATA Akio + - Oleksandr Pavlyk - Ori Broda + - Paul Ivanov - Pauli Virtanen - Peter Andreas Entschev + - Peter Bell + - Pierre de Buyl - Piyush Jaipuriayar + - Prithvi MK + - Raghuveer Devulapalli + - Ralf Gommers - Richard Harris + - Rishabh Chakrabarti + - Riya Sharma + - Robert Kern - Roman Yurchak - Ryan Levy + - Sebastian Berg - Sergei Lebedev + - Shekhar Prasad Rajak + - Stefan van der Walt - Stephan Hoyer - Steve Stagg + - SuryaChand P + - S?ren Rasmussen + - Thibault Hallouin + - Thomas A Caswell - Tobias Uelwer + - Tony LaTorre + - Toshiki Kataoka - Tyler Moncur + - Tyler Reddy - Valentin Haenel - Vrinda Narayan + - Warren Weckesser - Weitang Li - Wojtek Ruszczewski - Yu Feng - Yu Kobayashi + - Yury Kirienko + - @aashuli + - @luzpaz - @parul + - @spacescientist + Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From ansgar.wehrhahn at physics.uu.se Mon Jul 29 10:57:44 2019 From: ansgar.wehrhahn at physics.uu.se (Ansgar Wehrhahn) Date: Mon, 29 Jul 2019 16:57:44 +0200 Subject: [Numpy-discussion] ENH: 2D polynomial fit Message-ID: <24c5ea39-e060-f66f-f96c-0c830fa2162b@physics.uu.se> Hi, I made a function for 2d polynomial fits that I would like to contribute to numpy, as it is something that is not yet covered by numpy. I already made a pull request (https://github.com/numpy/numpy/pull/14151), but new functions need to be discussed. Any comments? Ansgar N?r du har kontakt med oss p? Uppsala universitet med e-post s? inneb?r det att vi behandlar dina personuppgifter. F?r att l?sa mer om hur vi g?r det kan du l?sa h?r: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy From sebastian at sipsolutions.net Mon Jul 29 13:19:24 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 29 Jul 2019 10:19:24 -0700 Subject: [Numpy-discussion] Adding generic dtypes In-Reply-To: References: , <24ecc85b4ae0f423ca0ff0e9bed260663a101bf9.camel@sipsolutions.net> Message-ID: Hi Hameer, all, sorry for the super late follow-up. Not that there is any content, but I just put this up quickly: https://hackmd.io/xSYBZI-cQ-etapoExfM1Cg a bit also because I am not sure if the zoom link will work, so I want to be able to update it. See you, Sebastian On Thu, 2019-07-25 at 18:34 +0000, Hameer Abbasi wrote: > Hi Sebastian; all: > > I plan to attend the dtype meeting. Monday 11 AM California time is > fine for me. > > Best regards, > Hameer Abbasi > > Get Outlook for iOS > > From: NumPy-Discussion < > numpy-discussion-bounces+einstein.edison=gmail.com at python.org> on > behalf of Sebastian Berg > Sent: Thursday, July 25, 2019 11:23 pm > To: numpy-discussion at python.org > Subject: Re: [Numpy-discussion] Adding generic dtypes > > Hey, > > On Thu, 2019-07-25 at 11:35 -0600, Charles R Harris wrote: > > Hi All, > > > > The possibility of disabling default creation of object arrays has > > come up again. I'm wondering if one way to get there is to allow > > generic dtypes. The `numpy/core/numerictypes.py` module defines a > > hierarchy, and if we could allow things like `dtype=integer` or > > `dtype=no_object` and such we would gain more control over the > > resulting array types. At some point we could make `no_object` the > > default if we so desired. > > > > Thoughts? > > personally, that is what I am currently thinking along. I was > calling > them AbstractDTypes. These would also (probably mainly) be used for > type promotion during ufunc calls (ufunc type resolution). > > OTOH, for `no_object` itself, I feel it would be OK with just > special > case it, since the I do believe the plan should be to deprecate it. > > I have to think a bit about how "flexible" dtypes come in (and value > based promotion, which is a bit of a funny beast). > > I would like to plan a zoom meeting beginning of next week to > discuss > this type of thing a bit more. Maybe Monday at 11 California time, > but > right now I am very flexible, so whoever would be interested, feel > free > to shoot me a mail to move things. > Not sure how concrete things will be, but without talking about it, > it > is difficult for me to settle design ideas a bit more. > > Best, > > Sebastian > > > > > > Chuck > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Tue Jul 30 14:28:45 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 30 Jul 2019 11:28:45 -0700 Subject: [Numpy-discussion] NumPy Community Meeting Wednesday, July 17 Message-ID: <4a61a9d97e36fe986a4e59bb43e86ce6837c2a4e.camel@sipsolutions.net> Hi all, There will be a NumPy Community meeting Wednesday July 31 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg Best wishes Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From Cedric.Hannotier at ulb.ac.be Wed Jul 31 08:05:39 2019 From: Cedric.Hannotier at ulb.ac.be (=?UTF-8?Q?C=C3=A9dric_Hannotier?=) Date: Wed, 31 Jul 2019 14:05:39 +0200 Subject: [Numpy-discussion] Perturbations in Polynomial roots Message-ID: <0fc7d6cf5712183419632c289c8253bf@imapproxy.vub.ac.be> Dear all, I am not sure if I misunderstood the Polynomial.roots() method, but I get strange behavior. The roots are way outside of the expected values. That is not the case with np.roots(). What am I doing wrong? Code: import numpy as np def roots1(coef, x, y): for i in range(y.size): coefN = coef.copy() coefN[0] -= y[i] print(np.roots(coefN[::-1]) - x[i]) def roots2(coef, x, y): for i in range(y.size): print((np.polynomial.polynomial.Polynomial(coef) - y[i]).roots() - x[i]) x = np.arange(4) * 0.1 alpha, beta, gamma = 0.16, 1, 1e-15 coef = np.array([alpha, beta, gamma]) y = alpha + beta*x + gamma*x**2 print("roots") roots1(coef, x, y) print('Polynomial') roots2(coef, x, y) Outputs: roots [-1.e+15 0.e+00] [-1.e+15 0.e+00] [-1.00000000e+15 2.77555756e-17] [-1.00000000e+15 5.55111512e-17] Polynomial [-1.e+15 0.e+00] [-1.0e+15 2.5e-02] [-1.0e+15 -7.5e-02] [-1.e+15 -5.e-02] Best Regards, C?dric Hannotier