NumPy-Discussion October 2016

numpy-discussion@python.org

47 participants
31 discussions

Scipy installation on Window with mingw32
by Shreyank Amartya 18 Oct '16

18 Oct '16

Hi, I am trying install to theano which also requires numpy and scipy on windows 7 with mingw32 compilers. I have successfully installed numpy using mingw32 but however when trying to install scipy I get this error: Looking for python27.dll Building msvcr library: "c:\python27\libs\libmsvcr90.a" (from C:\Windows\win sxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\ msvcr90.dll) objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0 .21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized Traceback (most recent call last): File "<string>", line 1, in <module> File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\setup.py", line 415, in <module> setup_package() File "c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\setup.py", line 411, in setup_package setup(**metadata) File "c:\python27\lib\site-packages\numpy\distutils\core.py", line 169, in setup return old_setup(**new_attr) File "c:\python27\lib\distutils\core.py", line 151, in setup dist.run_commands() File "c:\python27\lib\distutils\dist.py", line 953, in run_commands self.run_command(cmd) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", l ine 62, in run r = self.setuptools_run() File "c:\python27\lib\site-packages\numpy\distutils\command\install.py", l ine 36, in setuptools_run return distutils_install.run(self) File "c:\python27\lib\distutils\command\install.py", line 563, in run self.run_command('build') File "c:\python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\build.py", lin e 47, in run old_build.run(self) File "c:\python27\lib\distutils\command\build.py", line 127, in run self.run_command(cmd_name) File "c:\python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "c:\python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 147, in run self.build_sources() File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 164, in build_sources self.build_extension_sources(ext) File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 323, in build_extension_sources sources = self.generate_sources(sources, ext) File "c:\python27\lib\site-packages\numpy\distutils\command\build_src.py", line 376, in generate_sources source = func(extension, build_dir) File "scipy\spatial\setup.py", line 35, in get_qhull_misc_config if config_cmd.check_func('open_memstream', decl=True, call=True): File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", li ne 312, in check_func self._check_compiler() File "c:\python27\lib\site-packages\numpy\distutils\command\config.py", li ne 39, in _check_compiler old_config._check_compiler(self) File "c:\python27\lib\distutils\command\config.py", line 102, in _check_co mpiler dry_run=self.dry_run, force=1) File "c:\python27\lib\site-packages\numpy\distutils\ccompiler.py", line 59 6, in new_compiler compiler = klass(None, dry_run, force) File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 96, in __init__ msvcr_success = build_msvcr_library() File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 360, in build_msvcr_library generate_def(dll_file, def_file) File "c:\python27\lib\site-packages\numpy\distutils\mingw32ccompiler.py", line 274, in generate_def raise ValueError("Symbol table not found") ValueError: Symbol table not found ---------------------------------------- Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\ \users\\22193\\appdata\\local\\temp\\pip-build-d3f_pb\\scipy\\setup.py';exec(com pile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __f ile__, 'exec'))" install --record c:\users\22193\appdata\local\temp\pip-wxyrfu-r ecord\install-record.txt --single-version-externally-managed --compile" failed w ith error code 1 in c:\users\22193\appdata\local\temp\pip-build-d3f_pb\scipy\ C:\Python27\Scripts> I realize this is due to some problem with resolving symbols in python27.dll. Is there a workaround for this? I need this to work on this setup as this is my workstation at office and I cannot install Ubuntu which would have been way easier. Things I have tried: I used to get an error while installing scipy for lapack/blas resources not found, I was able to get through by downloading and compiling them from source. I have tried to install scipy from http://www.lfd.uci.edu/~gohlke/pythonlibs/ and it does install scipy successfully but I get a 64-bit compatibility error when I try to import theano. Please help as I'm stuck here. Thanks Shreyank Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.

2 1

add elementwise addition & subtraction to einsum
by Matthew Harrigan 16 Oct '16

16 Oct '16

Hello, This is a follow on for issue 8139 <https://github.com/numpy/numpy/issues/8139>. I propose adding elementwise addition and subtraction functionality to einsum. I love einsum as it clearly and concisely defines complex linear algebra. However elementwise addition is a very common linear algebra operation and einsum does not currently support it. The Einstein field equations <https://en.wikipedia.org/wiki/Einstein_field_equations#Mathematical_form>, what the notation was originally developed to document, contain that functionality. It is fairly common in stress analysis (my background), for example see these lectures notes <http://www.brown.edu/Departments/Engineering/Courses/En221/Notes/Index_nota…> . Specifically I propose adding "+" and "-" characters which separate current einsum statements which are then combined elementwise. An example is A = np.einsum('ij,jk+ij,jk', B, C, D, E), which is A = B * C + D * E. I wrote a crude function <https://github.com/mattharrigan/numpy/blob/einsum-elementwise/numpy/core/ei…> to demonstrate the functionality. I believe the functionality is useful, in keeping with the spirit of a clean concise API, and doesn't break the existing API, which could warrant acceptance. Additionally I believe it opens the possibility of many interesting performance optimizations. For instance, many of the optimizations in this NEP <http://docs.scipy.org/doc/numpy/neps/deferred-ufunc-evaluation.html> could be done internally to the einsum function, which may be easier to accomplish given the narrower scope (but I am ignorant of all the low level C internals of numpy). The example in the beginning could become A = np.einsum('...+...+...', B, C, D). Thank you for your time and consideration. Matt

1 0

Integers to negative integer powers, time for a decision.
by Charles R Harris 10 Oct '16

10 Oct '16

Hi All, The time for NumPy 1.12.0 approaches and I like to have a final decision on the treatment of integers to negative integer powers with the `**` operator. The two alternatives looked to be *Raise an error for arrays and numpy scalars, including 1 and -1 to negative powers.* *Pluses* - Backward compatible - Allows common powers to be integer, e.g., arange(3)**2 - Consistent with inplace operators - Fixes current wrong behavior. - Preserves type *Minuses* - Integer overflow - Computational inconvenience - Inconsistent with Python integers *Always return a float * *Pluses* - Computational convenience *Minuses* - Loss of type - Possible backward incompatibilities - Not applicable to inplace operators Thoughts? Chuck

10 18

delete pixel from the raster image with specific range value
by Xristos Xristoou 09 Oct '16

09 Oct '16

any idea how to delete pixel from the raster image with specific range value using numpy/scipy or gdal? for example i have a raster image with the 5 class : 1. 0-100 2. 100-200 3. 200-300 4. 300-500 5. 500-1000 and i want to delete class 1 range value or maybe i want to delete class 1,2,4,5 if i need only class 3

2 1

Re: [Numpy-discussion] Integers to negative integer powers, time for a decision.
by Krisztián Horváth 08 Oct '16

08 Oct '16

Sorry, I was not clear enough. I meant that the second option (always float) would be more coherent with Python3. On Oct 8, 2016 9:36 PM, "Charles R Harris" <charlesr.harris(a)gmail.com> wrote: On Sat, Oct 8, 2016 at 1:31 PM, Krisztián Horváth <raksi.raksi(a)gmail.com> wrote: > Hello, > > I think it should be consistent with Python3. So, it should give back a > float. > > Best regards, > Krisztian > > Can't do that and also return integers for positive powers. It isn't possible to have behavior completely compatible with python for arrays: can't have mixed type returns, can't have arbitrary precision integers. Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

NumPy 1.11.2 released
by Charles R Harris 04 Oct '16

04 Oct '16

*Hi All,* I'm pleased to announce the release of Numpy 1.11.2. This release supports Python 2.6 - 2.7, and 3.2 - 3.5 and fixes bugs and regressions found in Numpy 1.11.1. Wheels for Linux, Windows, and OSX can be found on PyPI. Sources are available on both PyPI and Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.11.2/>. Thanks to all who were involved in this release. Contributors and merged pull requests are listed below. *Contributors to v1.11.2* - Allan Haldane - Bertrand Lefebvre - Charles Harris - Julian Taylor - Loïc Estève - Marshall Bockrath-Vandegrift + - Michael Seifert + - Pauli Virtanen - Ralf Gommers - Sebastian Berg - Shota Kawabuchi + - Thomas A Caswell - Valentin Valls + - Xavier Abellan Ecija + A total of 14 people contributed to this release. People with a "+" by their names contributed a patch for the first time. *Pull requests merged for v1.11.2* - #7736 <https://github.com/numpy/numpy/pull/7736>: Backport 4619, BUG: many functions silently drop keepdims kwarg - #7738 <https://github.com/numpy/numpy/pull/7738>: Backport 5706, ENH: add extra kwargs and update doc of many MA... - #7778 <https://github.com/numpy/numpy/pull/7778>: DOC: Update Numpy 1.11.1 release notes. - #7793 <https://github.com/numpy/numpy/pull/7793>: Backport 7515, BUG: MaskedArray.count treats negative axes incorrectly - #7816 <https://github.com/numpy/numpy/pull/7816>: Backport 7463, BUG: fix array too big error for wide dtypes. - #7821 <https://github.com/numpy/numpy/pull/7821>: Backport 7817, BUG: Make sure npy_mul_with_overflow_<type> detects... - #7824 <https://github.com/numpy/numpy/pull/7824>: Backport 7820, MAINT: Allocate fewer bytes for empty arrays. - #7847 <https://github.com/numpy/numpy/pull/7847>: Backport 7791, MAINT,DOC: Fix some imp module uses and update... - #7849 <https://github.com/numpy/numpy/pull/7849>: Backport 7848, MAINT: Fix remaining uses of deprecated Python... - #7851 <https://github.com/numpy/numpy/pull/7851>: Backport 7840, Fix ATLAS version detection - #7870 <https://github.com/numpy/numpy/pull/7870>: Backport 7853, BUG: Raise RuntimeError when reloading numpy is... - #7896 <https://github.com/numpy/numpy/pull/7896>: Backport 7894, BUG: construct ma.array from np.array which contains... - #7904 <https://github.com/numpy/numpy/pull/7904>: Backport 7903, BUG: fix float16 type not being called due to... - #7917 <https://github.com/numpy/numpy/pull/7917>: BUG: Production install of numpy should not require nose. - #7919 <https://github.com/numpy/numpy/pull/7919>: Backport 7908, BLD: Fixed MKL detection for recent versions of... - #7920 <https://github.com/numpy/numpy/pull/7920>: Backport #7911: BUG: fix for issue#7835 (ma.median of 1d) - #7932 <https://github.com/numpy/numpy/pull/7932>: Backport 7925, Monkey-patch _msvccompile.gen_lib_option like... - #7939 <https://github.com/numpy/numpy/pull/7939>: Backport 7931, BUG: Check for HAVE_LDOUBLE_DOUBLE_DOUBLE_LE in... - #7953 <https://github.com/numpy/numpy/pull/7953>: Backport 7937, BUG: Guard against buggy comparisons in generic... - #7954 <https://github.com/numpy/numpy/pull/7954>: Backport 7952, BUG: Use keyword arguments to initialize Extension... - #7955 <https://github.com/numpy/numpy/pull/7955>: Backport 7941, BUG: Make sure numpy globals keep identity after... - #7972 <https://github.com/numpy/numpy/pull/7972>: Backport 7963, BUG: MSVCCompiler grows 'lib' & 'include' env... - #7990 <https://github.com/numpy/numpy/pull/7990>: Backport 7977, DOC: Create 1.11.2 release notes. - #8005 <https://github.com/numpy/numpy/pull/8005>: Backport 7956, BLD: remove __NUMPY_SETUP__ from builtins at end... - #8007 <https://github.com/numpy/numpy/pull/8007>: Backport 8006, DOC: Update 1.11.2 release notes. - #8010 <https://github.com/numpy/numpy/pull/8010>: Backport 8008, MAINT: Remove leftover imp module imports. - #8012 <https://github.com/numpy/numpy/pull/8012>: Backport 8011, DOC: Update 1.11.2 release notes. - #8020 <https://github.com/numpy/numpy/pull/8020>: Backport 8018, BUG: Fixes return for np.ma.count if keepdims... - #8024 <https://github.com/numpy/numpy/pull/8024>: Backport 8016, BUG: Fix numpy.ma.median. - #8031 <https://github.com/numpy/numpy/pull/8031>: Backport 8030, BUG: fix np.ma.median with only one non-masked... - #8032 <https://github.com/numpy/numpy/pull/8032>: Backport 8028, DOC: Update 1.11.2 release notes. - #8044 <https://github.com/numpy/numpy/pull/8044>: Backport 8042, BUG: core: fix bug in NpyIter buffering with discontinuous... - #8046 <https://github.com/numpy/numpy/pull/8046>: Backport 8045, DOC: Update 1.11.2 release notes. Enjoy, Chuck

4 3

update on mailing list issues
by Ralf Gommers 04 Oct '16

04 Oct '16

Hi all, We've had a number of issues with the reliability of the mailman setup that powers the mailing lists for NumPy, SciPy and several other projects. To address that we'll start migrating to the python.org provided infrastructure, which should be much more reliable. The full set of lists is here: https://mail.scipy.org/mailman/listinfo. Looks like we have to migrate at least: AstroPy IPython-dev IPython-user NumPy-Discussion SciPy-Dev SciPy-User SciPy-organisers Some of the other ones that are not clearly obsolete but have almost zero activity (APUG, Nipy-devel) we'll have to contact the owners. *-tickets may be useful to archive. The other ones will just be cleaned up, unless someone indicates that there's a reason to keep them around. And a pre-emptive thanks to Didrik and Enthought for taking on the task of migrating the archives and user details. Cheers, Ralf

2 2

Dropping sourceforge for releases.
by Charles R Harris 04 Oct '16

04 Oct '16

Hi All, Ralf has suggested dropping sourceforge as a NumPy release site. There was discussion of doing that some time back but we have not yet done it. Now that we put wheels up on PyPI for all supported architectures source forge is not needed. I note that there are still some 15,000 downloads a week from the site, so it is still used. Thoughts? Chuck

3 3

ANN: pandas v0.19.0 released
by Joris Van den Bossche 03 Oct '16

03 Oct '16

Hi all, I'm happy to announce pandas 0.19.0 has been released. This is a major release from 0.18.1 and includes a number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. See the Whatsnew <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html> file for more information. We recommend that all users upgrade to this version. This is the work of 5 months of development by 117 contributors. A big thank you to all contributors! Joris --- *What is it:* pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *Highlights of the 0.19.0 release include:* - New method merge_asof for asof-style time-series joining, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - The .rolling() method is now time-series aware, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - read_csv now supports parsing Categorical data, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - A function union_categorical has been added for combining categoricals, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - PeriodIndex now has its own period dtype, and changed to be more consistent with other Index classes. See here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - Sparse data structures gained enhanced support of int and bool dtypes, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> - Comparison operations with Series no longer ignores the index, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> for an overview of the API changes. - Introduction of a pandas development API for utility functions, see here <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-…> . - Deprecation of Panel4D and PanelND. We recommend to represent these types of n-dimensional data with the xarray package <http://xarray.pydata.org/en/stable/>. - Removal of the previously deprecated modules pandas.io.data, pandas.io.wb, pandas.tools.rplot. See the Whatsnew <http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html> file for more information. *How to get it:* Source tarballs and windows/mac/linux wheels are available on PyPI (thanks to Christoph Gohlke for the windows wheels, and to Matthew Brett for setting up the mac/linux wheels). Conda packages are already available via the conda-forge channel (conda install pandas -c conda-forge). It will be available on the main channel shortly. *Issues:* Please report any issues on our issue tracker: https://github.com/pydata/pandas/issues *Thanks to all the contributors:* - adneu - Adrien Emery - agraboso - Alex Alekseyev - Alex Vig - Allen Riddell - Amol - Amol Agrawal - Andy R. Terrel - Anthonios Partheniou - babakkeyvani - Ben Kandel - Bob Baxley - Brett Rosen - c123w - Camilo Cota - Chris - chris-b1 - Chris Grinolds - Christian Hudon - Christopher C. Aycock - Chris Warth - cmazzullo - conquistador1492 - cr3 - Daniel Siladji - Douglas McNeil - Drewrey Lupton - dsm054 - Eduardo Blancas Reyes - Elliot Marsden - Evan Wright - Felix Marczinowski - Francis T. O’Donovan - Gábor Lipták - Geraint Duck - gfyoung - Giacomo Ferroni - Grant Roch - Haleemur Ali - harshul1610 - Hassan Shamim - iamsimha - Iulius Curt - Ivan Nazarov - jackieleng - Jeff Reback - Jeffrey Gerard - Jenn Olsen - Jim Crist - Joe Jevnik - John Evans - John Freeman - John Liekezer - Johnny Gill - John W. O’Brien - John Zwinck - Jordan Erenrich - Joris Van den Bossche - Josh Howes - Jozef Brandys - Kamil Sindi - Ka Wo Chen - Kerby Shedden - Kernc - Kevin Sheppard - Matthieu Brucher - Maximilian Roos - Michael Scherer - Mike Graham - Mortada Mehyar - mpuels - Muhammad Haseeb Tariq - Nate George - Neil Parley - Nicolas Bonnotte - OXPHOS - Pan Deng / Zora - Paul - Pauli Virtanen - Paul Mestemaker - Pawel Kordek - Pietro Battiston - pijucha - Piotr Jucha - priyankjain - Ravi Kumar Nimmi - Robert Gieseke - Robert Kern - Roger Thomas - Roy Keyes - Russell Smith - Sahil Dua - Sanjiv Lobo - Sašo Stanovnik - Shawn Heide - sinhrks - Sinhrks - Stephen Kappel - Steve Choi - Stewart Henderson - Sudarshan Konge - Thomas A Caswell - Tom Augspurger - Tom Bird - Uwe Hoffmann - wcwagner - WillAyd - Xiang Zhang - Yadunandan - Yaroslav Halchenko - YG-Riku - Yuichiro Kaneko - yui-knk - zhangjinjie - znmean - 颜发才（Yan Facai）

1 0

Re: [Numpy-discussion] automatically avoiding temporary arrays
by Hush Hush 03 Oct '16

03 Oct '16

The same idea was published two years ago: http://hiperfit.dk/pdf/Doubling.pdf On Mon, Oct 3, 2016 at 8:53 AM, <numpy-discussion-request(a)scipy.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: automatically avoiding temporary arrays (Benjamin Root) > 2. Re: Dropping sourceforge for releases. (David Cournapeau) > 3. Re: Dropping sourceforge for releases. (Vincent Davis) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 2 Oct 2016 09:10:45 -0400 > From: Benjamin Root <ben.v.root(a)gmail.com> > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Subject: Re: [Numpy-discussion] automatically avoiding temporary > arrays > Message-ID: > <CANNq6Fn9eGOnrSGz8Duo8J9oTe3N6xTCSqDDP7nRyNqjFKpAjQ@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Just thinking aloud, an idea I had recently takes a different approach. The > problem with temporaries isn't so much that they exists, but rather they > they keep on malloc'ed and cleared. What if numpy kept a small LRU cache of > weakref'ed temporaries? Whenever a new numpy array is requested, numpy > could see if there is already one in its cache of matching size and use it. > If you think about it, expressions that result in many temporaries would > quite likely have many of them being the same size in memory. > > Don't know how feasible it would be to implement though. > > Cheers! > Ben Root > > > On Sat, Oct 1, 2016 at 2:38 PM, Chris Barker <chris.barker(a)noaa.gov> > wrote: > > > Julian, > > > > This is really, really cool! > > > > I have been wanting something like this for years (over a decade? wow!), > > but always thought it would require hacking the interpreter to intercept > > operations. This is a really inspired idea, and could buy numpy a lot of > > performance. > > > > I'm afraid I can't say much about the implementation details -- but great > > work! > > > > -Chris > > > > > > > > > > On Fri, Sep 30, 2016 at 2:50 PM, Julian Taylor < > > jtaylor.debian(a)googlemail.com> wrote: > > > >> On 30.09.2016 23:09, josef.pktd(a)gmail.com wrote: > >> > On Fri, Sep 30, 2016 at 9:38 AM, Julian Taylor > >> > <jtaylor.debian(a)googlemail.com> wrote: > >> >> hi, > >> >> Temporary arrays generated in expressions are expensive as the imply > >> >> extra memory bandwidth which is the bottleneck in most numpy > >> operations. > >> >> For example: > >> >> > >> >> r = a + b + c > >> >> > >> >> creates the b + c temporary and then adds a to it. > >> >> This can be rewritten to be more efficient using inplace operations: > >> >> > >> >> r = b + c > >> >> r += a > >> > > >> > general question (I wouldn't understand the details even if I looked.) > >> > > >> > how is this affected by broadcasting and type promotion? > >> > > >> > Some of the main reasons that I don't like to use inplace operation in > >> > general is that I'm often not sure when type promotion occurs and when > >> > arrays expand during broadcasting. > >> > > >> > for example b + c is 1-D, a is 2-D, and r has the broadcasted shape. > >> > another case when I switch away from broadcasting is when b + c is int > >> > or bool and a is float. Thankfully, we get error messages for casting > >> > now. > >> > >> the temporary is only avoided when the casting follows the safe rule, so > >> it should be the same as what you get without inplace operations. E.g. > >> float32-temporary + float64 will not be converted to the unsafe float32 > >> += float64 which a normal inplace operations would allow. But > >> float64-temp + float32 is transformed. > >> > >> Currently the only broadcasting that will be transformed is temporary + > >> scalar value, otherwise it will only work on matching array sizes. > >> Though there is not really anything that prevents full broadcasting but > >> its not implemented yet in the PR. > >> > >> > > >> >> > >> >> This saves some memory bandwidth and can speedup the operation by 50% > >> >> for very large arrays or even more if the inplace operation allows it > >> to > >> >> be completed completely in the cpu cache. > >> > > >> > I didn't realize the difference can be so large. That would make > >> > streamlining some code worth the effort. > >> > > >> > Josef > >> > > >> > > >> >> > >> >> The problem is that inplace operations are a lot less readable so > they > >> >> are often only used in well optimized code. But due to pythons > >> >> refcounting semantics we can actually do some inplace conversions > >> >> transparently. > >> >> If an operand in python has a reference count of one it must be a > >> >> temporary so we can use it as the destination array. CPython itself > >> does > >> >> this optimization for string concatenations. > >> >> > >> >> In numpy we have the issue that we can be called from the C-API > >> directly > >> >> where the reference count may be one for other reasons. > >> >> To solve this we can check the backtrace until the python frame > >> >> evaluation function. If there are only numpy and python functions in > >> >> between that and our entry point we should be able to elide the > >> temporary. > >> >> > >> >> This PR implements this: > >> >> https://github.com/numpy/numpy/pull/7997 > >> >> > >> >> It currently only supports Linux with glibc (which has reliable > >> >> backtraces via unwinding) and maybe MacOS depending on how good their > >> >> backtrace is. On windows the backtrace APIs are different and I don't > >> >> know them but in theory it could also be done there. > >> >> > >> >> A problem is that checking the backtrace is quite expensive, so > should > >> >> only be enabled when the involved arrays are large enough for it to > be > >> >> worthwhile. In my testing this seems to be around 180-300KiB sized > >> >> arrays, basically where they start spilling out of the CPU L2 cache. > >> >> > >> >> I made a little crappy benchmark script to test this cutoff in this > >> branch: > >> >> https://github.com/juliantaylor/numpy/tree/elide-bench > >> >> > >> >> If you are interested you can run it with: > >> >> python setup.py build_ext -j 4 --inplace > >> >> ipython --profile=null check.ipy > >> >> > >> >> At the end it will plot the ratio between elided and non-elided > >> runtime. > >> >> It should get larger than one around 180KiB on most cpus. > >> >> > >> >> If no one points out some flaw in the approach, I'm hoping to get > this > >> >> into the next numpy version. > >> >> > >> >> cheers, > >> >> Julian > >> >> > >> >> > >> >> _______________________________________________ > >> >> NumPy-Discussion mailing list > >> >> NumPy-Discussion(a)scipy.org > >> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >> >> > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion(a)scipy.org > >> > https://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > >> > >> > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion(a)scipy.org > >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker(a)noaa.gov > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > >

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion October 2016