Mailman 3 February 2019 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Does numpy depend upon a Fortran library?
by Marc F Paterno 11 Apr '19

11 Apr '19

Hello, I have encountered a problem with a binary incompatibility between the Fortran runtime library installed with numpy when using 'pip install --user numpy', and that used by the rest of my program, which is built using gfortran from GCC 8.2. The numpy installation uses libgfortran.5.dylib, and GCC 8.2 provides libgfortran.5.dylib. While investigating the source of this problem, I downloaded the numpy source (https://files.pythonhosted.org/packages/04/b6/d7faa70a3e3eac39f943cc6a6a64c…), and tried building it. The resulting libraries have no coupling to any Fortran library that I can find. I can find no Fortran source code files in the numpy source, except in tests or documentation. I am working on a MacBook laptop, running macOS Mojave, and so am using the Accelerate framework to supply BLAS. I do not understand why the pip installation of numpy includes a Fortran runtime library. Can someone explain to me what I am missing? thanks Marc

5 6

Removing undocumented __buffer__ attribute lookup
by Matti Picus 28 Feb '19

28 Feb '19

In digging around the code, I found a gem in PyArray_FromBuffer (exposed to python as numpy.frombuffer). If a PyObject* does not have a tp_as_buffer->bf_getbuffer function, we check if the python object has a __buffer__ attribute. If so we use that as buf in PyObject_GetBuffer(buf, ...). This seems to stem back to the original numerics code, where getBuffer would look up the attribute and call it as a method. PyArray_FromBuffer does not call the attribute as a method, it simply passes it on to PyObject_GetBuffer, which will then raise an error saying it cannot convert a method. You can try this out by creating a class with a __buffer__ method and calling numpy.frombuffer on it. I submitted a pull request to remove the code. Since it is undocumented and (as far as I can tell) broken, I do not think we need a deprecation cycle. More details, including links to the original numeric code from 2005, in the PR https://github.com/numpy/numpy/pull/13049 Any thoughts or objections? Matti

3 2

[ANN] 12ᵗʰ Advanced Scientific Programming in Python in Camerino, Italy, 2—7 September, 2019
by Tiziano Zito 28 Feb '19

28 Feb '19

12ᵗʰ Advanced Scientific Programming in Python ============================================== a Summer School by the G-Node and the University of Camerino https://python.g-node.org Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 2–7 September, 2019. Camerino, Italy. Application =========== You can apply online: https://python.g-node.org/wiki/applications Application deadline: 23:59 UTC, 26 May, 2019. There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://python.g-node.org/wiki/faq Participation is for free, i.e. no fee is charged! Participants however should take care of travel, living, and accommodation expenses by themselves. Program ======= • Version control with git and how to contribute to open source projects with GitHub • Tidy data analysis and visualization • Testing and debugging scientific code • Advanced NumPy • Organizing, documenting, and distributing scientific code • Advanced scientific Python: context managers and generators • Writing parallel applications in Python • Profiling and speeding up scientific code with Cython and numba • Programming in teams Faculty ======= • Caterina Buizza, Personal Robotics Lab, Imperial College London, UK • Jenni Rinker, Department of Wind Energy, Technical University of Denmark, Roskilde, Denmark • Juan Nunez-Iglesias, Bioimage Analysis Research Fellow, Monash University, Australia • Nelle Varoquaux, Department of Statistics, UC Berkeley, CA, USA • Pamela Hathway, Neural Reckoning, Imperial College London, UK • Pietro Berkes, NAGRA Kudelski, Lausanne, Switzerland • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin, Germany • Stéfan van der Walt, Berkeley Institute for Data Science, UC Berkeley, CA, USA • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin, Germany Organizers ========== For the German Neuroinformatics Node of the INCF (G-Node), Germany: • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin, Germany • Caterina Buizza, Personal Robotics Lab, Imperial College London, UK • Zbigniew Jędrzejewski-Szmek, Red Hat Inc., Warsaw, Poland • Jakob Jordan, Department of Physiology, University of Bern, Switzerland For the University of Camerino, Italy: • Barbara Re, Computer Science Division, School of Science and Technology, University of Camerino Italy Website: https://python.g-node.org Contact: python-info(a)g-node.org

1 0

Outreachy internship
by Stefan van der Walt 28 Feb '19

28 Feb '19

Hi everyone, The team at BIDS would like to take on an intern from Outreachy (https://www.outreachy.org), as part of our effort to grow the NumPy developer community. The internship is similar to a GSoC, except that we will pay their salary, as well as do the mentoring. We wanted to check in beforehand to ensure that everyone is comfortable with us doing this on behalf of the NumPy community. Your thoughts and recommendations are welcome! Best regards, Stéfan

5 4

Reminder: weekly status meeting Feb 27 at 12:00 pacific time
by Matti Picus 26 Feb '19

26 Feb '19

The draft agenda is at https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. Past meeting notes can be seen at https://github.com/BIDS-numpy/docs/tree/master/status_meetings The BIDS team

1 0

NumPy 1.16.2 released.
by Charles R Harris 26 Feb '19

26 Feb '19

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.16.2. This is a quick release fixing several problems encountered on Windows. The Python versions supported are 2.7 and 3.5-3.7. The Windows problems addressed are: - DLL load problems for NumPy wheels on Windows, - distutils command line parsing on Windows. There is also a regression fix correcting signed zeros produced by divmod, see the release notes for details. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.4. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ``ImportError``. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 <https://github.com/numpy/numpy/issues/12736> for discussion of the issue. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.2/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.2>. *Contributors* A total of 5 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Eric Wieser * Matti Picus * Tyler Reddy * Tony LaTorre + *Pull requests merged* A total of 7 pull requests were merged for this release. * #12909: TST: fix vmImage dispatch in Azure * #12923: MAINT: remove complicated test of multiarray import failure mode * #13020: BUG: fix signed zero behavior in npy_divmod * #13026: MAINT: Add functions to parse shell-strings in the platform-native... * #13028: BUG: Fix regression in parsing of F90 and F77 environment variables * #13038: BUG: parse shell escaping in extra_compile_args and extra_link_args * #13041: BLD: Windows absolute path DLL loading Cheers, Charles Harris

1 0

Stepping down as NumFOCUS liason
by Nathaniel Smith 22 Feb '19

22 Feb '19

Hi all, As part of our relationship with NumFOCUS, we have to have some liasons so they know who to contact etc. So far that's been Ralf and me. But, I haven't had as much time for numpy recently, and I think it would be better to hand this off to someone who's more active. Is anyone interested? Someone on the steering council would probably be most natural. -n

3 2

Adding count parameter to unpackbits (PR-10855)
by Sebastian Berg 21 Feb '19

21 Feb '19

Hi all, I was about to merge it, but was noot sure it was really discussed before (and it would be a long time ago). The pull requests: https://github.com/numpy/numpy/pull/10855 Proposes to add a count parameter to unpackbits: ``count`` allows subsetting the number of bits that will be unpacked up-front, rather than reshaping and subsetting later, making the ``packbits`` operation invertible, and the unpacking less wasteful. Counts larger than the number of available bits add zero padding. Negative counts trim bits off the end instead of counting from the beginning. None counts implement the existing behavior of unpacking everything. This seems like a pretty small API addition, but I thought I would ping the list again before merging in case anyone objects to it or feels it should be modified. In other words, the count parameter is much the same as indexing the result: np.unpackbits(x)[:count] although it will zero pad if count is out of bounds. Which is necessary if it is to invert a `np.packbits` when less than 8 bits are packed. If no one speaks up, I plan to merge it soon. Best, Sebastian

1 0

Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN?
by Avi Gross 18 Feb '19

18 Feb '19

Changing horses mid-stream is not generally a good idea. But python is easy to extend so there may be other choices out there. The reality is that what matters more than what choices a language makes is consistency across the language. If you can learn the choices and even quirks and work with them, you can get things done. I have used languages where vectors or lists are zero-based and some where they are one-based. Both work, albeit you may need to adjust an algorithm to add or subtract one from the index you use or leave the first entry empty. If you use a pandas data structure without specifying your own index, you get stuck with the default behavior. If you wanted a 1 or 0 based index, you could add that carefully and so on. In theory, you could create an alternate numpy or alternate pandas just like many of the alternate modules you can find which often have very different designs for graphics and there probably already are some out there. Integrating them with other tools may not be as simple. If you want to use the machine learning tools available, some functions demand they be given the data as a pandas DataFrame or a numpy array. Giving them something else like a python list, even if it is just an object extended from the above, may well break things. This makes it very hard to port some things to or from other languages with different design constraints as an algorithm that looks straightforward in one because it goes with the overall grain of the language may need major surgery to fit into the other and perhaps might best be rewritten using some other algorithm that fits better. A different discussion elsewhere about errors was instructive. Many programmers used to write complex code that checks for all kinds of errors or conditions. Some python programmers take an approach of expecting errors to be something you catch so why bother checking for things like an array index being non-existent. Well, if an error you expect in pandas is not caught, consider checking before using software that does not enforce it. And some "errors" are not errors. As mentioned, in R it is NOT a design error that using an NA value in a calculation silently propagates the NA. Throughout the language you must either check if there is an NA using the is.na() function or by telling routines to drop/ignore any NA as in sum(something, na.rm=TRUE). There are many design decisions you make this way. So, if your language does something you don't want when it uses an implicit index, fine. Give it an explicit index. If it does not properly check the range you provide for validity, do it yourself. And if the language is so unsuitable for your needs, you can often switch. In particular, there are now even ways you can start your code in one language like python or R and then shunt some data structures across a divide where the other language can do things another way and if needed, shunt it back and repeat. That may not work well for your application and, as noted, switching rides in midstream has its dangers. From: NumPy-Discussion <numpy-discussion-bounces+avigross=verizon.net(a)python.org> On Behalf Of Robert Kern Sent: Sunday, February 17, 2019 3:15 PM To: Discussion of Numerical Python <numpy-discussion(a)python.org> Subject: Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN? On Sat, Feb 16, 2019 at 8:42 PM C W <tmrsg11(a)gmail.com <mailto:tmrsg11@gmail.com> > wrote: Dan, No one here is trying to convince you to use Python. If you don't like it, don't use it. The problem is not me, it's the language. No need to take it out on me personally. I've used other languages, Python is lacking in this area. I'm being very frank here, just think about it. No one was taking it out on you personally. We're just stating that we're not interested in having a discussion about which semantics are best, much less convincing you that Python's choice is the right one. They have been that way for a long time, and the time for making those decisions is long past. We could not change them now if we wanted to. Empirically, I've been on these lists for about 20 years now, and I have not seen this pop up as a frequent issue causing bugs in real code, so I would submit that if there is a lack (or benefit) compared to other languages, it is small That said, Python is not alone here. Perl and Ruby have Python's semantics. R introduces NaNs but does not raise an error. Matlab and Julia do raise errors. -- Robert Kern

1 0

Re: [Numpy-discussion] [SciPy-User] Why slicing Pandas column and then subtract gives NaN?
by Mike C 17 Feb '19

17 Feb '19

Thanks a lot, Thomas. I don’t have index when I read in the data. I just want to slice two series to the same length, and subtract. That’s it! I also don’t what numpy methods wrapped within methods. They work, but hard do understand. How would you do it? In Matlab or R, it’s very simple, one line. ________________________________ From: SciPy-User <scipy-user-bounces+tmrsg11=gmail.com(a)python.org> on behalf of Thomas Kluyver <takowl(a)gmail.com> Sent: Thursday, February 14, 2019 4:54 PM To: SciPy Users List Cc: Discussion of Numerical Python Subject: Re: [SciPy-User] [Numpy-discussion] Why slicing Pandas column and then subtract gives NaN? Maybe it's useful to look a bit more at what pandas is doing and why. The 'index' on a series or dataframe labels each row - e.g. if your series is measuring total sales for each day, its index would be the dates. When you combine (e.g. subtract) two series, pandas automatically lines up the indices. So it will join up the numbers for February 14th, even if they're not in the same position in the data. In your example, you haven't specified an index, so pandas generates an integer index which doesn't really mean anything, and aligning on it doesn't do what you want. What are you trying to do? If Numpy does exactly what you want, then the answer might be to use Numpy. > Isn't Numpy built on top of Pandas? It's the other way round: pandas is built on Numpy. Pandas indices are an extra layer of functionality on top of what Numpy does. On Thu, 14 Feb 2019 at 20:22, C W <tmrsg11(a)gmail.com<mailto:tmrsg11@gmail.com>> wrote: Hi Paul, Thanks for your response! I did not find a Pandas list for users, only for developers. I'd love to be on there. result = a.subtract(b.shift()).dropna() This seems verbose, several layers of parenthesis follow by a dot method. I'm new to Python, I thought Python code would be pity and short. Is this what everyone will write? Thank you! On Wed, Feb 13, 2019 at 6:50 PM Paul Hobson <pmhobson(a)gmail.com<mailto:pmhobson@gmail.com>> wrote: This is more a question for the pandas list, but since i'm here i'll take a crack. * numpy aligns arrays by position. * pandas aligns by label. So what you did in pandas is roughly equivalent to the following: a = pandas.Series([85, 86, 87, 86], name='a').iloc[1:4].to_frame() b = pandas.Series([15, 72, 2, 3], name='b').iloc[0:3].to_frame() result = a.join(b,how='outer').assign(diff=lambda df: df['a'] - df['b']) print(result) a b diff 0 NaN 15.0 NaN 1 86.0 72.0 14.0 2 87.0 2.0 85.0 3 86.0 NaN NaN So what I think you want would be the following: a = pandas.Series([85, 86, 87, 86], name='a') b = pandas.Series([15, 72, 2, 3], name='b') result = a.subtract(b.shift()).dropna() print(result) 1 71.0 2 15.0 3 84.0 dtype: float64 On Wed, Feb 13, 2019 at 2:51 PM C W <tmrsg11(a)gmail.com<mailto:tmrsg11@gmail.com>> wrote: Dear list, I have the following to Pandas Series: a, b. I want to slice and then subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works in Numpy. Example 1: did not work >>>a = pd.Series([85, 86, 87, 86]) >>>b = pd.Series([15, 72, 2, 3]) >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN >>> type(a[1:4]) <class 'pandas.core.series.Series'> Example 2: worked If I use values() method, it's converted to a Numpy object. And it works! >>> a.values[1:4]-b.values[0:3] array([71, 15, 84]) >>> type(a.values[1:4]) <class 'numpy.ndarray'> What's the reason that Pandas in example 1 did not work? Isn't Numpy built on top of Pandas? So, why is everything ok in Numpy, but not in Pandas? Thanks in advance! _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)python.org<mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ SciPy-User mailing list SciPy-User(a)python.org<mailto:SciPy-User@python.org> https://mail.python.org/mailman/listinfo/scipy-user _______________________________________________ SciPy-User mailing list SciPy-User(a)python.org<mailto:SciPy-User@python.org> https://mail.python.org/mailman/listinfo/scipy-user

7 8

Reminder: weekly status meeting Tues Feb 19 12:00 PST
by Matti Picus 17 Feb '19

17 Feb '19

This week the status meeting will be on Tues Feb 19 at 12:00 PST, not at the usual time The draft agenda is at https://hackmd.io/TtqnUDvPQgaGeej7zR8seA?both Everyone is invited to join. Matti, Tyler and Stefan

1 0

Why slicing Pandas column and then subtract gives NaN?
by C W 14 Feb '19

14 Feb '19

Dear list, I have the following to Pandas Series: a, b. I want to slice and then subtract. Like this: a[1:4] - b[0:3]. Why does it give me NaN? But it works in Numpy. Example 1: did not work >>>a = pd.Series([85, 86, 87, 86]) >>>b = pd.Series([15, 72, 2, 3]) >>> a[1:4]-b[0:3] 0 NaN 1 14.0 2 85.0 3 NaN >>> type(a[1:4]) <class 'pandas.core.series.Series'> Example 2: worked If I use values() method, it's converted to a Numpy object. And it works! >>> a.values[1:4]-b.values[0:3] array([71, 15, 84]) >>> type(a.values[1:4]) <class 'numpy.ndarray'> What's the reason that Pandas in example 1 did not work? Isn't Numpy built on top of Pandas? So, why is everything ok in Numpy, but not in Pandas? Thanks in advance!

2 2

'nansqrt' function?
by Mauro Cavalcanti 14 Feb '19

14 Feb '19

Dear ALL, In the process of porting an existing (but abandoned) package to the latest version of Numpy, I stumbled upon a call to a 'numpy.nansqrt' function, which seems not to exist. Here is the specific code: def normTrans(y): denom = np.nansqrt(np.nansum(y**2)) return y/denom As far as I could find, there is no such 'nansqrt' function in the current version of Numpy, so I suspect that the above code has not been properly tested. Am I right, or that function had existed in some past version of Numpy? Thanks in advance for any comments or suggestions. Best regards,

4 9

Porting code for Numpy 1.13+ to get rid of "boolean index did not match indexed array along dimension 1" error
by Mauro Cavalcanti 13 Feb '19

13 Feb '19

Dear ALL, I am trying to port an eigenalysis function that runs smoothly on Numpy 1.12 but fail miserably on Numpy 1.13 or higher with the dreadful error "boolean index did not match indexed array along dimension 1". Here is a fragment of the code, where the error occurrs: evals, evecs = np.linalg.eig(Syy) idx = evals.argsort()[::-1] evals = np.real(evals[idx]) U = np.real(evecs[:, idx]) evals = evals[evals > tolerance] U = U[:, evals > tolerance] # Here is where the error occurs So, I ask: is there a way out of this? Thanks in advance for any assistance you can provide.

2 2

Reminder: weekly status meeting Feb. 13 at 12:00 Pacific Time
by Tyler Reddy 12 Feb '19

12 Feb '19

Draft agenda: https://hackmd.io/f_e6dnssTkuIC0Pa0TEV6w?view There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. BIDS team

1 0

ANN: SciPy 1.2.1
by Tyler Reddy 09 Feb '19

09 Feb '19

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.2.1, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.2.1 One of a few ways to install this release with pip: pip install scipy==1.2.1 ========================== SciPy 1.2.1 Release Notes ========================== SciPy 1.2.1 is a bug-fix release with no new features compared to 1.2.0. Most importantly, it solves the issue that 1.2.0 cannot be installed from source on Python 2.7 because of non-ASCII character issues. It is also notable that SciPy 1.2.1 wheels were built with OpenBLAS 0.3.5.dev, which may alleviate some linear algebra issues observed in SciPy 1.2.0. Authors ======= * Eric Larson * Mark Mikofski * Evgeni Burovski * Ralf Gommers * Eric Moore * Tyler Reddy Issues closed for 1.2.1 ------------------------ * `#9606 <https://github.com/scipy/scipy/issues/9606>`__: SyntaxError: Non-ASCII character '\xe2' in file scipy/stats/_continuous_distns.py on line 3346, but no encoding declared * `#9608 <https://github.com/scipy/scipy/issues/9608>`__: Version 1.2.0 introduces `too many indices for array` error in... * `#9709 <https://github.com/scipy/scipy/issues/9709>`__: scipy.stats.gaussian_kde normalizes the weights keyword argument... * `#9733 <https://github.com/scipy/scipy/issues/9733>`__: scipy.linalg.qr_update gives NaN result * `#9724 <https://github.com/scipy/scipy/issues/9724>`__: CI: Is scipy.scipy Windows Python36-32bit-full working? Pull requests for 1.2.1 ------------------------ * `#9612 <https://github.com/scipy/scipy/pull/9612>`__: BUG: don't use array newton unless size is greater than 1 * `#9615 <https://github.com/scipy/scipy/pull/9615>`__: ENH: Add test for encoding * `#9720 <https://github.com/scipy/scipy/pull/9720>`__: BUG: stats: weighted KDE does not modify the weights array * `#9739 <https://github.com/scipy/scipy/pull/9739>`__: BUG: qr_updates fails if u is exactly in span Q * `#9725 <https://github.com/scipy/scipy/pull/9725>`__: TST: pin mingw for Azure Win CI * `#9736 <https://github.com/scipy/scipy/pull/9736>`__: TST: adjust to vmImage dispatch in Azure * `#9681 <https://github.com/scipy/scipy/pull/9681>`__: BUG: Fix failing stats tests (partial backport) * `#9662 <https://github.com/scipy/scipy/pull/9662>`__: TST: interpolate: avoid pytest deprecations Checksums ========= MD5 ~~~ 982810997da9daab2e512a6c27918889 scipy-1.2.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 9500cad49b7eac9786c90cba146ad357 scipy-1.2.1-cp27-cp27m-manylinux1_i686.whl fe9ba7f16e0e7f7c9cd59d39ea8e9545 scipy-1.2.1-cp27-cp27m-manylinux1_x86_64.whl f1823b26b2afda2027f78df427791700 scipy-1.2.1-cp27-cp27m-win32.whl c0692c60b4baaafd99fb2bf0c689bbf1 scipy-1.2.1-cp27-cp27m-win_amd64.whl b5a5b9b146bac064a7921c5121d5e19c scipy-1.2.1-cp27-cp27mu-manylinux1_i686.whl fd637f4dc4308ae9cf7655e3e5f3a27d scipy-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl 8faae90250009230cd2f7845ba4735f2 scipy-1.2.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl a28a93cb20987666f788d43c50e2ab79 scipy-1.2.1-cp34-cp34m-manylinux1_i686.whl 26264ebb568728065e3ef73b5419793a scipy-1.2.1-cp34-cp34m-manylinux1_x86_64.whl fbb79da3d5823ba0f23b0f8d50023bfd scipy-1.2.1-cp34-cp34m-win32.whl f82cf6834e4e2a2e4008fec5da2f4b0b scipy-1.2.1-cp34-cp34m-win_amd64.whl cc248fd969b11589d98bcb550330e08d scipy-1.2.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 3b5e8eb89a49fb7bc506cf17a54b926b scipy-1.2.1-cp35-cp35m-manylinux1_i686.whl 0cd66785fd9f8390494f42beb499f56a scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl 379d37ae932af2096e037b4defd4cf60 scipy-1.2.1-cp35-cp35m-win32.whl 5dd4c47c501c96dc0c20b72521058133 scipy-1.2.1-cp35-cp35m-win_amd64.whl 6fa013443295e55c96eb174578dcffee scipy-1.2.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 5069317ab38e3dfbb8923341c2b23a69 scipy-1.2.1-cp36-cp36m-manylinux1_i686.whl 8e6a1fd1354428074d8457fe46cd4f7c scipy-1.2.1-cp36-cp36m-manylinux1_x86_64.whl 27142bff898a18368032165c27c11466 scipy-1.2.1-cp36-cp36m-win32.whl 489fd885aefd07bded69a8e0452c16b0 scipy-1.2.1-cp36-cp36m-win_amd64.whl 84ef33013d9871f23db0ecad25552636 scipy-1.2.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 7b7ae846b70e60188b3d772e9d60d3d2 scipy-1.2.1-cp37-cp37m-manylinux1_i686.whl 21b29788cd2c037366d2ff5d54ae7517 scipy-1.2.1-cp37-cp37m-manylinux1_x86_64.whl e11ce933d012481ea27fe49314b606aa scipy-1.2.1-cp37-cp37m-win32.whl 23424815af8e69f945bb406a0c1be6c6 scipy-1.2.1-cp37-cp37m-win_amd64.whl fe88877767563e50078df86dadf7b1f4 scipy-1.2.1.tar.gz 312110f3ea37269003e587207eb72855 scipy-1.2.1.tar.xz b4e2b70f7885a746618e8bcdf77857d7 scipy-1.2.1.zip SHA256 ~~~~~~ f06819b028b8ef9010281e74c59cb35483933583043091ed6b261bb1540f11cc scipy-1.2.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 70e37cec0ac0fe95c85b74ca4e0620169590fd5d3f44765f3c3a532cedb0e5fd scipy-1.2.1-cp27-cp27m-manylinux1_i686.whl 9726791484f08e394af0b59eb80489ad94d0a53bbb58ab1837dcad4d58489863 scipy-1.2.1-cp27-cp27m-manylinux1_x86_64.whl 790cbd3c8d09f3a6d9c47c4558841e25bac34eb7a0864a9def8f26be0b8706af scipy-1.2.1-cp27-cp27m-win32.whl 014cb900c003b5ac81a53f2403294e8ecf37aedc315b59a6b9370dce0aa7627a scipy-1.2.1-cp27-cp27m-win_amd64.whl 78e12972e144da47326958ac40c2bd1c1cca908edc8b01c26a36f9ffd3dce466 scipy-1.2.1-cp27-cp27mu-manylinux1_i686.whl f15f2d60a11c306de7700ee9f65df7e9e463848dbea9c8051e293b704038da60 scipy-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl e2cfcbab37c082a5087aba5ff00209999053260441caadd4f0e8f4c2d6b72088 scipy-1.2.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl b86ae13c597fca087cb8c193870507c8916cefb21e52e1897da320b5a35075e5 scipy-1.2.1-cp34-cp34m-manylinux1_i686.whl 865afedf35aaef6df6344bee0de391ee5e99d6e802950a237f9fb9b13e441f91 scipy-1.2.1-cp34-cp34m-manylinux1_x86_64.whl e742f1f5dcaf222e8471c37ee3d1fd561568a16bb52e031c25674ff1cf9702d5 scipy-1.2.1-cp34-cp34m-win32.whl f6b88c8d302c3dac8dff7766955e38d670c82e0d79edfc7eae47d6bb2c186594 scipy-1.2.1-cp34-cp34m-win_amd64.whl 7274735fb6fb5d67d3789ddec2cd53ed6362539b41aa6cc0d33a06c003aaa390 scipy-1.2.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 963815c226b29b0176d5e3d37fc9de46e2778ce4636a5a7af11a48122ef2577c scipy-1.2.1-cp35-cp35m-manylinux1_i686.whl 281a34da34a5e0de42d26aed692ab710141cad9d5d218b20643a9cb538ace976 scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl 5a10661accd36b6e2e8855addcf3d675d6222006a15795420a39c040362def66 scipy-1.2.1-cp35-cp35m-win32.whl def0e5d681dd3eb562b059d355ae8bebe27f5cc455ab7c2b6655586b63d3a8ea scipy-1.2.1-cp35-cp35m-win_amd64.whl 79792c8fe8e9d06ebc50fe23266522c8c89f20aa94ac8e80472917ecdce1e5ba scipy-1.2.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 6dcc43a88e25b815c2dea1c6fac7339779fc988f5df8396e1de01610604a7c38 scipy-1.2.1-cp36-cp36m-manylinux1_i686.whl f31338ee269d201abe76083a990905473987371ff6f3fdb76a3f9073a361cf37 scipy-1.2.1-cp36-cp36m-manylinux1_x86_64.whl 9de84a71bb7979aa8c089c4fb0ea0e2ed3917df3fb2a287a41aaea54bbad7f5d scipy-1.2.1-cp36-cp36m-win32.whl b2c324ddc5d6dbd3f13680ad16a29425841876a84a1de23a984236d1afff4fa6 scipy-1.2.1-cp36-cp36m-win_amd64.whl 588f9cc4bfab04c45fbd19c1354b5ade377a8124d6151d511c83730a9b6b2338 scipy-1.2.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 628f60be272512ca1123524969649a8cb5ae8b31cca349f7c6f8903daf9034d7 scipy-1.2.1-cp37-cp37m-manylinux1_i686.whl 870fd401ec7b64a895cff8e206ee16569158db00254b2f7157b4c9a5db72c722 scipy-1.2.1-cp37-cp37m-manylinux1_x86_64.whl d78702af4102a3a4e23bb7372cec283e78f32f5573d92091aa6aaba870370fe1 scipy-1.2.1-cp37-cp37m-win32.whl ba0488d4dbba2af5bf9596b849873102d612e49a118c512d9d302ceafa36e01a scipy-1.2.1-cp37-cp37m-win_amd64.whl e085d1babcb419bbe58e2e805ac61924dac4ca45a07c9fa081144739e500aa3c scipy-1.2.1.tar.gz be0e1b322b4b9f2ab88a35db32c494f12639937cf671d4cc210238feb8d5a89d scipy-1.2.1.tar.xz 8140810cf19707c732f330ba6e54968bc85160768850bfbf38d995193e597e2e scipy-1.2.1.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJcXk5KAAoJELD/41ZX0J71zZ0P/3/6ZgXpYGEA013LKNh+/PrV YSA2G7b4jGYExim4cgFHDrwBPoFwiVcfdBTPMxSYn8mtvuqyn4TlcmxqGKQkOwoS zdOEUHdxPFLt8nAy5rWjiCYHzC4qCpGnt+3ROVzRjGA2BLSdzqBg+njT2+9FKNrH 5UoZP4s3TCA8Yn4z2bYV+2viHTAXFsEhHFraNuK5MQu045E57gt5VwYm+ylEeE3Q npcO2U8vio7Hr8ARVOyvQqL8HsQy5/8q8lGTBLbBAP+ujPXi3lRwdb2KqPl3ojfE 7afkEdC3W44CCxPoJfLOdwFnsaOUGVtFeNKmcS2YNWregMpjeG68AjjHO1vCeM34 twNXrwhO2gROwYnp6Q22TTiMsPTTUvce+qGmAyD1d4osOG5zz1A/YQWxHw8Runl0 LPrVt46aOUJTvjCAFs6c7McY04j6lILMyGZlxpyGwLEkP9YAABGvbOtHDMBBWvJs CFxLAo4gEWfJKcJ0bnAKQ9HNeJ45JJ3PQLD5eEmOl+TOQjl0nzY4c7D4LYB0wpl2 NY7tr/TcASX7TlwbjzmlccJUDmNhekcZVTbx65NFjmb2/nUgqAB9ZVjJKyQDpZng rLXZitBmd8IRb8xXp5Smbgt0VccwbMfJF4fulnrVokwILhsIbwZlYQDCG6KOPV27 mObh7S2ZazRQ76Gd4rcu =HBJ8 -----END PGP SIGNATURE-----

1 0

New sorting routines.
by Charles R Harris 08 Feb '19

08 Feb '19

Hi All, I've put up gh-12945 <https://github.com/numpy/numpy/pull/12945> that preserves forward compatibility of NumPy after the addition of timsort by reusing the mergesort slot in PyArray_ArrFuncs for stable sorts in general, not just mergesort. The same method can be used when we add radixsort for integer types. See the discussion at the PR link for more details on the nature of the problem and considerations for the dtype proposals. I'm waiting on agreement that this is the way to go before proceeding. Chuck

1 0

Reminder: weekly status meeting Feb. 6 at 12:00 Pacific Time
by Tyler Reddy 06 Feb '19

06 Feb '19

Draft agenda: https://hackmd.io/5AUINxNfS3iN0txIaN32HA?view There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. BIDS team

1 0

NumPy 1.16.1 release
by Charles R Harris 01 Feb '19

01 Feb '19

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.1. This release fixes bugs reported against the 1.16.0 release, and also backports several enhancements from master that seem appropriate for a release series that is the last to support Python 2.7. The supported Python versions are 2.7 and 3.5-3.7. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.4. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an *ImportError*. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 <https://github.com/numpy/numpy/issues/12736> for discussion of the issue. Note that previously this problem resulted in an *AttributeError*. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.1/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.1>. *Enhancements* * #12767: ENH: add mm->q floordiv * #12768: ENH: port np.core.overrides to C for speed * #12769: ENH: Add np.ctypeslib.as_ctypes_type(dtype), improve `np.ctypeslib.as_ctypes` * #12773: ENH: add "max difference" messages to np.testing.assert_array_equal... * #12820: ENH: Add mm->qm divmod * #12890: ENH: add _dtype_ctype to namespace for freeze analysis *Contributors* A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Antoine Pitrou * Arcesio Castaneda Medina + * Charles Harris * Chris Markiewicz + * Christoph Gohlke * Christopher J. Markiewicz + * Daniel Hrisca + * EelcoPeacs + * Eric Wieser * Kevin Sheppard * Matti Picus * OBATA Akio + * Ralf Gommers * Sebastian Berg * Stephan Hoyer * Tyler Reddy Cheers, Charles Harris

1 0