Mailman 3 September 2015 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How … [View More]

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program … [View More]makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil [View Less]

10 10

C-API: multidimensional array indexing?
by Johann Bauer March 31, 2016

March 31, 2016

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, … [View More]

3 3

Generalize hstack/vstack --> stack; Block matrices like in matlab
by Stefan Otte Jan. 9, 2016

Jan. 9, 2016

Hey, quite often I work with block matrices. Matlab offers the convenient notation [ a b; c d ] to stack matrices. The numpy equivalent is kinda clumsy: vstack([hstack([a,b]), hstack([c,d])]) I wrote the little function `stack` that does exactly that: stack([[a, b], [c, d]]) In my case `stack` replaced `hstack` and `vstack` almost completely. If you're interested in including it in numpy I created a pull request [1]. I'm looking forward to getting some feedback! Best, Stefan … [View More]

12 24

future of f2py and Fortran90+
by Eric Firing Dec. 4, 2015

Dec. 4, 2015

F2py is a great tool, but my impression is that it is being left behind by the evolution of Fortran from F90 onward. This is unfortunate; it would be nice to be able to easily wrap new Fortran libraries. I'm curious: has anyone been looking into what it would take to enable f2py to handle modern Fortran in general? And into prospects for getting such an effort funded? Eric

5 6

Cython-based OpenMP-accelerated quartic polynomial solver
by Juha Jeronen Oct. 6, 2015

Oct. 6, 2015

Hi all, I recently developed a Cython-based, OpenMP-accelerated quartic (and cubic, quadratic) polynomial solver to address a personal research need for quickly solving a very large number of independent low-degree polynomial equations for both real and complex coefficients. For example, on an Intel i7-4710MQ, running with 8 threads this code solves approximately 1.2e7 quartic polynomial equations per second. (With OMP_NUM_THREADS=4 the solver gives approximately 8.9e6 solutions per … [View More]second, so it seems HyperThreading in this case helps about 30%.) The algorithms for cubics and quadratics come from Numerical Recipes (3rd ed.), and the quartic problem is internally reduced to a cubic and two quadratics, using well-known standard tricks. Since to my understanding, for solving polynomial equations NumPy only provides roots(), which works one problem at a time, I'm writing to inquire whether there would be interest to include this functionality to NumPy, if I contribute the code (and clean it up for integration)? I have some previous open source contribution experience. I have authored the IVTC and Phosphor deinterlacers for VLC, and a modular postprocessing filter framework for the Panda3D game engine (posted at the forums on panda3d.org, projected for version 1.10). Currently the polynomial solver is available in a git repository hosted by our university, the relevant files being: https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworod… I'm working on a research grant, which allows me to hold the copyright; license is BSD. Polysolve2 is the fast Cython+OpenMP version, while polysolve is an earlier (slower) experimental version using only NumPy array operations. The example module contains a usage example, and setup (in the standard manner) instructs distutils and Cython as to how to build polysolve2 (including setting the relevant math flags and enabling OpenMP). (The setup script includes also some stuff specific to my solver for the Ziegler problem; basically, the "tworods" module can be ignored. I apologize for the inconvenience.) Thoughts? Best regards, -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen(a)jyu.fi University of Jyväskylä Department of Mathematical Information Technology ------------------------------------------------- [View Less]

12 39

Sign of NaN
by Charles R Harris Sept. 30, 2015

Sept. 30, 2015

Hi All, Due to a recent commit, Numpy master now raises an error when applying the sign function to an object array containing NaN. Other options may be preferable, returning NaN for instance, so I would like to open the topic for discussion on the list. Thoughts? Chuck

12 22

composition of the steering council (was Re: Governance model request)
by Nathaniel Smith Sept. 30, 2015

Sept. 30, 2015

[splitting this out from the thread-o-doom] On Wed, Sep 23, 2015 at 8:47 AM, Chris Barker - NOAA Federal <chris.barker(a)noaa.gov> wrote: [snip] > So here is what I think is on the table: > > We have the steering council. Which leaves two questions: > > -How big should it be? > -Who will be on the original, "seed" council? > > Note that as I understand the current draft of the governance doc, > once established, the council itself decides who is on the council. … [View More]So > at this point we really are ONLY deciding how it's going to start. It > has to be bootstrapped somehow. > > However, that had been contentious enough that it would probably be a > good idea to hash out some guidelines about the council membership. We actually do have some of those already -- dunno if they're perfect, but they exist :-). To make sure everyone's on the same page, here's a condensed summary of what the draft currently says: Joining the council: contributor must have produced "substantial" and "sustained" contributions over at least one year + be voted on by the current council + be interested and willing to serve. (Then there's some language emphasising that "contributions" should *not* be read narrowly as a euphemism for "lines of code".) Leaving the council: Happens by choice of member, or if inactive for one year and can't be contacted, or if inactive for two years. Former members are listed as "emeritus" to recognize their past service. Rejoining the council: aside from their entry on the list of emeritus members, a former-member and a never-member are treated identically in general, and the rules for re-joining are the same as the rules for joining. Proposal for seed council: "everyone who's merged a pull request since Jan 1, 2014". (Actual text is here: http://thread.gmane.org/gmane.comp.python.numeric.general/61106 , see section "Council membership".) We didn't talk much about these -- I think mostly on the theory that the exact details really aren't going to matter much in the end. These specific rules are exactly the rules that Jupyter/IPython use, stolen without changes. My interpretation is that these rules were designed to produce a council consisting of a broad spectrum of contributors who are actively engaged, in tune and up-to-date with the issues currently facing the project, and broadly respected by the broader community. The rationale for doing things this way (if we keep it) would be that the steering council's "primary responsibility is to facilitate the ordinary community-based decision making procedure", so you need people who are actively engaged in community discussions; and, if things break down then the people best positioned to resolve it are the ones who have the best view of what went wrong, understand the personalities involved, and so forth. In practice, I'm sure any council interventions would involve most members deferring to whoever they judge has the most expertise, whether or not that person is on the council -- it's not like they'll ever be making a decision in a vacuum. Regarding the seed council, I just tried to pick an objective criterion and an arbitrary date that seemed generally in keeping with idea of "should be active in the last 1-to-2-years-ish". Fiddling with the exact date in particular makes very little difference -- between pushing it back to 2 years ago today or forward to 1 year ago today, the only thing that changes is whether Pauli makes the list or not. (And Pauli is obviously a great council candidate, though I don't know whether he even wants to be on it.) > Personally, I have no idea how big the council should be. Too big, and > there is no point, consensus is harder to reach the larger the group, > and the main (only?) role of the council is to resolve issues where > consensus has not been reached in the larger community. But what is > too big? I'm a little wary of the idea of capping the council size. Assuming you're pre-filtering for basic competence and good faith (as we are), then the only way making the council smaller helps with decision making is that it arbitrarily throws away the opinion of some dedicated and valuable contributors. Plus then we have to start making judgements like "well, person A has been around for a while but pretty inactive recently, and person B is doing awesome stuff, should we kick person A off the council to let person B on or...?" Judging whether someone is or isn't a "substantial contributor" is fine, we can do that. Having to make a relative judgement of which of two people is the *more* "substantial contributor", though -- that sounds awful. And given how conflict-adverse groups can be, I suspect that capping the council size would in practice just have the effect that it never takes in new blood. (The old effect where "science advances one retirement at a time".) I'd be interested to hear what Jupyter/IPython's experience with this is, though: they currently have a 10 (!) person steering council, I assume they'd love to have more. Having lots of contributors who are active and engaged enough to meet the steering council qualifications is a good problem to have :-). Technically their situation is slightly different because their council runs on regular voting rather than Apache-style consensus voting (a luxury they can afford because they have a BDFL to step in if regular voting ends up disenfranchising the minority), but I sort of get the impression that in practice they just don't vote unless they know it will be unanimous, and it's worked out for them so far. (To understate the case.) > As for make-up of the council, I think we need to expand beyond people > who have recently contributed core code. > > Yes, the council does need to have expertise to make technical > decisions, but if you think about the likely contentious issues like > ABI breakage, a core-code focused view is incomplete. So there should > be representation by: > > Someone(s) with a long history of working with the code -- that > institutional memory of why decisions were made the way they were > could be key. Sure -- though I can't really imagine any way of framing a rule like this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess is that such a rule would not actually have any effect on the council membership in practice. > Someone(s) that may not have worked on the core code, but is a major > player in the community, perhaps as the leader of a Numpy-dependent > project. This would provide representation for the broad community. Pointing out features of the current draft again for reference: In the current text, the "NumFOCUS subcommittee" does have an external member to provide some oversight. (So mathematically speaking, this means that the subcommittee is not a subset -- go figure. I blame IPython.) But, this oversight is specifically for financial matters only, not technical ones: "This Subcommittee shall NOT make decisions about the direction, scope or technical direction of the Project." Thomas Caswell (one of the leaders of matplotlib) volunteered to be our external member to start. We certainly could ask him to sit on the steering council as well, but honestly my guess is that this would have no effect, either positive or negative. (I know if someone asked me to serve on a hypothetical matplotlib steering council, then I would just... never do anything, because who am I to second-guess matplotlib's actual developers.) It's not like we don't already hear from downstream projects on a regular basis. And if things have gone *so* pear-shaped that we have a situation where the steering council feels they need to issue a ruling, *and simultaneously* the members of the council are so out-of-touch that they don't know or care about the needs of downstream projects, then the situation is unsalvageable and we should fork and start over. But I mean, it probably wouldn't hurt either. I'm not super-wedded to the current text. I just think we should limit how much effort we spend trying to cover every possible situation ahead of time. If we're clever enough to solve a problem now hypothetically, then we're probably also clever enough to solve it later when it becomes concrete. -n -- Nathaniel J. Smith -- http://vorpus.org [View Less]

9 23

Cython-based OpenMP-accelerated quartic, polynomial solver
by Matti Picus Sept. 29, 2015

Sept. 29, 2015

2 1

Numpy 1.10.0rc2 released
by Charles R Harris Sept. 29, 2015

Sept. 29, 2015

Hi all, I'm pleased to announce the availability of Numpy 1.10.0rc12. Sources and 32 bit binary packages for Windows may be found at Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.10.0rc2/?upload_just_c…>. There have been a few fixes since rc1. If there are no more problems I hope to release the final in a week or so. Cheers Chuck

1 0

New version of "scipy lecture notes"
by Gael Varoquaux Sept. 28, 2015

Sept. 28, 2015

Dear Pythonistas, We have just released a new version of the "scipy lecture notes": http://www.scipy-lectures.org/ These are a consistent set of materials to learn the core aspects of the scientific Python ecosystem, from beginner to expert. They are written and maintained by a set of volunteers and published under a CC-BY license. Highlights of the latest version includes: * a chapter giving a introduction to statistics in Python * a new layout with emphasis on readability including on … [View More]

1 0

Including .f2py_f2cmap in numpy.distutils?
by Eric Hermes Sept. 28, 2015

Sept. 28, 2015

Hello, I am attempting to set up a numpy.distutils setup.py for a small python program that uses a Fortran module. Currently, the setup is able to compile and install the program seemingly successfully, but the f2py script erroneously maps the data types I am using to float, rather than double. I have the proper mapping set up in a .f2py_f2cmap in the source directory, but it does not seem to be copied to the build directory at compile time, and I cannot figure out how to make it get copied. … [View More]

2 1

Fixing the dtype of np.full's return value
by Antony Lee Sept. 27, 2015

Sept. 27, 2015

Hi all, The docstring of np.full indicates that the result of the dtype is `np.array(fill_value).dtype`, as long as the keyword argument `dtype` itself is not set. This is actually not the case: the current implementation always returns a float array when `dtype` is not set, see e.g. In [1]: np.full(1, 1) Out[1]: array([ 1.]) In [2]: np.full(1, None) Out[2]: array([ nan]) In [3]: np.full(1, None).dtype Out[3]: dtype('float64') In [4]: np.array(None) Out[4]: array(None, dtype=object) The … [View More]

2 1

Numpy 1.10.0rc2 coming Monday, Sep 28.
by Charles R Harris Sept. 27, 2015

Sept. 27, 2015

Hi All, Just a heads up. If you have encountered any unreported errors with rc1, please let us know. Chuck

1 0

Python needs goto
by Charles R Harris Sept. 25, 2015

Sept. 25, 2015

At last, goto for python <https://github.com/snoack/python-goto>! Usage: from goto import with_goto @with_goto def range(start, stop): i = start result = [] label .begin if i == stop: goto .end result.append(i) i += 1 goto .begin label .end return result HT: LWN Chuck

10 18

"Numpy for Matlab Users" is missing
by Thomas Haslwanter Sept. 25, 2015

Sept. 25, 2015

Does anybody know what happened to the following links on the NumPy homepage (http://www.numpy.org/): - http://wiki.scipy.org/Wiki/NumPy_for_Matlab_Users - http://www.numpy.org/ - http://wiki.scipy.org/Numpy_Functions_by_Category They have not been available for some time now ...

2 1

ANN: Numpy 1.10.0rc1 released.
by Charles R Harris Sept. 24, 2015

Sept. 24, 2015

Hi all, I'm pleased to announce the availability of Numpy 1.10.0rc1. Sources and 32 bit binary packages for Windows may be found at Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.10.0rc1/?upload_just_c…>. Please test this out, as I would like to move to the final release as rapidly as possible and the lack of error reports from the beta has left me nervous. It's been quiet, too quiet. In the movies, we would all die in the next five minutes. Cheers Chuck

6 9

Re: [Numpy-discussion] Python needs goto
by R Schumacher Sept. 24, 2015

Sept. 24, 2015

An older non-decorator implementation, with examples (slower) http://entrian.com/goto/ At 04:25 AM 9/24/2015, you wrote: >Hello. > >Can you give an example where GOTO is useful ? >LeÂ 24 sept. 2015 07:24, "Charles R Harris" ><<mailto:charlesr.harris@gmail.com>charlesr.harris(a)gmail.com> a Ã©critÂ : >At last, <https://github.com/snoack/python-goto>goto for python! > >Usage: >from goto import with_goto > > >@with_goto > >def range(… [View More]

1 0

Re: [Numpy-discussion] "Become an Open Source Contributor" workshop
by Scott Collis Sept. 24, 2015

Sept. 24, 2015

Hey Jamie, List, Having just come back from a conference where our toolkit, Py-ART [1] has picked up a nice following of people keen to contribute I was wondering if you will be opening this up via a google hangout or similar? I would love to advertise this to our users. We all want more contributors and a big roadblock is understanding the fork and pull request system of GitHub We did run a course that had some GitGub etc here: https://github.com/scollis/SusSoPrac You are welcome to use … [View More]anything liberally! Cheers, Scott On 9/23/15 4:39 PM, numpy-discussion-request(a)scipy.org wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. "Become an Open Source Contributor" workshop > (Jaime Fern?ndez del R?o) > 2. Re: composition of the steering council (was Re: Governance > model request) (Travis Oliphant) > 3. Re: Governance model request (Stefan van der Walt) > 4. Re: Governance model request (Matthew Brett) > 5. Re: composition of the steering council (was Re: Governance > model request) (Chris Barker) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 23 Sep 2015 14:06:08 -0700 > From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)scipy.org>, Discussion of > Numerical Python <numpy-discussion(a)scipy.org> > Subject: [Numpy-discussion] "Become an Open Source Contributor" > workshop > Message-ID: > <CAPOWHWnk7mNm64_FuQkV5KCX=vxAyctBh7P1X7KbcaMgohSAOg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Apologies for the cross-posting. > > The Data Science Student Society of the University of California San Diego, > or DS3 @ UCSD as they like to call themselves, will be holding biweekly > Python themed workshops starting this fall. On the week of October 19th, > they will be having yours truly doing a "Become an Open Source Contributor" > piece. It will be a shortish event, 60-90 minutes, so my idea was to cover > the following: > > 1. (15 min) An introduction to the Python data science landscape. > 2. (30 min) An overview of the GitHub workflow that most (all?) of the > projects follow. > 3. (30-45 min) A hands on session, where we would make sure everyone > gets set up in GitHub, and forks and clones their favorite project. Time > and participant willingness permitting, I would like to take advantage of > my commit bits, and have some of the participants submit a simple PR, e.g. > fixing a documentation typo, to NumPy or SciPy, and hit the green button > right there, so that they get to leave as knighted FOSS contributors. > > And this is what I am hoping to get from you, the community: > > 1. If anyone in the area would like to get involved, please contact me. > I have recruited a couple of volunteers from PySanDiego, but could use more > help. > 2. I'm also hoping to get some help, especially with the introductory > part. Given that the crowd will mostly be university students and some > faculty, it would be great if someone who actually knew what they were > talking about could deliver a short, 10 minute talk, on Python, data > science, and academia. I'm sure we could arrange it to have someone join > by video conference. > 3. If you have organized anything similar in the past, and have material > that I could use to, ahem, draw inspiration from, or recommendations to > make, or whatever, I'd love to hear from you. > > Thanks for reading! > > Jaime > [View Less]

1 0

interpretation of the draft governance document (was Re: Governance model request)
by Nathaniel Smith Sept. 24, 2015

Sept. 24, 2015

Hi Travis, On Tue, Sep 22, 2015 at 3:08 AM, Travis Oliphant <travis(a)continuum.io> wrote: > > > On Tue, Sep 22, 2015 at 4:33 AM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> On Tue, Sep 22, 2015 at 1:24 AM, Travis Oliphant <travis(a)continuum.io> >> wrote: >>> >>> I actually do agree with your view of the steering council as being >>> usually not really being needed. You are creating a straw-man by >>> … [View More]indicating otherwise. I don't believe a small council should do anything >>> *except* resolve disputes that cannot be resolved without one. Like you, I >>> would expect that would almost never happen --- but I would argue that >>> extrapolating from Debian's experience is not actually relevant here. >> >> >> To be clear, Debian was only one example -- what I'm extrapolating from is >> every community-driven F/OSS project that I'm aware of. >> >> It's entirely possible my data set is incomplete -- if you have some other >> examples that you think would be better to extrapolate from, then I'd be >> genuinely glad to hear them. You may have noticed that I'm a bit of an >> enthusiast on this topic :-). >> > > > Yes, you are much better at that than I am. I'm not even sure where I > would look for this kind of data. > >>> >>> >>> >>> So, if the steering council is not really needed then why have it at all? >>> Let's just eliminate the concept entirely. >>> >> >> In my view, the reasons for having such a council are: >> 1) The framework is useful even if you never use it, because it means >> people can run "what if" scenarios in their mind and make decisions on that >> basis. In the US legal system, only a vanishingly small fraction of cases go >> to the Supreme Court -- but the rules governing the Supreme Court have a >> huge effect on all cases, because people can reason about what would happen >> *if* they tried to appeal to the Supreme Court. > > > O.K. That is a good point. I can see the value in that. > > >> >> 2) It provides a formal structure for interfacing with the outside world. >> E.g., one can't do anything with money or corporate contributions without >> having some kind of written-down and enforceable rules for making decisions >> (even if in practice you always stick to the "everyone is equal and we >> govern by consensus" part of the rules). > > > O.K. > >> >> 3) There are rare but important cases where discussions have to be had in >> private. The main one is "personnel decisions" like inviting people to join >> the council; another example Fernando has mentioned to me is that when they >> need to coordinate a press release between the project and a funding body, >> the steering council reviews the press release before it goes public. > > > O.K. > > >> >> That's pretty much it, IMO. >> >> The framework we all worked out at the dev meeting in Austin seems to >> handle these cases well AFAICT. > > > How did we "all" work it out when not everyone was there? This is where I > get lost. You talk about community decision making and yet any actual > decision is always a subset of the community. I suppose you just rely on > the "if nobody complains than it's o.k." rule? That really only works if > the project is moving slowly. By "all" I just meant "all of us who were there" (which was a majority of the active maintainers + a number of other interested parties -- the list of attendees is in the meeting notes if you're curious). In general I totally agree with your concern about only including a subset of the community. That's why we followed up by posting to the list a full set of notes on tentative-decisions-made, and the draft governance document in particular, for further discussion. We've already had multiple threads talking about it, even before this one. And it's pretty explicit in the document itself that no non-trivial decision can be considered final unless it's *at least* been posted on the mailing list. We didn't try to legislate the exact review requirements for every decision, because it's impossible to have a set of rules that scales from trivial typos (which just get merged, only github subscribers even know it happened) to foundational discussions like this one. This means that one of the things we trust contributors (esp. senior contributors) to do is to use their knowledge of the project to make judgement calls about how risky or controversial a given change will be, or if there's some particular expertise that should be consulted. (E.g. we might make sure to ping Robert Kern if there's some np.random change being discussed; I'm hesitant to finalize the PyUFunc ABI changes being discussed in the other thread until Ralf gets back, because I know that among the core maintainers he's particularly critical of the idea of breaking ABI.) And if we new information later comes to light then a decision can always be revisited -- people may get grumpy if you try to re-open an issue that's been considered settled for a year, but if you have a good reason and nothing irrevocable has happened (e.g. a veto obviously can't remove code from an existing release), then, well, it's annoying but what can you do, let's hear your reason. It's could certainly happen that sometimes the steering council + mailing list readers will all miss something important. But this is unavoidable in any system -- we're obviously not going to, like, institute a one month waiting period on every single decision or something. Ultimately you have to trust the core maintainers to have good judgement about which changes to accept, together with good meta-judgement about how controversial or broad-reaching any given change is likely to be, and then hope that the rest of the community will also supplement as they can. >>> But there are real questions that have to have an answer or an approach >>> to making a decision. The answer to these questions cannot really be a >>> vague notion of "lack of vigorous opposition by people who read the mailing >>> list" which then gets parried about as "the community decided this." The >>> NumPy user base is far, far larger than the number of people that read this >>> list. >> >> >> According to the dev meeting rules, no particularly "vigorous opposition" >> is required -- anyone who notices that something bad is happening can write >> a single email and stop an idea dead in its tracks, with only the steering >> council able to overrule. We expect this will rarely if ever happen, because >> the threat will be enough to keep everyone honest and listening, but about >> the only way we could possibly be *more* democratic is if we started phoning >> up random users at home to ask their opinion. > > > O.K. so how long is the time allowed for this kind of opposition to be > noted? See above. For regular discussions, there are some rough guidelines (uncontroversial bug fixes can just be merged; substantive API changes need at least a few days review on the mailing list). This governance discussion has been left open for a few weeks, and: "worst case, if a change is more controversial than expected, or a crucial critique is delayed because someone was on vacation, then it's no big deal: we apologize for misjudging the situation, [back up, and sort things out](http://producingoss.com/en/producingoss.html#version-control-relaxatio…." (I think we all thought the governance discussion was done, actually. But then you posted, and so now we're talking about it some more. No worries; if there's an issue, we'd rather know, right?) For formal council votes (the ones we expect will rarely if every happen), then we do have a slightly more formal rule: that the vote "should be held open for long enough to give all interested Council Members a chance to respond -- at least one week." The words "at least" are in there to emphasize that the goal is to get an honest read of the council; e.g. it's not legitimate to play games by scheduling a vote when you know someone is on vacation. >> >> This is actually explicitly designed to prevent the situation where >> whoever talks the loudest and longest wins, and to put those with more and >> less time available on an equal footing. >> >>> >>> For better or for worse, we will always be subject to the "tyranny of who >>> has time to contribute lately". Fundamentally, I would argue that this >>> kind of "tyranny" should at least be tempered by additional considerations >>> from long-time contributors who may also be acting more indirectly than is >>> measured by a simple git log. >> >> >> I guess I am missing something fundamental here. Who are these long-time >> contributors who will sit on your council of 1/3/5 but who don't even read >> the mailing list? How will they know when their special insight is >> necessary? > > > The long-time contributors wouldn't necessarily sit on that council. But, > I would support the idea of an advisory council that could act if it saw > things going the wrong direction. This is where those people would sit. In the draft governance document, anyone who cares enough to pay attention effectively has a seat on this advisory council. I assume this is a superset of the people that you would nominate? > In the case of a 1 / 3 / 5 member council -- I would not argue to be on it > at all (but I would argue that some care should be taken to be sure it has > people with some years of experience if they are available). I'm only > asking to be on a steering council that is larger than 5 people, and I don't > actually prefer that the steering council be larger than 5 people. > >> >> No, absolutely not. The proposal is that these issues are decided by open >> discussion on the mailing list. (With the possible exception of #4 -- I >> suspect that given an extreme situation like this, then once all other >> efforts to mitigate the situation had failed the steering council would >> probably feel compelled to talk things over to double-check they had >> consensus before acting, and likely some part of this would have to be done >> in private.) > > > O.K. Then, I am misunderstanding. Oh good, I'm glad when things turn out to be misunderstandings, because those are (relatively) easy to solve :-). >> >> This is pretty explicit in the document (and again, this is text and ideas >> stolen directly from Jupyter/IPython): >> >> "During the everyday project activities, council members participate in >> all discussions, code review and other project activities as peers with all >> other Contributors and the Community. In these everyday activities, Council >> Members do not have any special power or privilege through their membership >> on the Council. [...] the Council may, if necessary [do pretty much >> anything, but] the Council's primary responsibility is to facilitate the >> ordinary community-based decision making procedure described above. If we >> ever have to step in and formally override the community for the health of >> the Project, then we will do so, but we will consider reaching this point to >> indicate a failure in our leadership." > > > Granting commit rights to the repo does not seem to me to be an "everyday > project activity" --- so I suppose I was confused. I suppose that could > happen with no serious objections on the list. It seems that the people > who actually have the commit bit presently should be consulted more than the > general public, though. I see, right, slight miscommunication here -- I was thinking about the decision of "what process do we [in general] use to decide who gets a commit bit", not the decision "should [this person] get a commit bit". Currently, yeah, the general process is that commit bits get given out by a somewhat informal private discussion among active committers (or, presumably, the "the steering council" if it does get formalized). But if someone wants to suggest that we switch to some other process instead, or formalize the current process, then anything like that would be proposed and debated on the mailing list. (One of my favorite wacky policies is that there are projects like Rubinius which automatically grant commit bits to everyone who submits a successful patch. I guess the theory is that worst case, they go and merge something they shouldn't have, more senior folks see it going by and they revert it again, no biggie! Usually the problem is just the opposite -- there are never enough reviewers. But I have not quite had the guts to seriously propose this for numpy ;-).) -n -- Nathaniel J. Smith -- http://vorpus.org [View Less]

5 9

draft NEP for breaking ufunc ABI in a controlled way
by Nathaniel Smith Sept. 24, 2015

Sept. 24, 2015

Hi all, Here's a first draft NEP for comments. -- Synopsis ======== Improving numpy's dtype system requires that ufunc loops start having access to details of the specific dtype instance they are acting on: e.g. an implementation of np.equal for strings needs access to the dtype object in order to know what "n" to pass to strncmp. Similar issues arise with variable length strings, missing values, categorical data, unit support, datetime with timezone support, etc. -- this is a major blocker … [View More]for improving numpy. Unfortunately, the current ufunc inner loop function signature makes it very difficult to provide this information. We might be able to wedge it in there, but it'd be ugly. The other option would be to change the signature. What would happen if we did this? For most common uses of the C API/ABI, we could do this easily while maintaining backwards compatibility. But there are also some rarely-used parts of the API/ABI that would be prohibitively difficult to preserve. In addition, there are other potential changes to ufuncs on the horizon (e.g. extensions of gufuncs to allow them to be used more generally), and the current API exposure is so massive that any such changes will be difficult to make in a fully compatible way. This NEP thus considers the possibility of closing down the ufunc API to a minimal, maintainable subset of the current API. To better understand the consequences of this potential change, I performed an exhaustive analysis of all the code on Github, Bitbucket, and Fedora, among others. The results make me highly confident that of all the publically available projects in the world, the only ones which touch the problematic parts of the ufunc API are: Numba, dynd-python, and `gulinalg <https://github.com/ContinuumIO/gulinalg>`_ (with the latter's exposure being trivial). Given this, I propose that for 1.11 we: 1) go ahead and hide/disable the problematic parts of the ABI/API, 2) coordinate with the known affected projects to minimize disruption to their users (which is made easier since they are all projects that are almost exclusively distributed via conda, which enforces strict NumPy ABI versioning), 3) publicize these changes widely so as to give any private code that might be affected a chance to speak up or adapt, and 4) leave the "ABI version tag" as it is, so as not to force rebuilds of the vast majority of projects that will be unaffected by these changes. This NEP defers the question of exactly what the improved API should be, since there's no point in trying to nail down the details until we've decided whether it's even possible to change. Details ======= The problem ----------- Currently, a ufunc inner loop implementation is called via the following function prototype:: typedef void (*PyUFuncGenericFunction) (char **args, npy_intp *dimensions, npy_intp *strides, void *innerloopdata); Here ``args`` is an array of pointers to 1-d buffers of input/output data, ``dimensions`` is a pointer to the number of entries in these buffers, ``strides`` is an array of integers giving the strides for each input/output array, and ``innerloopdata`` is an arbitrary void* supplied by whoever registered the ufunc loop. (For gufuncs, extra shape and stride information about the core dimensions also gets packed into the ends of these arrays in a somewhat complicated way.) There are 4 key items that define a NumPy array: data, shape, strides, dtype. Notice that this function only gets access to 3 of them. Our goal is to fix that. For example, a better signature would be:: typedef void (*PyUFuncGenericFunction_NEW) (char **data, npy_intp *shapes, npy_intp *strides, PyArray_Descr *dtypes, /* NEW */ void *innerloopdata); (In practice I suspect we might want to make some more changes as well, like upgrading gufunc core shape/strides to proper arguments instead of tacking it onto the existing arrays, and adding an "escape valve" void* reserved for future extensions. But working out such details is outside the scope of this NEP; the above will do for illustration.) The goal of this NEP is to clear the ground so that we can start supporting ufunc inner loops that take dtype arguments, and make other enhancements to ufunc functionality going forward. Proposal -------- Currently, the public API/ABI for ufuncs consists of the functions:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData together with direct access to PyUFuncObject's internal fields:: typedef struct { PyObject_HEAD int nin, nout, nargs; int identity; PyUFuncGenericFunction *functions; void **data; int ntypes; int check_return; const char *name; char *types; const char *doc; void *ptr; PyObject *obj; PyObject *userloops; int core_enabled; int core_num_dim_ix; int *core_num_dims; int *core_dim_ixs; int *core_offsets; char *core_signature; PyUFunc_TypeResolutionFunc *type_resolver; PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; npy_uint32 *op_flags; npy_uint32 iter_flags; } PyUFuncObject; Obviously almost any future changes to how ufuncs work internally will involve touching some part of this public API/ABI. Concretely, the proposal here is that we avoid this by disabling the following functions (i.e., any attempt to call them should simply raise a ``NotImplementedError``):: PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData and that we reduce the publicly visible portion of PyUFuncObject down to:: typedef struct { PyObject_HEAD int nin, nout, nargs; } PyUFuncObject; Data on current API/ABI usage ----------------------------- In order to assess how much code would be affected by this proposal, I used a combination of Github search and Searchcode.com to trawl through the majority of all publicly available open source code. Neither search tool provides a fine-grained enough query language to directly tell us what we want to know, so I instead followed the strategy of first, casting a wide net: picking a set of search terms that are likely to catch all possibly-broken code (together with many false positives), and second, using automated tools to sift out the false positives and see what remained. Altogether, I reviewed 4464 search results. The tool I wrote to do this is `available on github <https://github.com/njsmith/codetrawl>`_, and so is `the analysis code itself <https://github.com/njsmith/ufunc-abi-analysis>`_. Uses of PyUFuncObject internals ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are no functions in the public API which return ``PyUFuncObject*`` values directly, so any code that access PyUFuncObject fields will have to mention that token in the course of defining a variable, performing a cast, setting up a typedef, etc. Therefore, I searched Github for all files written in C, C++, Objective C, Python, or Cython, which mentioned either "PyUFuncObject AND types" or "PyUFuncObject AND NOT types". (This is to work around limitations on how many results Github search is willing to return to a single query.) In addition, I searched for ``PyUFuncObject`` on searchcode.com. The full report on these searches is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-… The following were screened out as non-problems: - Copies of NumPy itself (an astonishing number of people have checked in copies of it to their own source tree) - NumPy forks / precursors / etc. (e.g. Numeric also had a type called PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) - Cython-generated boilerplate used to generate the "object has changed size" warning (which we `unconditionally filter out anyway <https://github.com/numpy/numpy/blob/master/numpy/__init__.py#L226>`_) - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which require casting the first argument to ``PyUFuncObject*`` - Misc. other unproblematic stuff (like Cython header declarations that never get used) There were also several cases that actually referenced PyUFuncObject internal fields: - The "rational" dtype from numpy-dtypes, which is used in a few projects, accesses ``ufunc->nargs`` as a safety check, but does not touch any other fields (`see here <https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e…>`_). - Numba: does some rather elaborate things to support the definition of on-the-fly JITted ufuncs. These seem to be clear deficiencies in the ufunc API (e.g., there's no good way to control the lifespan of the array of function pointers passed to ``PyUFunc_FromFuncAndData``), so we should work with them to provide the API they need to do this in a maintainable way. Some of the relevant code: https://github.com/numba/numba/tree/master/numba/npyufunc https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd1… - dynd-python: Contains some code that attempts to extract the inner loops from a numpy ufunc object and wrap them into the dynd 'ufunc' equivalent: https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df… - gulinalg: I'm not sure if anyone is still using this code since most of it was merged into numpy itself, but it's not a big deal in any case: all it contains is a `debugging function <https://github.com/ContinuumIO/gulinalg/blob/2ef365c48427c026dab4f45dc6f8b1…>`_ that dumps some internal fields from the PyUFuncObject. If you look, though, all calls to this function are already commented out :-). The full report is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-… In the course of this analysis, it was also noted that the standard Cython pxd files contain a wrapper for ufunc objects:: cdef class ufunc [object PyUFuncObject]: ... which means that Cython code can access internal struct fields via an object of type ``ufunc``, and thus escape our string-based search above. Therefore I also examined all Cython files on Github or searchcode.com that matched the query ``ufunc``, and searched for any lines matching any of the following regular expressions:: cdef\W+ufunc catches: 'cdef ufunc fn' cdef\W+.*\.\W*ufunc catches: 'cdef np.ufunc fn' <.*ufunc\W*> catches: '(<ufunc> fn).nargs', '(< np.ufunc > fn).nargs' cdef.*\(.*ufunc catches: 'cdef doit(np.ufunc fn, ...):' (I considered parsing the actual source and analysing it that way, but decided I was too lazy. This could be done if anyone is worried that the above regexes might miss things though.) There were zero files that contained matches for any of the above regexes: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-r… Uses of PyUFunc_ReplaceLoopBySignature ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Applying the same screening as above, the only code that was found that used this function is also in Numba: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_Replac… Uses of PyUFunc_SetUsesArraysAsData ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Aside from being semi-broken since 1.7 (it never got implemented for "masked" ufunc loops, i.e. those that use where=), there appear to be zero uses of this functionality either inside or outside NumPy: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUse… Rationale --------- **Rationale for preserving the remaining API functions**:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType In addition to being widely used, these functions can easily be preserved even if we change how ufuncs work internally, because they only ingest loop function pointers, they never return them. So they can easily be modified to wrap whatever loop function(s) they receive inside an adapter function that calls them at the appropriate time, and then register that adapter function using whatever API we add in the future. **Rationale for preserving the particular fields that are preserved**: Preserving ``nargs`` lets us avoid a little bit of breakage with the random dtype, and it doesn't seem like preserving ``nin``, ``nout``, ``nargs`` fields will produce any undue burden on future changes to ufunc internals; even if we were to introduce variadic ufuncs we could always just stick a -1 in the appropriate fields or whatever. **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this function *returns* the PyUFunc_GenericFunction that was replaced; if we stop representing all loops using the legacy PyUFunc_GenericFunction type, then this will not be possible to do in the future. **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the ``innerloopdata`` on a ufunc loop, then this function acts as a sentinel value, and causes the ``innerloopdata`` to instead be set to a pointer to the passed-in PyArrayObjects. In principle we could preserve this function, but it has a number of deficiencies: - No-one appears to use it. - It's been buggy for several releases and no-one noticed. - AFAIK the only reason it was included in the first place is that it provides a backdoor for ufunc loops to get access to the dtypes -- but we are planning to fix this in a better way. - It can't be shimmed as easily as the loop registration functions, because we don't anticipate that the new-and-improved ufunc loop functions will *get* access to the array objects, only to the dtypes; so this would have to remain cluttering up the core dispatch path indefinitely. - We have good reason for *not* wanting to get ufunc loops get access to the actual array objects, because one of the goals on our roadmap is exactly to enable the use of ufuncs on non-ndarray objects. Giving ufuncs access to dtypes alone creates a clean boundary here: it guarantees that ufunc loops can work equally on all duck-array objects (so long as they have a dtype), and enforces the invariant that anything which affects the interpretation of data values should be attached to the dtype, not to the array object. Rejected alternatives --------------------- **Do nothing**: there's no way we'll ever be able to touch ufuncs at all if we don't hide those fields sooner or later. While any amount of breakage is regrettable, the costs of cleaning things up now are less than the costs of never improving numpy's APIs. **Somehow sneak the dtype information in via ``void *innerloopdata``**: This might let us preserve the signature of PyUFunc_GenericFunction, and thus preserve PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of leaving way too much internal state exposed, and it's not even clear how this would work, given that we actually do want to preserve the use of ``innerloopdata`` for actual per-loop data. (This is where the PyUFunc_SetUsesArraysAsData hack fails.) -- Nathaniel J. Smith -- http://vorpus.org [View Less]

7 14

Problem while writing array with np.savetxt
by Andrew Nelson Sept. 24, 2015

Sept. 24, 2015

Dear list, whilst trying to write an array to disk I am coming across the following. What am I doing wrong? Surely f is a file handle? (python 3.4.3, numpy 1.9.2) import numpy as np a = np.arange(10.) with open('test.dat', 'w') as f: np.savetxt(f, a) --------------------------------------------------------------------------- TypeError Traceback (most recent call last)<ipython-input-57-cf77f423517e> in <module>() 2 a = np.arange(10.) … [View More]

3 2

ANN: numtraits v0.2
by Thomas Robitaille Sept. 24, 2015

Sept. 24, 2015

Hi everyone, We have released a small experimental package called numtraits that builds on top of the traitlets package and provides a NumericalTrait class that can be used to validate properties such as: * number of dimension (for arrays) * shape (for arrays) * domain (e.g. positive, negative, range of values) * units (with support for astropy.units, pint, and quantities) The idea is to be able to write a class like: class Sphere(HasTraits): radius = NumericalTrait(domain='… [View More]

4 5

Governance model request
by Travis Oliphant Sept. 24, 2015

Sept. 24, 2015

After long conversations at BIDS this weekend and after reading the entire governance document, I realized that the steering council is very large and I don't agree with the mechanism by which it is chosen. A one year time frame is pretty short on the context of a two decades old project and I believe the current council has too few people who have been around the community long enough to help unstuck difficult situations if that were necessary. I would recommend three possible adjustments … [View More]

21 63