Mailman 3 April 2017 - NumPy-Discussion

testing needed for f2py with char/string arrays
by Julian Taylor April 15, 2017

April 15, 2017

hi, we need to deprecate the NPY_CHAR typenumber [0] in order to enable us to add new core dtypes without adding ugly hacks to our ABI. Technically the typenumber was deprecated way back in 1.6 when it accidentally broke our ABI. But due to lack of time f2py never got updated to actually follow through. In order to unblock our dtype development cleanly we want to finally do the deprecation properly. As nobody really knows how f2py works and there are no existing unit tests covering the char dtype the change is very likely to break something. The change is available here: https://github.com/numpy/numpy/pull/8948 It attempts to map the NPY_CHAR dtype to the equivalent NPY_STRING with itemsize 1. I have only been able to come up with a test that covers one of the changed places. So if you have a f2py usecase that in some way involves passing arrays of strings back and forth between python and fortran, please test that branch or post a reproducable example here. Thanks, Julian [0] https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarray…

1 0

Fixing inconsistent behaviour of reduceat()
by Dive Nex April 13, 2017

April 13, 2017

Hi all, I would like to try to reach a consensus about a long standing inconsistent behavior of reduceat() reported and discussed here https://github.com/numpy/numpy/issues/834 In summary, it seems an elegant and logical design choice, that all users will expect, for out = ufunc.reduceat(a, indices) to produce, for all indices j (except for the last one) the following out[j] = ufunc.reduce(a[indices[j]:indices[j+1]]) However the current documented and actual behavior is for the case indices[i] >= indices[i+1] to return simply out[j] = a[indices[i]] I cannot see any application where this behavior is useful or where this choice makes sense. This seems just a bug that should be fixed. What do people think? PS: A quick fix for the current implementation is out = ufunc.reduceat(a, indices) out *= np.diff(indices) > 0

2 1

call for testing: unicode loadtxt/genfromtxt
by Julian Taylor April 9, 2017

April 9, 2017

hi, It has been very very long overdue but we finally have an attempt of making our text io functions actually use text IO instead of bytes IO. This means genfromtxt, loadtxt, fromregex and savetxt should support unicode input files of any python supported encoding and universal newlines. This is the first stepping stone to finally making numpy python3 compatible. The code is available in: https://github.com/numpy/numpy/pull/4208 Great effort has been spent to keep it backward compatible but we only have our testsuite as a reference which for sure does not cover all of the workarounds employed for this issue in the last 8 years. So we need people to dig out their ugliest hacks and test if they still work with this changeset. Functions that need testing are: loadtxt genfromtxt fromregex savetxt Test on any input that worked in older versions of numpy (including gzip compressed) and inputs that did not work because they where encoded in something other than latin1 or had issues with linebreaks. The PR adds an encoding keyword argument to all functions dealing with text input and output. All streams opened by the function have been changed from byte streams to text streams. As previously only latin1 encoded byte streams were supported, all input bytestreams are still decoded as such. Converters added by the user may have been relying on the input to them being bytes. To deal with that the default encoding argument is 'bytes' which corresponds to the default encoding (None) and enables conversion to latin1 encoded bytes before passing to user converters. If you want to use converters based on strings now you have to explicitly set encoding to something else (e.g. None). Currently the functions do not support the newlines keyword argument the python IO strings support. This probably will still get added. Related issues and discussions: https://github.com/numpy/numpy/issues/4600 https://github.com/numpy/numpy/issues/3184 https://github.com/numpy/numpy/issues/4939 https://github.com/numpy/numpy/issues/4543 http://numpy-discussion.10968.n7.nabble.com/using-loadtxt-to-load-a-text-fi… http://numpy-discussion.10968.n7.nabble.com/genfromtxt-universal-newline-su… https://github.com/dhomeier/numpy/commit/995ec93 cheers, Julian

1 0

__array_ufunc__
by Charles R Harris April 8, 2017

April 8, 2017

Hi All, After a week of review and rework, the new and improved __array_ufunc__ has turned the corner and is headed down the homestretch. Now is the time for interested parties to give it a final lookover at https://github.com/numpy/numpy/pull/8247. Chuck

1 0

bitwise or'ing rows
by Mads Ipsen April 4, 2017

April 4, 2017

Hi If I have an n x m array of bools, is there a handy way for me to perform a 'bitwise_and' or 'bitwise_or' along an axis, for example all the rows or all the columns? For example a = [[1,0,0,0], [0,0,1,0], [0,0,0,0]] (0 and 1 meaning True and False) a.bitwise_or(axis=0) giving [1,0,1,0] Best regards, Mads -- +---------------------------------------------------------------------+ | Mads Ipsen | +----------------------------------+----------------------------------+ | Overgaden Oven Vandet 106, 4.tv | phone: +45-29716388 | | DK-1415 København K | email: mads.ipsen(a)gmail.com | | Denmark | map : https://goo.gl/maps/oQ6y6 | +----------------------------------+----------------------------------+

2 2

Fwd: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated)
by Ralf Gommers April 3, 2017

April 3, 2017

Hi all, For those who did not see the call on the NumFOCUS list, the below may be of interest. I also have a proposal idea: a redesign of numpy.org. Our website is very poor, both in terms of content and design. If a good designer spends 40 hours on it, that should be enough to change it into something nice (or at least not embarassing). What do you think? Other ideas? Cheers, Ralf ---------- Forwarded message ---------- From: Gina Helfrich <gina(a)numfocus.org> Date: Fri, Mar 17, 2017 at 10:51 AM Subject: Re: [numfocus] Grants up to $3k available to NumFOCUS projects (sponsored & affiliated) To: numfocus(a)googlegroups.com, devs(a)numfocus.org There is no specific template, but proposals should be kept under 2 pages. Maximum 1 submission per project. Required elements of the proposal are: - title - project description - benefit to project/community - project team - and budget Submit proposals to info(a)numfocus.org Best, Gina On Thu, Mar 16, 2017 at 1:13 PM, Bob Carpenter <carp(a)alias-i.com> wrote: > Is there a format you're expecting for proposals? > > Are applications limited to one per project? > > Hard to imagine a project won't have something to do with $3K, > so I imagine you'll get applications from all of the projects if > the proposals take less than an hour or two to put together. > > - Bob > > > On Mar 16, 2017, at 12:14 PM, Gina Helfrich <gina(a)numfocus.org> wrote: > > > > Call for Proposals - Small Development Grants > > > > NumFOCUS is asking for proposals from its sponsored and affiliated > projects for targeted small development projects with a clear benefit to > those projects. This call is motivated by the success of our 2016 > end-of-year fundraising drive; we want to direct the donated funds to our > projects in a way that has impact and visibility to donors and the wider > community. > > > > There are no restrictions on what the funding can be used for. Whether > it’s code development, documentation work, an educational, sustainability > or diversity initiative, or yet something else, we trust the projects > themselves to understand what they need and explain that need in the > proposal. > > > > Available Funding: > > • Up to $3,000 per proposal > > • Allocated funding is $9,000; depending on the number and quality > of proposals this may be adjusted up or down. > > > > Eligibility: > > • Proposals must be approved by the leadership of a NumFOCUS > sponsored or affiliated project. > > • Proposed work must have a clear outcome, achievable within 2017. > > • The call is open to applicants from any nationality and can be > performed at any university, institute or business worldwide (US export > laws permitting). > > > > Timeline: > > • Mid-March 2017: Call for Proposals released > > • 3 April 2017: deadline for proposal submissions > > • 17 April: successful proposals announced > > > > -- > > Dr. Gina Helfrich > > Communications Director, NumFOCUS > > gina(a)numfocus.org > > 512-222-5449 > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "NumFOCUS" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to numfocus+unsubscribe(a)googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "NumFOCUS" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to numfocus+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Dr. Gina Helfrich Communications Director, NumFOCUS gina(a)numfocus.org 512-222-5449 <(512)%20222-5449> <http://www.numfocus.org> <https://twitter.com/NumFOCUS> <https://www.facebook.com/NumFOCUS/> -- You received this message because you are subscribed to the Google Groups "NumFOCUS" group. To unsubscribe from this group and stop receiving emails from it, send an email to numfocus+unsubscribe(a)googlegroups.com. For more options, visit https://groups.google.com/d/optout.

6 10

speed of random number generator compared to Julia
by Pierre Haessig April 3, 2017

April 3, 2017

Hello, I have a simulation code which uses intensively random number generation (e.g. normal rng). I wanted to compare the performance of this simulation across some Numpy implementations and some Julia implementation. In the end, in both languages, the respective best performing implementations are dominated by the rng, but Julia wins by a factor of 4-5, because the rng is 4-5x faster. Here are some timing results: I*n IPython (Python 3.5.2 from Anaconda)* import numpy as np N = 10**6 %timeit np.random.normal(size=N) 10 loops, best of 3: 37.1 ms per loop %timeit np.random.uniform(size=N) 100 loops, best of 3: 10.2 ms per loop *In Julia (0.4.7 x86_64 from Debian testing)* N = 10^6 @time randn(N); 0.007802 seconds (6 allocations: 7.630 MB) @time rand(N); 0.002059 seconds (8 allocations: 7.630 MB) (with some variations between trials) Results are consistent in the sense that generating Gaussian numbers is 3-4 times slower than uniform in Python and in Julia. But how come Julia is 4-5x faster since Numpy uses C implementation for the entire process ? (Mersenne Twister -> uniform double -> Box-Muller transform to get a Gaussian https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/randomkit.c). Also I noticed that Julia uses a different algorithm (Ziggurat Method from Marsaglia and Tsang , https://github.com/JuliaLang/julia/blob/master/base/random.jl#L700) but this doesn't explain the difference for uniform rng. best, Pierre

3 7

Re: [Numpy-discussion] speed of random number generator compared to Julia
by Nathaniel Smith April 3, 2017

April 3, 2017

On Apr 3, 2017 8:59 AM, "Pierre Haessig" <pierre.haessig(a)crans.org> wrote: Le 03/04/2017 à 15:44, Jaime Fernández del Río a écrit : This <https://github.com/JuliaLang/julia/blob/7fb758a275a0b4cf0e3f4cbf482c065cb32…> says that Julia uses this library <http://www.math.sci.hiroshima-u.ac.jp/%7Em-mat/MT/SFMT/#dSFMT>, which is different from the home brewed version of the Mersenne twister in NumPy. The second link I posted claims their speed comes from generating double precision numbers directly, rather than generating random bytes that have to be converted to doubles, as is the case of NumPy through this magical incantation <https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/randomkit.c#…>. They also throw the SIMD acronym around, which likely means their random number generation is parallelized. My guess is that most of the speed-up comes from the SIMD parallelization: the Mersenne algorithm does a lot of work <https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/randomkit.c#…> to produce 32 random bits, so that likely dominates over a couple of arithmetic operations, even if divisions are involved. Thanks for the feedback. I'm not good in enough in reading Julia to be 100% sure, but I feel like that the random.jl (https://github.com/JuliaLang/ julia/blob/master/base/random.jl) contains a Julia implementation of Mersenne Twister... but I have no idea whether it is the "fancy" SIMD version or the "old" 32bits version. That code contains many references to "dSFMT", which is the name of the "fancy" algorithm. IIUC dSFMT is related to the mersenne twister but is actually a different generator altogether -- advertising that Julia uses the mersenne twister is somewhat misleading IMHO. Of course this is really the fault of the algorithm's designers for creating multiple algorithms that have "mersenne twister" as part of their names... -n

1 0