Mailman 3 July 2016 - NumPy-Discussion

unsubscribe
by Paul Lucek 06 Jul '16

06 Jul '16

On Wednesday, July 6, 2016, <numpy-discussion-request(a)scipy.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org <javascript:;> > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org <javascript:;> > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org <javascript:;> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Joseph Fox-Rabinovitz) > 2. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Benjamin Root) > 3. Re: Added atleast_nd, request for clarification/cleanup of > atleast_3d (Marten van Kerkwijk) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 6 Jul 2016 09:12:32 -0400 > From: Joseph Fox-Rabinovitz <jfoxrabinovitz(a)gmail.com <javascript:;>> > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org > <javascript:;>> > Subject: Re: [Numpy-discussion] Added atleast_nd, request for > clarification/cleanup of atleast_3d > Message-ID: > <CAAa1KPa0RXYH3cCSyRGqpS_hzx9=K= > sp4dQJRbXoGzkEgBdGPw(a)mail.gmail.com <javascript:;>> > Content-Type: text/plain; charset=UTF-8 > > I can add a keyword-only argument that lets you put the new dims > before or after the existing ones. I am not sure how to specify > arbitrary patterns for the new dimensions, but that should take care > of most use cases. > > The use case that motivated this function in the first place is that I > am doing some processing on 4D arrays and I need to reduce them but > return a result with the original dimensionality (but not shape). > atleast_nd seemed like a better solution than atleast_4d. > > -Joe > > > On Wed, Jul 6, 2016 at 3:41 AM, <josef.pktd(a)gmail.com <javascript:;>> > wrote: > > > > > > On Wed, Jul 6, 2016 at 3:29 AM, <josef.pktd(a)gmail.com <javascript:;>> > wrote: > >> > >> > >> > >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers <ralf.gommers(a)gmail.com > <javascript:;>> > >> wrote: > >>> > >>> > >>> > >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith <njs(a)pobox.com > <javascript:;>> wrote: > >>> > >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > >>>> <jfoxrabinovitz(a)gmail.com <javascript:;>> wrote: > >>>> > > >>>> > Hi, > >>>> > > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with > a > >>>> > function np.atleast_nd in PR#7804 > >>>> > (https://github.com/numpy/numpy/pull/7804). > >>>> > > >>>> > As a result of this PR, I have a couple of questions about > >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird with > >>>> > the dimensions: If the input is 1D, it prepends and appends a size-1 > >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This > is > >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does > >>>> > `np.atleast_nd`). > >>>> > > >>>> > - Is there any reason for this behavior? > >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in > >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This > would > >>>> > be a slight API change since the output would not be exactly the > same. > >>>> > >>>> Changing atleast_3d seems likely to break a bunch of stuff... > >>>> > >>>> Beyond that, I find it hard to have an opinion about the best design > for > >>>> these functions, because I don't think I've ever encountered a > situation > >>>> where they were actually what I wanted. I'm not a big fan of coercing > >>>> dimensions in the first place, for the usual "refuse to guess" > reasons. And > >>>> then generally if I do want to coerce an array to another dimension, > then I > >>>> have some opinion about where the new dimensions should go, and/or I > have > >>>> some opinion about the minimum acceptable starting dimension, and/or > I have > >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column > matrix; > >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on that > >>>> requirements list.) > >>>> > >>>> I don't know how typical I am in this. But it does make me wonder if > the > >>>> atleast_* functions act as an attractive nuisance, where new users > take > >>>> their presence as an implicit recommendation that they are actually a > useful > >>>> thing to reach for, even though they... aren't that. And maybe we > should be > >>>> recommending folk move away from them rather than trying to extend > them > >>>> further? > >>>> > >>>> Or maybe they're totally useful and I'm just missing it. What's your > use > >>>> case that motivates atleast_nd? > >>> > >>> I think you're just missing it:) atleast_1d/2d are used quite a bit in > >>> Scipy and Statsmodels (those are the only ones I checked), and in the > large > >>> majority of cases it's the best thing to use there. There's a bunch of > >>> atleast_2d calls with a transpose appended because the input needs to > be > >>> treated as columns instead of rows, but that's still efficient and > readable > >>> enough. > >> > >> > >> > >> As Ralph pointed out its usage in statsmodels. I do find them useful as > >> replacement for several lines of ifs and reshapes > >> > >> We stilll need in many cases the atleast_2d_cols, that appends the > newaxis > >> if necessary. > >> > >> roughly the equivalent of > >> > >> if x.ndim == 1: > >> x = x[:, None] > >> else: > >> x = np.atleast_2d(x) > >> > >> Josef > >> > >>> > >>> > >>> For 3D/nD I can see that you'd need more control over where the > >>> dimensions go, but 1D/2D are fine. > > > > > > > > statsmodels has currently very little code with ndim >2, so I have no > > overview of possible use cases, but it would be necessary to have full > > control over the added axis since axis have a strict meaning and stats > still > > prefer Fortran order to default numpy/C ordering. > > > > Josef > > > > > >>> > >>> > >>> > >>> Ralf > >>> > >>> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion(a)scipy.org <javascript:;> > >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >> > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org <javascript:;> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > ------------------------------ > > Message: 2 > Date: Wed, 6 Jul 2016 09:51:16 -0400 > From: Benjamin Root <ben.v.root(a)gmail.com <javascript:;>> > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org > <javascript:;>> > Subject: Re: [Numpy-discussion] Added atleast_nd, request for > clarification/cleanup of atleast_3d > Message-ID: > < > CANNq6FkrEmDM3DYW2ztHxxELW+wBeq2uj+viF3VNq4KsFEu_dQ(a)mail.gmail.com > <javascript:;>> > Content-Type: text/plain; charset="utf-8" > > While atleast_1d/2d/3d predates my involvement in numpy, I am probably > partly to blame for popularizing them as I helped to fix them up a fair > amount. I wouldn't call its use "guessing". Rather, I would treat them as > useful input sanitizers. If your function is going to be doing 2d indexing > on an input, then it is very convenient to have atleast_2d() at the top of > your function, not only to sanitize the input, but to make it clear that > your code expects at least two dimensions. > > One place where it is used is in np.loadtxt(..., ndmin=N) to protect > against the situation of a single row of data becoming a 1-D array rather > than a 2-D array (or an empty text file returning something completely > useless). > > I have previously pointed out the oddity with atleast_3d(). I can't > remember the explanation I got though. Maybe someone can find the old > thread that has the explanation, if any? > > I think the keyword argument approach for controlling the behavior might be > a good approach, provided that a suitable design could be devised. 1 & 2 > dimensions is fairly trivial to control, but 3+ dimensions has too many > degrees of freedom for me to consider. > > Cheers! > Ben Root > > > On Wed, Jul 6, 2016 at 9:12 AM, Joseph Fox-Rabinovitz < > jfoxrabinovitz(a)gmail.com <javascript:;>> wrote: > > > I can add a keyword-only argument that lets you put the new dims > > before or after the existing ones. I am not sure how to specify > > arbitrary patterns for the new dimensions, but that should take care > > of most use cases. > > > > The use case that motivated this function in the first place is that I > > am doing some processing on 4D arrays and I need to reduce them but > > return a result with the original dimensionality (but not shape). > > atleast_nd seemed like a better solution than atleast_4d. > > > > -Joe > > > > > > On Wed, Jul 6, 2016 at 3:41 AM, <josef.pktd(a)gmail.com <javascript:;>> > wrote: > > > > > > > > > On Wed, Jul 6, 2016 at 3:29 AM, <josef.pktd(a)gmail.com <javascript:;>> > wrote: > > >> > > >> > > >> > > >> On Wed, Jul 6, 2016 at 2:21 AM, Ralf Gommers <ralf.gommers(a)gmail.com > <javascript:;>> > > >> wrote: > > >>> > > >>> > > >>> > > >>> On Wed, Jul 6, 2016 at 7:06 AM, Nathaniel Smith <njs(a)pobox.com > <javascript:;>> wrote: > > >>> > > >>>> On Jul 5, 2016 9:09 PM, "Joseph Fox-Rabinovitz" > > >>>> <jfoxrabinovitz(a)gmail.com <javascript:;>> wrote: > > >>>> > > > >>>> > Hi, > > >>>> > > > >>>> > I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d > with > > a > > >>>> > function np.atleast_nd in PR#7804 > > >>>> > (https://github.com/numpy/numpy/pull/7804). > > >>>> > > > >>>> > As a result of this PR, I have a couple of questions about > > >>>> > `np.atleast_3d`. `np.atleast_3d` appears to do something weird > with > > >>>> > the dimensions: If the input is 1D, it prepends and appends a > size-1 > > >>>> > dimension. If the input is 2D, it appends a size-1 dimension. This > > is > > >>>> > inconsistent with `np.atleast_2d`, which always prepends (as does > > >>>> > `np.atleast_nd`). > > >>>> > > > >>>> > - Is there any reason for this behavior? > > >>>> > - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` > in > > >>>> > terms of `np.atleast_nd`, which is actually much simpler)? This > > would > > >>>> > be a slight API change since the output would not be exactly the > > same. > > >>>> > > >>>> Changing atleast_3d seems likely to break a bunch of stuff... > > >>>> > > >>>> Beyond that, I find it hard to have an opinion about the best design > > for > > >>>> these functions, because I don't think I've ever encountered a > > situation > > >>>> where they were actually what I wanted. I'm not a big fan of > coercing > > >>>> dimensions in the first place, for the usual "refuse to guess" > > reasons. And > > >>>> then generally if I do want to coerce an array to another dimension, > > then I > > >>>> have some opinion about where the new dimensions should go, and/or I > > have > > >>>> some opinion about the minimum acceptable starting dimension, and/or > > I have > > >>>> a maximum dimension in mind. (E.g. "coerce 1d inputs into a column > > matrix; > > >>>> 0d or 3d inputs are an error" -- atleast_2d is zero-for-three on > that > > >>>> requirements list.) > > >>>> > > >>>> I don't know how typical I am in this. But it does make me wonder if > > the > > >>>> atleast_* functions act as an attractive nuisance, where new users > > take > > >>>> their presence as an implicit recommendation that they are actually > a > > useful > > >>>> thing to reach for, even though they... aren't that. And maybe we > > should be > > >>>> recommending folk move away from them rather than trying to extend > > them > > >>>> further? > > >>>> > > >>>> Or maybe they're totally useful and I'm just missing it. What's your > > use > > >>>> case that motivates atleast_nd? > > >>> > > >>> I think you're just missing it:) atleast_1d/2d are used quite a bit > in > > >>> Scipy and Statsmodels (those are the only ones I checked), and in the > > large > > >>> majority of cases it's the best thing to use there. There's a bunch > of > > >>> atleast_2d calls with a transpose appended because the input needs to > > be > > >>> treated as columns instead of rows, but that's still efficient and > > readable > > >>> enough. > > >> > > >> > > >> > > >> As Ralph pointed out its usage in statsmodels. I do find them useful > as > > >> replacement for several lines of ifs and reshapes > > >> > > >> We stilll need in many cases the atleast_2d_cols, that appends the > > newaxis > > >> if necessary. > > >> > > >> roughly the equivalent of > > >> > > >> if x.ndim == 1: > > >> x = x[:, None] > > >> else: > > >> x = np.atleast_2d(x) > > >> > > >> Josef > > >> > > >>> > > >>> > > >>> For 3D/nD I can see that you'd need more control over where the > > >>> dimensions go, but 1D/2D are fine. > > > > > > > > > > > > statsmodels has currently very little code with ndim >2, so I have no > > > overview of possible use cases, but it would be necessary to have full > > > control over the added axis since axis have a strict meaning and stats > > still > > > prefer Fortran order to default numpy/C ordering. > > > > > > Josef > > > > > > > > >>> > > >>> > > >>> > > >>> Ralf > > >>> > > >>> > > >>> _______________________________________________ > > >>> NumPy-Discussion mailing list > > >>> NumPy-Discussion(a)scipy.org <javascript:;> > > >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion > > >>> > > >> > > > > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion(a)scipy.org <javascript:;> > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org <javascript:;> > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > >

1 0

f2py output module name
by klo uo 05 Jul '16

05 Jul '16

Hi, I'm following this guide: http://docs.scipy.org/doc/numpy-dev/f2py/getting-started.html#the-quick-and… I'm on Windows with gfortran and VS2015. When I run: f2py -c -m fib3 fib3.f as output I dont get "fib3.pyd", but "fib3.cp35-win_amd64.pyd". Does anyone know how to get correctly named module in this case? Thanks

2 2

Picking rows with the first (or last) occurrence of each key
by Skip Montanaro 05 Jul '16

05 Jul '16

(I'm probably going to botch the description...) Suppose I have a 2D array of Python objects, the first n elements of each row form a key, the rest of the elements form the value. Each key can (and generally does) occur multiple times. I'd like to generate a new array consisting of just the first (or last) row for each key occurrence. Rows retain their relative order on output. For example, suppose I have this array with key length 2: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] Selecting the first occurrence of each key would return this array: [ 'a', 27, 14.5 ] [ 'b', 12, 99.0 ] [ 'a', 17, 100.3 ] while selecting the last occurrence would return this array: [ 'a', 27, 15.7 ] [ 'a', 17, 100.3 ] [ 'b', 12, -329.0 ] In real life, my array is a bit larger than this example, with the input being on the order of a million rows, and the output being around 5000 rows. Avoiding processing all those extra rows at the Python level would speed things up. I don't know what this filter might be called (though I'm sure I haven't thought of something new), so searching Google or Bing for it would seem to be fruitless. It strikes me as something which numpy or Pandas might already have in their bag(s) of tricks. Pointers appreciated, Skip

5 10

numpy.dtype has the wrong size, try recompiling
by Jaime Fernández del Río 05 Jul '16

05 Jul '16

This question on Stack Overflow: http://stackoverflow.com/questions/38197086/sklearn-numpy-dtype-has-the-wro… If I remember correctly this has something to do with Cython, right? Can't remeber the details though, can someone help that poor soul out? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 1

ANN: PyTables 3.2.3 released.
by Tom Kooij 04 Jul '16

04 Jul '16

=========================== Announcing PyTables 3.2.3 =========================== We are happy to announce PyTables 3.2.3. What's new ========== This is a bug fix release. It solves many issues reported in the months since the release of 3.2.2. In case you want to know more in detail what has changed in this version, please refer to: http://www.pytables.org/release_notes.html For an online version of the manual, visit: http://www.pytables.org/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org Github: http://www.github.com/PyTables/PyTables About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

AUTO: Leon Woo is out of the office (returning 11/07/2016)
by Leon Woo 02 Jul '16

02 Jul '16

I am out of the office until 11/07/2016. For standard requests within the scope of EMG PWM Berlin, please write to EMG PWM Berlin@DBEMEA. For non-standard requests, please cc Hien Pham-Thu. Note: This is an automated response to your message "NumPy-Discussion Digest, Vol 118, Issue 2" sent on 02.07.2016 14:00:01. This is the only notification you will receive while this person is away.-- Informationen (einschließlich Pflichtangaben) zu einzelnen, innerhalb der EU tätigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enthält vertrauliche und/ oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

1 0

Accelerate or OpenBLAS for numpy / scipy wheels?
by Matthew Brett 01 Jul '16

01 Jul '16

Hi, I just succeeded in getting an automated dual arch build of numpy and scipy, using OpenBLAS. See the last three build jobs in these two build matrices: https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119 https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673 Tests are passing on 32 and 64-bit. I didn't upload these to the usual Rackspace container at wheels.scipy.org to avoid confusion. So, I guess the question now is - should we switch to shipping OpenBLAS wheels for the next release of numpy and scipy? Or should we stick with the Accelerate framework that comes with OSX? In favor of the Accelerate build : faster to build, it's what we've been doing thus far. In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK library cross platform, when we have the Windows builds working. Faster to fix bugs with good support from main developer. No multiprocessing crashes for Python 2.7. Any thoughts? Cheers, Matthew

7 13