Mailman 3 December 2015 - NumPy-Discussion

Numpy 1.10.2rc2 released
by Charles R Harris 09 Dec '15

09 Dec '15

Hi All, I'm pleased to announce the release of Numpy 1.10.2rc2. After two months of stomping bugs I think the house is clean and we are almost ready to put it up for sale. However, bugs are persistent and may show up at anytime, so please inspect and test thoroughly. Windows binaries and source releases can be found at the usual place on Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.10.2rc2/>. If there are no reports of problems in the next week I plan to release the final. Further bug squashing will be left to the 1.11 release except possibly for regressions. The release notes give more detail on the changes. *bon appétit,* Chuck

3 2

Memory mapping and NPZ files
by Mathieu Dubois 09 Dec '15

09 Dec '15

Dear all, If I am correct, using mmap_mode with Npz files has no effect i.e.: f = np.load("data.npz", mmap_mode="r") X = f['X'] will load all the data in memory. Can somebody confirm that? If I'm correct, the mmap_mode argument could be passed to the NpzFile class which could in turn perform the correct operation. One way to handle that would be to use the ZipFile.extract method to write the Npy file on disk and then load it with numpy.load with the mmap_mode argument. Note that the user will have to remove the file to reclaim disk space (I guess that's OK). One problem that could arise is that the extracted Npy file can be large (it's the purpose of using memory mapping) and therefore it may be useful to offer some control on where this file is extracted (for instance /tmp can be too small to extract the file here). numpy.load could offer a new option for that (passed to ZipFile.extract). Does it make sense? Thanks in advance, Mathieu

1 0

array of random numbers fails to construct
by DAVID SAROFF (RIT Student) 09 Dec '15

09 Dec '15

This works. A big array of eight bit random numbers is constructed: import numpy as np spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8) This fails. It eats up all 64GBy of RAM: spectrumArray = np.random.randint(0,255, (2**21,2**12)).astype(np.uint8) The difference is a factor of two, 2**21 rather than 2**20, for the extent of the first axis. -- David P. Saroff Rochester Institute of Technology 54 Lomb Memorial Dr, Rochester, NY 14623 david.saroff(a)mail.rit.edu | (434) 227-6242

8 12

Re: [Numpy-discussion] Q: Use of scipy.signal.bilinear
by R Schumacher 08 Dec '15

08 Dec '15

Sorry - I'll join there. - Ray At 10:00 AM 12/8/2015, you wrote: >On Tue, Dec 8, 2015 at 9:30 AM, R Schumacher ><<mailto:rays@blue-cove.com>rays(a)blue-cove.com> wrote: >We have a function which describes a frequency >response correction to piezo devices we use. To >flatten the FFT, it is similar to: >Cdis_t = .5 >N = 8192 >for n in range(8192): >Â B3 = n * 2560 / N >Â Fc(n) = 1 / >((B3/((1/(Cdis_t*2*pi))**2+B3**2)**0.5)*(-0.01*log(B3) + 1.04145)) > >In practice it really only matters for low frequencies. > >I suggested that we might be able to do a time >domain correction as a forward-reverse FFT >filter using the function, but another said it >can also be applied in the time domain using a bilinear transform. >So, can one use ><http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.signal.bil…>http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.signal.bil… >and, how does one generate b,a from the given >Fourrier domain flattening function? > > >This should go to either ><mailto:scipy-user@scipy.org>scipy-user(a)scipy.org > or <mailto:scipy-dev@scipy.org>scipy-dev(a)scipy.org > >Chuck >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion(a)scipy.org >https://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

Q: Use of scipy.signal.bilinear
by R Schumacher 08 Dec '15

08 Dec '15

We have a function which describes a frequency response correction to piezo devices we use. To flatten the FFT, it is similar to: Cdis_t = .5 N = 8192 for n in range(8192): B3 = n * 2560 / N Fc(n) = 1 / ((B3/((1/(Cdis_t*2*pi))**2+B3**2)**0.5)*(-0.01*log(B3) + 1.04145)) In practice it really only matters for low frequencies. I suggested that we might be able to do a time domain correction as a forward-reverse FFT filter using the function, but another said it can also be applied in the time domain using a bilinear transform. So, can one use http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.signal.bil… and, how does one generate b,a from the given Fourrier domain flattening function? I'd guess someone here has done this... - Ray

2 1

Re: [Numpy-discussion] Where is Jaime?
by Peter Creasey 08 Dec '15

08 Dec '15

>> > >> > Is the interp fix in the google pipeline or do we need a workaround? >> > >> >> Oooh, if someone is looking at changing interp, is there any chance >> that fp could be extended to take complex128 rather than just float >> values? I.e. so that I could write: >> >> >>> y = interp(mu, theta, m) >> rather than >> >>> y = interp(mu, theta, m.real) + 1.0j*interp(mu, theta, m.imag) >> >> which *sounds* like it might be simple and more (Num)pythonic. > > That sounds like an excellent improvement and you should submit a PR > implementing it :-). > > "The interp fix" in question though is a regression in 1.10 that's blocking > 1.10.2, and needs a quick minimal fix asap. > Good answer - as soon as I hit 'send' I wondered how many bugs get introduced by people trying to attach feature requests to bug fixes. I will take a look at the code later and pm you if I get anywhere... Peter

2 1

Where is Jaime?
by Charles R Harris 07 Dec '15

07 Dec '15

Anyone hear from Jaime lately? He seems to have gone missing. Chuck

3 6

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 111, Issue 9
by Sydney Shall 07 Dec '15

07 Dec '15

On 07/12/2015 09:38, numpy-discussion-request(a)scipy.org wrote: > Message: 4 > Date: Sun, 06 Dec 2015 22:01:40 -0500 > From: "DAVID SAROFF (RIT Student)"<dps7802(a)rit.edu> > To: Discussion of Numerical Python<numpy-discussion(a)scipy.org> > Cc: Stefi Baum<stefibaumodea(a)gmail.com> > Subject: Re: [Numpy-discussion] array of random numbers fails to > construct > Message-ID: > <CABMftcSkX9kL=G2sepOn80YCKQygFGHa7WBdmAxy3_1iVAtJcw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Allan, > > I see with a google search on your name that you are in the physics > department at Rutgers. I got my BA in Physics there. 1975. Biological > physics. A thought: Is there an entropy that can be assigned to the dna in > an organism? I don't mean the usual thing, coupled to the heat bath. > Evolution blindly explores metabolic and signalling pathways, and tends > towards disorder, as long as it functions. Someone working out signaling > pathways some years ago wrote that they were senselessly complex, branched > and interlocked. I think that is to be expected. Evolution doesn't find > minimalist, clear, rational solutions. Look at the amazon rain forest. What > are all those beetles and butterflies and frogs for? It is the wrong > question. I think some measure of the complexity could be related to the > amount of time that ecosystem has existed. Similarly for genomes. Dear David, You are mistaken in this remark in your message; > Evolution blindly explores metabolic and signalling pathways, and > tends towards disorder, as long as it functions. In fact, biological evolution does just the opposite. It overcomes disorder and creates complexity at the expense of 'pulling in' energy from the outside, from the environemnt. Of course you are correct that biological evolution does NOT look for nor does it achieve optimum solutions. It merely replaces current mechanism with another mechanism biologically derived from the current mechanism, provided only that the replacement mechanism is marginally, fractionally superior in the totality of the life of the ecosystem. Have a good day, Sydney -- Sydney

2 1

Re: [Numpy-discussion] Where is Jaime?
by Peter Creasey 07 Dec '15

07 Dec '15

> > Is the interp fix in the google pipeline or do we need a workaround? > Oooh, if someone is looking at changing interp, is there any chance that fp could be extended to take complex128 rather than just float values? I.e. so that I could write: >>> y = interp(mu, theta, m) rather than >>> y = interp(mu, theta, m.real) + 1.0j*interp(mu, theta, m.imag) which *sounds* like it might be simple and more (Num)pythonic. Peter

2 1

Travis is busted.
by Charles R Harris 05 Dec '15

05 Dec '15

Just to note that travis is broken at the moment, it can't find the whitelisted gfortran package.

1 0

f2py, numpy.distutils and multiple Fortran source files
by David Verelst 05 Dec '15

05 Dec '15

Hi, For the wafo [1] package we are trying to include the extension compilation process in setup.py [2] by using setuptools and numpy.distutils [3]. Some of the extensions have one Fortran interface source file, but it depends on several other Fortran sources (modules). The manual compilation process would go as follows: gfortran -fPIC -c source_01.f gfortran -fPIC -c source_02.f f2py -m module_name -c source_01.o source_02.o source_interface.f Can this workflow be incorporated into setuptools/numpy.distutils? Something along the lines as: from numpy.distutils.core import setup, Extension ext = Extension('module.name', depends=['source_01.f', 'source_02.f'], sources=['source_interface.f']) (note that the above does not work) [1] https://github.com/wafo-project/pywafo [2] https://github.com/wafo-project/pywafo/blob/pipinstall/setup.py [3] http://docs.scipy.org/doc/numpy/reference/distutils.html Regards, David

3 3

NumPy Pyston support
by Boxiang Sun 05 Dec '15

05 Dec '15

Hi all, I am an open source contributor of Pyston, and working on NumPy Pyston support(Pyston: https://github.com/dropbox/pyston) I disabled some tests which cause segfaults. This is current situation(in my local branch): ---------------------------------- Ran 3240 tests in 80.811s FAILED (KNOWNFAIL=3, SKIP=6, errors=334, failures=74) ---------------------------------- Indeed, the time consuming was longer than CPython. I think this is because of the additional exceptions / frame introspection etc... And no one took yet a look at the perf. Pyston runtime miss some features. I will try to fix the problems which discovered in running NumPy test suite. This is basically an announcement. I will update the situation when I get breakthrough, ask for help if I encounter problems, submit commit to NumPy if needed. Looking for feedback. Regards, Sun

1 0

future of f2py and Fortran90+
by Eric Firing 04 Dec '15

04 Dec '15

F2py is a great tool, but my impression is that it is being left behind by the evolution of Fortran from F90 onward. This is unfortunate; it would be nice to be able to easily wrap new Fortran libraries. I'm curious: has anyone been looking into what it would take to enable f2py to handle modern Fortran in general? And into prospects for getting such an effort funded? Eric

5 6

Recognizing a cycle in a vector
by Manolo Martínez 03 Dec '15

03 Dec '15

Dear all, Suppose that I have a vector with the numerical solution of a differential equation -- more concretely, I am working with evolutionary game theory models, and the solutions are frequencies of types in a population that follows the replicator dynamics; but this is probably irrelevant. Sometimes these solutions are cyclical, yet I sample at points which do not correspond with the period of the cycle, so that np.allclose() cannot be directly applied. Is there any way to check for cycles in this situation? Thanks for any advice, Manolo

5 11

ANN: SciExp^2 (1.1.2)
by Lluís Vilanova 02 Dec '15

02 Dec '15

Hi all! I'm really happy to make the first public announcement of SciExp^2 (actually, it's release 1.1.2). Home page: https://projects.gso.ac.upc.edu/projects/sciexp2 Description ----------- SciExp² (aka SciExp square or simply SciExp2) stands for Scientific Experiment Exploration, which contains a comprehensive framework for easing the workflow of creating, executing and evaluating experiments. The driving idea behind SciExp² is the need for quick and effortless design-space exploration. It is divided into the following main pieces: * Launchgen: Aids in the process of defining experiments as a permutation of different parameters in the design space, creating the necessary files to run them (configuration files, scripts, etc.). * Launcher: Takes the result of launchgen and integrates with some well-known execution systems (e.g., simple shell scripts or gridengine) to execute and keep track of the experiments (e.g., re-run failed experiments, or run those whose files have been updated). In addition, experiments can be operated through filters that know about the parameters used during experiment generation. * Data: Aids in the process of collecting and analyzing the results of the experiments. Results are collected into arrays whose dimensions can be annotated by the user (e.g., to identify experiment parameters). It also provides functions to automatically extract experiment results into annotated arrays (implemented as numpy arrays with dimension metadata extensions). The framework is available in the form of Python modules which can be easily integrated into your own applications or used as a scripting environment. Notes ----- As you'll see, the data piece is my personal take on "labaled arrays", which I started well before the "datarray" project. It's just too bad that "datarray" did not succeed in unifying the common logic across the different projects with similar features. Cheers, Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth

1 0

Indexing structured masked arrays with multidimensional fields; what with fill_value?
by Gerrit Holl 01 Dec '15

01 Dec '15

Hello, usually, a masked array's .fill_value attribute has ndim=0 and the same dtype as the data attribute: In [27]: ar = array((0, [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], 0.0), dtype="int, (2,3)float, float") In [28]: arm = ma.masked_array(ar) In [29]: arm.fill_value.ndim Out[29]: 0 In [31]: arm.fill_value.dtype Out[31]: dtype([('f0', '<i8'), ('f1', '<f8', (2, 3)), ('f2', '<f8')]) What would be the optimal approach for .fill_value if I address the member "f1" in this case? The current behaviour is: In [32]: f = arm["f1"] In [36]: f.fill_value Out[36]: array([[ 1.00000000e+20, 1.00000000e+20, 1.00000000e+20], [ 1.00000000e+20, 1.00000000e+20, 1.00000000e+20]]) This breaks the usual behaviour that .fill_value has ndim=0, which can cause bugs such as reported in issue #6723: https://github.com/numpy/numpy/issues/6723 What should numpy do instead? In pull request 6728, I propose to change the behaviour so that arm["f1"].fill_value is set to arm.fill_value["f1"].flat[0]. This is an arbitrary and somewhat ad-hoc solution. If I have chosen to set arm.fill_value["f1"] to something else, such as array([[1., 2., 3.], [4., 5., 6.]]), then the rest of my fill_value is lost. I don't know if this might lead to problems. Does it matter? See also http://stackoverflow.com/questions/33921579/what-practical-impact-if-any-do… . regards, Gerrit.

1 0

ANN: SfePy 2015.4
by Robert Cimrman 01 Dec '15

01 Dec '15

I am pleased to announce release 2015.4 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - basic support for restart files - new type of linear combination boundary conditions - balloon inflation example For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman on behalf of the SfePy development team --- Contributors to this release in alphabetical order: Robert Cimrman Grant Stephens

1 0

Inconsistent/unexpected indexing semantics
by Lluís Vilanova 01 Dec '15

01 Dec '15

Hi, TL;DR: There's a pending pull request deprecating some behaviour I find unexpected. Does anyone object? Some time ago I noticed that numpy yields unexpected results in some very specific cases. An array can be used to index multiple elements of a single dimension: >>> a = np.arange(8).reshape((2,2,2)) >>> a[ np.array([[0], [0]]) ] array([[[[0, 1], [2, 3]]], [[[0, 1], [2, 3]]]]) Nonetheless, if a list is used instead, it is (unexpectedly) transformed into a tuple, resulting in indexing across multiple dimensions: >>> a[ [[0], [0]] ] array([[0, 1]]) I.e., it is interpeted as: >>> a[ [0], [0] ] array([[0, 1]]) Or what is the same: >>> a[( [0], [0] )] array([[0, 1]]) I've been informed that there's a pending pull request that deprecates this behaviour [1], which could in the future be reverted to what is expected (at least what I expect) from the documents (except for an obscure note in [2]). The discussion leading to this mail can be found here [3]. [1] https://github.com/numpy/numpy/pull/4434 [2] http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-ind… [3] https://github.com/numpy/numpy/issues/6564 Thanks, Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth

3 2