Mailman 3 August 2012 - NumPy-Discussion

Multidimensional neighbours
by Jose Gomez-Dans 17 Aug '12

17 Aug '12

Hi, I've just come across Travis Oliphant's array version of the game of life here <https://gist.github.com/3353411>. It's a really nice example of how to efficiently find neighbours using numpy and family. However, I wanted to apply these ideas to higher order arrays (eg GRID being three dimensional rather than two dimensional). My previous attempts at this include the (sorry for the ensuing horror!) neighbours = np.array( [ \ # Lower row... x[:-2, :-2, :-2], x[:-2, :-2, 1:-1], x[:-2, :-2, 2: ], x[:-2, 1:-1,:-2], x[:-2, 1:-1, 1:-1], x[:-2, 1:-1, 2:], x[:-2, 2:,:-2], x[:-2, 2:, 1:-1], x[:-2, 2:, 2:], # Middle row x[1:-1, :-2, :-2], x[1:-1, :-2, 1:-1], x[1:-1, :-2, 2: ], x[1:-1, 1:-1,:-2], x[1:-1, 1:-1, 2:], x[1:-1, 2:,:-2], x[1:-1, 2:, 1:-1], x[1:-1, 2:, 2:], # Top row x[2:, :-2, :-2], x[2:, :-2, 1:-1], x[2:, :-2, 2: ], x[2:, 1:-1,:-2], x[2:, 1:-1, 1:-1], x[2:, 1:-1, 2:], x[2:, 2:,:-2], x[2:, 2:, 1:-1], x[2:, 2:, 2:] ] ) I think this works (it's been a while! ;D), but it's a bit messy and the boundary condition is a bit crude. I was wondering whether there's some other nice way of doing it, like demonstrated in the above link. Also, what if you want to have a larger neighbourhood (say 5x5x2 instead of 3x3x3)? While I appreciate these index tricks, I also find them quite mind boggling! Thanks! Jose

3 3

ANN: FiPy 3.0
by Daniel Wheeler 16 Aug '12

16 Aug '12

We are pleased to announce the release of FiPy 3.0. http://www.ctcms.nist.gov/fipy The bump in major version number reflects more on the substantial increase in capabilities and ease of use than it does on a break in compatibility with FiPy 2.x. Few, if any, changes to your existing scripts should be necessary. The significant changes since version 2.1 are: • Coupled and vector equations are now supported. • A more robust mechanism for specifying boundary conditions is now used. • Most Meshes can be partitioned by meshing with Gmsh. • PyAMG and SciPy have been added to the solvers. • FiPy is capable of running under Python 3. • “getter” and “setter” methods have been pervasively changed to Python properties. • The test suite now runs much faster. • Tests can now be run on a full install using fipy.test(). This release addresses 66 tickets. ======================================================================== FiPy is an object oriented, partial differential equation (PDE) solver, written in Python, based on a standard finite volume (FV) approach. The framework has been developed in the Metallurgy Division and Center for Theoretical and Computational Materials Science (CTCMS), in the Material Measurement Laboratory (MML) at the National Institute of Standards and Technology (NIST). The solution of coupled sets of PDEs is ubiquitous to the numerical simulation of science problems. Numerous PDE solvers exist, using a variety of languages and numerical approaches. Many are proprietary, expensive and difficult to customize. As a result, scientists spend considerable resources repeatedly developing limited tools for specific problems. Our approach, combining the FV method and Python, provides a tool that is extensible, powerful and freely available. A significant advantage to Python is the existing suite of tools for array calculations, sparse matrices and data rendering. The FiPy framework includes terms for transient diffusion, convection and standard sources, enabling the solution of arbitrary combinations of coupled elliptic, hyperbolic and parabolic PDEs. Currently implemented models include phase field treatments of polycrystalline, dendritic, and electrochemical phase transformations as well as a level set treatment of the electrodeposition process. -- Daniel Wheeler

1 0

QQ Plot in Python and Noncentral Chisquare
by Yi, Chuang 16 Aug '12

16 Aug '12

Hello, I am a new user of Python. I have a couple of questions that would appreciate your guidance! * QQ Plot in Python: I could not find any functions in either Numpy or Scipy to do QQ Plot of two vectors of data. For example, in Matlab, one could just call qqplot(X,Y), which will generate the QQ plot of vector X against vector Y. Would you please let me know which python package has similar functionality? * Noncentral Chisquare: it looks to me that in Numpy and Scipy, the noncentral chisquare distribution only allows Integer degree of freedom. For example, the following code will produce an error message. In R, however, it also allows non-integer degree of freedom. Would you please let me know which python package has similar functionality? * data = numpy.random.noncentral_chisquare(0.5, 2, 100) * ValueError: df <= 0 Thank you very much, Regards, Chuang ---------------------------------------------------------------------- This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.

3 6

how to use numpy-vendor
by Ondřej Čertík 14 Aug '12

14 Aug '12

Hi, How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do "ldd" and all is clear. Ondrej [1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/

5 9

minor threads-related issue in numpy-release 1.6.2 and numpy-dev
by Aron Ahmadia 14 Aug '12

14 Aug '12

Hi all, Installing numpy 1.6.2 against a Python interpreter built with the --without threads currently fails due to missing references to PyGILState_Ensure and PyGILState_Release. The references appear to be coming from the following code in nditer.c.src: NPY_NO_EXPORT void NpyIter_DebugPrint(NpyIter *iter) { // <snip> PyGILState_STATE gilstate = PyGILState_Ensure(); // <snip> PyGILState_Release(gilstate); } Since this is debugging code, I'm guessing it doesn't get called very frequently, and I could probably just #ifdef it out or use the NPY macros for grabbing the GIL for a non-threaded build: (NPY_ALLOW_C_API and NPY_DISABLE_C_API), but I don't understand why it's grabbing the GIL in the first place. Where is it calling into the interpreter? Does it need the GIL for something else? I'm hesitant to touch this code and issue a pull request until I understand what it's trying to do. Heading on over to the master branch at numpy/numpy, I'm starting to notice more unprotected PyGILState references creeping into the development code. Even the Python developers seem to think that nobody is using --without-threaded, so I'm not going to make a strong case for being more careful, but I do want to point it out in case you want to keep the numpy sources correct for this case. Thanks, Aron

3 3

ANN: SciPy 0.11.0 release candidate 2
by Ralf Gommers 13 Aug '12

13 Aug '12

Hi, I am pleased to announce the availability of the second release candidate of SciPy 0.11.0. For this release many new features have been added, and over 120 tickets and pull requests have been closed. Also noteworthy is that the number of contributors for this release has risen to over 50. Some of the highlights are: - A new module, sparse.csgraph, has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release notes are copied below. For this release candidate all known issues (with the exception of one Qhull issue on Debian, s390x platform) have been solved. In the meantime also OS X 10.8 was released, this RC contains a few build fixes for that platform. If no more serious issues are reported, the final release will be in one week. Cheers, Ralf ========================== SciPy 0.11.0 Release Notes ========================== .. note:: Scipy 0.11.0 is not released yet! .. contents:: SciPy 0.11.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. Highlights of this release are: - A new module has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Our development attention will now shift to bug-fix releases on the 0.11.x branch, and on adding new features on the master branch. This release requires Python 2.4-2.7 or 3.1-3.2 and NumPy 1.5.1 or greater. New features ============ Sparse Graph Submodule ---------------------- The new submodule :mod:`scipy.sparse.csgraph` implements a number of efficient graph algorithms for graphs stored as sparse adjacency matrices. Available routines are: - :func:`connected_components` - determine connected components of a graph - :func:`laplacian` - compute the laplacian of a graph - :func:`shortest_path` - compute the shortest path between points on a positive graph - :func:`dijkstra` - use Dijkstra's algorithm for shortest path - :func:`floyd_warshall` - use the Floyd-Warshall algorithm for shortest path - :func:`breadth_first_order` - compute a breadth-first order of nodes - :func:`depth_first_order` - compute a depth-first order of nodes - :func:`breadth_first_tree` - construct the breadth-first tree from a given node - :func:`depth_first_tree` - construct a depth-first tree from a given node - :func:`minimum_spanning_tree` - construct the minimum spanning tree of a graph ``scipy.optimize`` improvements ------------------------------- The optimize module has received a lot of attention this release. In addition to added tests, documentation improvements, bug fixes and code clean-up, the following improvements were made: - A unified interface to minimizers of univariate and multivariate functions has been added. - A unified interface to root finding algorithms for multivariate functions has been added. - The L-BFGS-B algorithm has been updated to version 3.0. Unified interfaces to minimizers ```````````````````````````````` Two new functions ``scipy.optimize.minimize`` and ``scipy.optimize.minimize_scalar`` were added to provide a common interface to minimizers of multivariate and univariate functions respectively. For multivariate functions, ``scipy.optimize.minimize`` provides an interface to methods for unconstrained optimization (`fmin`, `fmin_powell`, `fmin_cg`, `fmin_ncg`, `fmin_bfgs` and `anneal`) or constrained optimization (`fmin_l_bfgs_b`, `fmin_tnc`, `fmin_cobyla` and `fmin_slsqp`). For univariate functions, ``scipy.optimize.minimize_scalar`` provides an interface to methods for unconstrained and bounded optimization (`brent`, `golden`, `fminbound`). This allows for easier comparing and switching between solvers. Unified interface to root finding algorithms ```````````````````````````````````````````` The new function ``scipy.optimize.root`` provides a common interface to root finding algorithms for multivariate functions, embeding `fsolve`, `leastsq` and `nonlin` solvers. ``scipy.linalg`` improvements ----------------------------- New matrix equation solvers ``````````````````````````` Solvers for the Sylvester equation (``scipy.linalg.solve_sylvester``, discrete and continuous Lyapunov equations (``scipy.linalg.solve_lyapunov``, ``scipy.linalg.solve_discrete_lyapunov``) and discrete and continuous algebraic Riccati equations (``scipy.linalg.solve_continuous_are``, ``scipy.linalg.solve_discrete_are``) have been added to ``scipy.linalg``. These solvers are often used in the field of linear control theory. QZ and QR Decomposition ```````````````````````` It is now possible to calculate the QZ, or Generalized Schur, decomposition using ``scipy.linalg.qz``. This function wraps the LAPACK routines sgges, dgges, cgges, and zgges. The function ``scipy.linalg.qr_multiply``, which allows efficient computation of the matrix product of Q (from a QR decompostion) and a vector, has been added. Pascal matrices ``````````````` A function for creating Pascal matrices, ``scipy.linalg.pascal``, was added. Sparse matrix construction and operations ----------------------------------------- Two new functions, ``scipy.sparse.diags`` and ``scipy.sparse.block_diag``, were added to easily construct diagonal and block-diagonal sparse matrices respectively. ``scipy.sparse.csc_matrix`` and ``csr_matrix`` now support the operations ``sin``, ``tan``, ``arcsin``, ``arctan``, ``sinh``, ``tanh``, ``arcsinh``, ``arctanh``, ``rint``, ``sign``, ``expm1``, ``log1p``, ``deg2rad``, ``rad2deg``, ``floor``, ``ceil`` and ``trunc``. Previously, these operations had to be performed by operating on the matrices' ``data`` attribute. LSMR iterative solver --------------------- LSMR, an iterative method for solving (sparse) linear and linear least-squares systems, was added as ``scipy.sparse.linalg.lsmr``. Discrete Sine Transform ----------------------- Bindings for the discrete sine transform functions have been added to ``scipy.fftpack``. ``scipy.interpolate`` improvements ---------------------------------- For interpolation in spherical coordinates, the three classes ``scipy.interpolate.SmoothSphereBivariateSpline``, ``scipy.interpolate.LSQSphereBivariateSpline``, and ``scipy.interpolate.RectSphereBivariateSpline`` have been added. Binned statistics (``scipy.stats``) ----------------------------------- The stats module has gained functions to do binned statistics, which are a generalization of histograms, in 1-D, 2-D and multiple dimensions: ``scipy.stats.binned_statistic``, ``scipy.stats.binned_statistic_2d`` and ``scipy.stats.binned_statistic_dd``. Deprecated features =================== ``scipy.sparse.cs_graph_components`` has been made a part of the sparse graph submodule, and renamed to ``scipy.sparse.csgraph.connected_components``. Calling the former routine will result in a deprecation warning. ``scipy.misc.radon`` has been deprecated. A more full-featured radon transform can be found in scikits-image. ``scipy.io.save_as_module`` has been deprecated. A better way to save multiple Numpy arrays is the ``numpy.savez`` function. The `xa` and `xb` parameters for all distributions in ``scipy.stats.distributions`` already weren't used; they have now been deprecated. Backwards incompatible changes ============================== Removal of ``scipy.maxentropy`` ------------------------------- The ``scipy.maxentropy`` module, which was deprecated in the 0.10.0 release, has been removed. Logistic regression in scikits.learn is a good and modern alternative for this functionality. Minor change in behavior of ``splev`` ------------------------------------- The spline evaluation function now behaves similarly to ``interp1d`` for size-1 arrays. Previous behavior:: >>> from scipy.interpolate import splev, splrep, interp1d >>> x = [1,2,3,4,5] >>> y = [4,5,6,7,8] >>> tck = splrep(x, y) >>> splev([1], tck) 4. >>> splev(1, tck) 4. Corrected behavior:: >>> splev([1], tck) array([ 4.]) >>> splev(1, tck) array(4.) This affects also the ``UnivariateSpline`` classes. Behavior of ``scipy.integrate.complex_ode`` ------------------------------------------- The behavior of the ``y`` attribute of ``complex_ode`` is changed. Previously, it expressed the complex-valued solution in the form:: z = ode.y[::2] + 1j * ode.y[1::2] Now, it is directly the complex-valued solution:: z = ode.y Minor change in behavior of T-tests ----------------------------------- The T-tests ``scipy.stats.ttest_ind``, ``scipy.stats.ttest_rel`` and ``scipy.stats.ttest_1samp`` have been changed so that 0 / 0 now returns NaN instead of 1. Other changes ============= The SuperLU sources in ``scipy.sparse.linalg`` have been updated to version 4.3 from upstream. The function ``scipy.signal.bode``, which calculates magnitude and phase data for a continuous-time system, has been added. The two-sample T-test ``scipy.stats.ttest_ind`` gained an option to compare samples with unequal variances, i.e. Welch's T-test. ``scipy.misc.logsumexp`` now takes an optional ``axis`` keyword argument. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong * Chad Baker * Brandon Beacher + * behrisch + * borishim + * Matthew Brett * Lars Buitinck * Luis Pedro Coelho + * Johann Cohen-Tanugi * David Cournapeau * dougal + * Ali Ebrahim + * endolith + * Bjørn Forsman + * Robert Gantner + * Sebastian Gassner + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Jonathan Helmus + * Andreas Hilboll + * Marc Honnorat + * Jonathan Hunt + * Maxim Ivanov + * Thouis (Ray) Jones * Christopher Kuster + * Josh Lawrence + * Denis Laxalde + * Travis Oliphant * Joonas Paalasmaa + * Fabian Pedregosa * Josef Perktold * Gavin Price + * Jim Radford + * Andrew Schein + * Skipper Seabold * Jacob Silterra + * Scott Sinclair * Alexis Tabary + * Martin Teichmann * Matt Terry + * Nicky van Foreest + * Jacob Vanderplas * Patrick Varilly + * Pauli Virtanen * Nils Wagner + * Darryl Wally + * Stefan van der Walt * Liming Wang + * David Warde-Farley + * Warren Weckesser * Sebastian Werk + * Mike Wimmer + * Tony S Yu + A total of 55 people contributed to this release. People with a "+" by their names contributed a patch for the first time.

1 0

Vagrant VM for building NumPy (1.7.x) Windows binaries
by Ondřej Čertík 13 Aug '12

13 Aug '12

Hi, I've created this repository: https://github.com/certik/numpy-vendor which uses Vagrant and Fabric to fully automate the setup & creation of NumPy binaries for Windows. The setup is especially tricky, I've thought several times already that I nailed it, and then always new things pop up. One can of course install things directly in Ubuntu, but it's tricky, there are a lot of things that can go wrong. The above approach should be 100% reproducible. So hopefully this repository will be useful for somebody new (like I am) to numpy releases. Also my hope is that more people can help out with the release just by running it on their machines and/or sending PRs against this repository. All essential logic is in these two files: https://github.com/certik/numpy-vendor/blob/master/fabfile.py https://github.com/certik/numpy-vendor/blob/master/setup-wine.sh The setup-wine.sh can be used directly in Ubuntu as well (it erases ~/.wine and reinstalls it). Times: on my computer and internet speed it takes a little less than 1h to prepare the VM (it needs to download the 300MB Ubuntu, then about 600MB of Latex, and few MB of other stuff, and then about 12 minutes of configuring), and about 0.5 for each binary, so about 6h total (the docs only take a few minutes). After doing all this, I just realized that I forgot to build numpy against atlas. So I will need to fix that. But apart from that, it should be ok. Travis, should I upload the binaries to sourceforge and do the beta release with this? Or should we wait until I fix the atlas. Also I need to do the Mac binaries. I'll have the Win binaries in about 6h from now, I am running one final build from scratch now. My apologies that it took me longer than expected. Ondrej P.S. Here is a PR for review (you need to apply it manually if you want the above to work): https://github.com/numpy/numpy/pull/384

3 4

Iterating over a given axis with buffering
by Sergio Pascual 13 Aug '12

13 Aug '12

Hi, I have a 3d array of bitswapped data, dimensions 2000x2000x20. I want to apply a function over the third axis to obtain a 2000x2000 array. This is what numpy.apply_along_axis does, but I need a faster version, so I'm writing it in C. I started by writing a generic ufunc version of the code. It is not what I need, because my particular function has several free parameters, but with this version I could get a hint of the performance. The ufunc was nice, because it handled the bitswapped data, and naturally the innermost loop was over the third axis. So I tried to write the final version using NpyIter. The only way I found to do the innermost loop over the third axis was passing the flag NPY_ITER_MULTI_INDEX, but then I can't use NPY_ITER_BUFFERED to handle my bitswapped, at least in numpy 1.6 they couldn't be used simultaneously. Any hint on how I can do this? Regards, Sergio

1 0

Re: [Numpy-discussion] Second try: possible bug in assignment to complex array
by Mark Bakker 11 Aug '12

11 Aug '12

Shall I file a bug report? Or is this fairly easy to fix? Mark > > On Fri, Aug 10, 2012 at 11:41 AM, <josef.pktd(a)gmail.com> wrote: > > > > > > > On Fri, Aug 10, 2012 at 10:00 AM, Travis Oliphant <travis(a)continuum.io > >wrote: > > > >> > >> On Aug 10, 2012, at 5:37 AM, Paul Anton Letnes wrote: > >> > >> > > >> > > >> > On 10. aug. 2012, at 09:54, Mark Bakker wrote: > >> > > >> >> I am giving this a second try. Can anybody help me out? > >> >> > >> >> I think there is a problem with assigning a 1D complex array of > length > >> one > >> >> to a position in another complex array. > >> >> > >> >> Example: > >> >> > >> >> a = ones(1,'D') > >> >> b = ones(1,'D') > >> >> a[0] = b > >> >> > >> > --------------------------------------------------------------------------- > >> >> TypeError Traceback (most recent call > >> last) > >> >> <ipython-input-37-0c4fc6d780e3> in <module>() > >> >> ----> 1 a[0] = b > >> >> > >> >> TypeError: can't convert complex to float > >> >> > >> >> This works correctly when a and b are real arrays: > >> >> > >> >> a = ones(1) > >> >> b = ones(1) > >> >> a[0] = b > >> >> > >> >> Bug or feature? > >> > > >> > The exact same thing happens on OS X 10.7.4, python 2.7.3, numpy > 1.6.1. > >> > > >> > Looks like a bug to me - or at least very surprising behavior. > >> > >> This is definitely an inconsistency. The error seems more correct > >> (though the error message needs improvement). > >> > >> Can someone try this on NumPy 1.5 and see if this inconsistency existed > >> there as well. > >> > > > > >>> np.__version__ > > '1.5.1' > > > > >>> a = np.ones(1,'D') > > >>> b = np.ones(1,'D') > > >>> a[0] = b > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > > > TypeError: can't convert complex to float > > >>> a = np.ones(1) > > >>> b = np.ones(1) > > >>> a[0] = b > > > > and > > >>> a = np.ones(1,'D') > >>> b = 2*np.ones(1) > >>> a[0] = b > >>> a > array([ 2.+0.j]) > >>> c = 3*np.ones(1, int) > >>> a[0] = c > >>> a > array([ 3.+0.j]) > > > > > > > Josef > > > > > >> Thanks, > >> > >> -Travis > >> > >> > > >> > Paul > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion(a)scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion(a)scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > >

1 0

Limit in NpyIter_MultiNew: NPY_MAXARGS
by Sergio Pascual 11 Aug '12

11 Aug '12

I have written a program that makes use of NpyIter_Multi to iterate simultaneously over a large number of 2d arrays. But I have noticed that there is a limit of 32 in the number of simultaneous iterators allowed. This number is NPY_MAXARGS. I typically need to iterate over hundreds of images. So, what is the rationale to have the limit in 32 instead of, say, 1024? If this number cannot easily be increased in the source code, what is the alternative to iterate over a large number of images? Could it be an array of NpyIter objects? Regards, Sergio

1 0

NumPy-Discussion August 2012