Mailman 3 August 2012 - NumPy-Discussion

Mark Florisson's GSoC, minivect, and NumPy
by Dag Sverre Seljebotn 27 Aug '12

27 Aug '12

I just wanted to draw the attention of NumPy devs to Mark Florisson's GSoC work. It is 'minivect', a tool to use for compiling array expressions (think (as a concept) a shared backend between Cython, Theano, numba, though it's only used in Cython currently). His M. Sc. thesis, "Techniques for Static and Dynamic Compilation of Array Expressions", is up here: https://github.com/markflorisson88/minivect/tree/master/thesis As you can see he even beats Intel Fortran for some array layouts, and in general have comparable performance with it. The benchmarks are mostly for two-operand operations, i.e. operations where NumPy semantics would be OK. IMO, if anybody ever wants to revamp NumPy's computation abilities and get that 2-3x speedup (e.g., make it multi-threaded), this is a very good place to start. Dag

1 0

vectorized multi-matrix multiplication
by Alex Flint 26 Aug '12

26 Aug '12

I have two lists of 3x3 arrays and I would like to compute the matrix product of the i-th element in the first list with the i-th element in the second list. Of course, I could just loop over the lists: for i in range(n): out[i] = dot( matrices1[i], matrices2[i] ) However, the list is quite long, and each matrix is very small (3x3) so this turns out to be quite slow. Is there a way to do this with a single numpy call? I have looked at tensordot but it outputs an N x N x 3 x 3 array, whereas I want an N x 3 x 3 output. I've also looked at various broadcasting tricks but I haven't found anything that works yet. Alex

2 2

[ANN] Call for abstracts: BigData minisymposium at CSE'13, February 2013, Boston
by Fernando Perez 23 Aug '12

23 Aug '12

Dear colleagues, next year's SIAM conference on Computational Science and Engineering, CSE'13, will take place in Boston, February 25-March 1 (http://www.siam.org/meetings/cse13), and for this version there will be a track focused on the topic of Big Data. This term has rapidly risen in recent discussions of science and even of mainstream business computing, and for good reasons. Today virtually all disciplines are facing a flood of quantitative information whose volumes have often grown faster than the quality of our tools for extracting insight from these data. SIAM hopes that CSE'13 will provide an excellent venue for discussing these problems, from the vantage point offered by a community whose expertise combines analytical insights, algorithmic development, software engineering and domain-specific applications. As part of this event, Titus Brown (http://ged.msu.edu) and I are organizing a minisymposium where we would like to have a group of presentations that address both novel algorithmic ideas and computational approaches as well as domain-specific problems. Data doesn't appear in a vacuum, and data from different domains presents a mix of common problems along with questions that may be specific to each; we hope that by engaging a dialog between those working on algorithmic and implementation questions and those with specific problems from the field, valuable insights can be obtained. If you would like to contribute to this minisymposium, please contact us directly at: "C. Titus Brown" <ctb(a)msu.edu>, "Fernando Perez" <Fernando.Perez(a)berkeley.edu> with your name and affiliation, the title of your proposed talk and a brief description (actual abstracts are due later so an informal description will suffice for now), by Wednesday August 29. For more details on the submission process, see: http://www.siam.org/meetings/cse13/submissions.php Please forward this to any interested colleagues. Regards, Titus and Fernando.

1 0

unsubscribe
by Amit Nagal 23 Aug '12

23 Aug '12

Please unsubscribe me Visit us at Booth No. 5 at 2012 ChemOutsourcing Conference, 10-13 Sept 2012, Ocean Place Resort - Long Branch, NJ, United States Visit us at Booth No. 4 at World Conference on Pharmacometrics, 5-7 Sept 2012, Grand Hilton Hotel, Seoul, Korea ________________________________ Notice: The information contained in this electronic mail message is intended only for the use of the designated recipient. This message is privileged and confidential. and the property of GVK BIO or its affiliates and subsidiaries. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by telephone +91-40-66929999<tel:%2B91-40-66929999> and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

2 1

Re: [Numpy-discussion] 64bit infrastructure
by Sayth Renshaw 22 Aug '12

22 Aug '12

> Message: 4 > Date: Fri, 17 Aug 2012 15:25:21 -0700 > From: Matthew Brett <matthew.brett(a)gmail.com> > Subject: Re: [Numpy-discussion] 64bit infrastructure > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAH6Pt5qa27Zb+UkOZB5EzsLK5UuoB-f8UcwFjdL2YcEBRZ=Wdw(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, > > On Fri, Aug 17, 2012 at 2:25 PM, Sayth Renshaw <flebber.crue(a)gmail.com> wrote: >>> >>> Hi Sayth, >>> >>> On Thu, Aug 16, 2012 at 8:51 PM, Sayth Renshaw <flebber.crue(a)gmail.com> wrote: >>>> Hi >>>> >>>> I was just wondering if the current absence of 64bit builds was as a result >>>> of an infrastructure or funding concern. >>>> >>>> I like many now have only 64 bit systems and am using the unofficial builds. >>>> >>>> If it was something we could raise funds to resolve, how much? Or is it more >>>> complex than that? >>> >>> Which platform, windows? >>> >>> Ondrej >>> >>> >> >> Yes I have a Windows 7 64Bit install and a ubuntu 64 Bit install. > > Isn't the numpy ubuntu install 64 bit by default? > > You may have seen that there are numpy and scipy 64-bit windows installers here: > > http://www.lfd.uci.edu/~gohlke/pythonlibs/ > > Do they work for you? > > Best, > > Matthew > > > ------------------------------ > Thanks Matthew yeah that's the one I am currently using and it is working ok but just thought if there was a way to help 64bit builds become official then I would ask. Sayth

8 18

__array_wrap__ and dot
by Aronne Merrelli 22 Aug '12

22 Aug '12

Hello list, I posted an issue many months ago [1] about confusing __array_wrap__ behavior in a subclass of np.matrix. Since that time I wasn't really using the code that used the matrix subclass, but lately I have starting using the code so I've run into the issue again. Looking into a little more detail, I think the root of the problem is how __array_wrap__ interacts with the dot functions. Operations like M*2 will be processed differently depending on whether M is a ndarray or matrix subclass; it appears that a matrix will always call np.dot for that operation, while an ndarray will go through np.multiply. Since dot acts differently than multiply here, you get different results between the two subclasses. I updated the gist to show the problem [2]. The output I get on my system is pasted at the bottom. It looks like __array_wrap__ gets called sometimes, depending on the type of one of the arguments. I'm not seeing any logical pattern to it, which makes it confusing. My expectation (perhaps incorrect) is that __array_wrap__ should be invoked any time dot is called. I guess that depends on whether dot should be considered a ufunc - I don't know the answer to that question. If someone with more knowledge of the internals could take a look at this, that would be great - I think this is a bug, but I'm not really sure what is the expected behavior here. Thanks, Aronne [1] http://mail.scipy.org/pipermail/numpy-discussion/2011-December/059665.html [2] https://gist.github.com/1511354 Output from run_test (see the gist code) using ipython: In [1]: import matrix_array_wrap_test as mtest; import numpy as np In [2]: np.__version__ Out[2]: '1.6.1' In [3]: np.dot Out[3]: <function numpy.core._dotblas.dot> In [4]: mtest.run_test() aw called? | creation | use Yes | o=ArrSubClass(2) | o2 = o + 2 Yes | o=ArrSubClass(2) | o2 = o + 2.0 Yes | o=ArrSubClass(2) | o2 = o * 2 Yes | o=ArrSubClass(2) | o2 = o * 2.0 Yes | o=ArrSubClass(2) | o2 = o * o.T No | o=ArrSubClass(2) | o2 = np.dot(o,2) No | o=ArrSubClass(2) | o2 = np.dot(o,2.0) No | o=ArrSubClass(2) | o2 = np.dot(o,o.T) Yes | o=ArrSubClass(2) | o2 = np.core.multiarray.dot(o,2) Yes | o=ArrSubClass(2) | o2 = np.core.multiarray.dot(o,2.0) No | o=ArrSubClass(2) | o2 = np.core.multiarray.dot(o,o.T) Yes | o=MatSubClass([1, 1]) | o2 = o + 2 Yes | o=MatSubClass([1, 1]) | o2 = o + 2.0 Yes | o=MatSubClass([1, 1]) | o2 = o * 2 No | o=MatSubClass([1, 1]) | o2 = o * 2.0 No | o=MatSubClass([1, 1]) | o2 = o * o.T Yes | o=MatSubClass([1, 1]) | o2 = np.dot(o,2) No | o=MatSubClass([1, 1]) | o2 = np.dot(o,2.0) No | o=MatSubClass([1, 1]) | o2 = np.dot(o,o.T) Yes | o=MatSubClass([1, 1]) | o2 = np.core.multiarray.dot(o,2) Yes | o=MatSubClass([1, 1]) | o2 = np.core.multiarray.dot(o,2.0) No | o=MatSubClass([1, 1]) | o2 = np.core.multiarray.dot(o,o.T) Yes | o=MatSubClass([1.0, 1.0]) | o2 = o + 2 Yes | o=MatSubClass([1.0, 1.0]) | o2 = o + 2.0 No | o=MatSubClass([1.0, 1.0]) | o2 = o * 2 No | o=MatSubClass([1.0, 1.0]) | o2 = o * 2.0 No | o=MatSubClass([1.0, 1.0]) | o2 = o * o.T No | o=MatSubClass([1.0, 1.0]) | o2 = np.dot(o,2) No | o=MatSubClass([1.0, 1.0]) | o2 = np.dot(o,2.0) No | o=MatSubClass([1.0, 1.0]) | o2 = np.dot(o,o.T) Yes | o=MatSubClass([1.0, 1.0]) | o2 = np.core.multiarray.dot(o,2) Yes | o=MatSubClass([1.0, 1.0]) | o2 = np.core.multiarray.dot(o,2.0) No | o=MatSubClass([1.0, 1.0]) | o2 = np.core.multiarray.dot(o,o.T)

1 0

[ANN] carray 0.5 released
by Francesc Alted 21 Aug '12

21 Aug '12

Announcing carray 0.5 ===================== What's new ---------- carray 0.5 supports completely transparent storage on-disk in addition to memory. That means that everything that can be done with an in-memory container can be done using the disk instead. The advantages of a disk-based container is that your addressable space is much larger than just your available memory. Also, as carray is based on a chunked and compressed data layout based on the super-fast Blosc compression library, and the different cache levels existing in both modern operating systems and the internal carray machinery, the data access speed is very good. The format chosen for the persistence layer is based on the 'bloscpack' library (thanks to Valentin Haenel for his inspiration) and described in 'persistence.rst', although not everything has been implemented yet. You may want to contribute by proposing enhancements to it. See: https://github.com/FrancescAlted/carray/wiki/PersistenceProposal CAVEAT: The bloscpack format is still evolving, so don't trust on forward compatibility of the format, at least until 1.0, where the internal format will be declared frozen. For more detailed info, see the release notes in: https://github.com/FrancescAlted/carray/wiki/Release-0.5 What it is ---------- carray is a chunked container for numerical data. Chunking allows for efficient enlarging/shrinking of data container. In addition, it can also be compressed for reducing memory/disk needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. carray can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr can use optimize the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, with the introduction of a carray/ctable disk-based container (in version 0.5), it can be used for seamlessly performing out-of-core computations. carray comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Resources --------- Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://carray.pytables.org/download Manual: http://carray.pytables.org/docs/manual Home of Blosc compressor: http://blosc.pytables.org User's mail list: carray(a)googlegroups.com http://groups.google.com/group/carray ---- Enjoy! -- Francesc Alted

1 0

A step toward merging odeint and ode
by Fabrice Silva 21 Aug '12

21 Aug '12

I made a pull request [1] to integrate the LSODA solver that is used in odeint into the modular scipy.integrate.ode generic class. In a similar way as for vode, it just wraps the already present lsoda.f file (see .pyf file) and exposes it within an IntegratorBase subclass adjusting the coefficients before calling lsoda. Note that lsoda provide automatic switching between stiff and non-stiff methods, feature that is not present in the available vode integrator. Final note: tests are ok! Regards, [1] https://github.com/scipy/scipy/pull/273 -- Fabrice Silva

2 6

Preventing lossy cast for new float dtypes ?
by David Cournapeau 20 Aug '12

20 Aug '12

Hi, I have started toying with implementing a quad precision dtype for numpy on supported platforms, using the __float128 + quadmath lib from gcc. I have noticed invalid (and unexpected) downcast to long double in some cases, especially for ufuncs (e.g. when I don't define my own ufunc for a given operation). Looking down in numpy ufunc machinery, I can see that the issue is coming from the assumption that long double is the highest precision possible for a float type, and the only way I can 'fix' this is to define kind to a value != 'f' in my dtype definition (in which case I get an expected invalid cast exception). Is there a way to still avoid those casts while keeping the 'f' kind ? thanks, David

2 1

Difference between np.loadtxt depending on whether you supply a file object or a filename
by Andrew Nelson 20 Aug '12

20 Aug '12

Dear list, I observe a difference when I try to load a 2D numpy array from a file object compared to if I supply a filename viz: >>> np.version.version '1.5.1' >>> f=open('fit_theoretical.txt') >>> a=np.loadtxt(f) >>> a.shape (1000,) >>> a=np.loadtxt('fit_theoretical.txt') >>> a.shape (500, 2) This strikes me as unexpected, it's not a documented behaviour. Any ideas? cheers, Andrew. -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 1

Functions for stride manipulations
by Sebastian Berg 20 Aug '12

20 Aug '12

Hey, Inspired by an existing PR into numpy, I created two functions based on stride_tricks which I thought might be useful inside numpy. So if I get any feedback/pointers, I would add some tests and create a PR. The first function rolling_window is to create for every point of the original array, a view of its neighbourhood in arbitrary dimensions with some options to make it easy to use. For example for a 3 dim array: rolling_window(a, (3,3,3), asteps=(3,3,3)).max((3,4,5)) Gives the maximum for all non-overlapping 3x3x3 subarrays and: rolling_window(a, 3, axes=0).max(-1) would create the rolling maximum over all 3 successive values along the 0th axis. Plus a function permute_axes which allows to give a tuple for switching axes and can also merge them. A (10, 4, 3) shaped array would give: permute_axes(a, (2, 1, 0)).shape -> (3, 4, 10) permute_axes(a, (1, (0, 2))).shape -> (4, 30) A copy is created when necessary, this basically allows to do multiple swap_axes and a reshape in a single call. I hope this might be useful... Regards, Sebastian

1 0

Re: [Numpy-discussion] 64bit infrastructure
by Sayth Renshaw 18 Aug '12

18 Aug '12

> > Hi Sayth, > > On Thu, Aug 16, 2012 at 8:51 PM, Sayth Renshaw <flebber.crue(a)gmail.com> wrote: >> Hi >> >> I was just wondering if the current absence of 64bit builds was as a result >> of an infrastructure or funding concern. >> >> I like many now have only 64 bit systems and am using the unofficial builds. >> >> If it was something we could raise funds to resolve, how much? Or is it more >> complex than that? > > Which platform, windows? > > Ondrej > > Yes I have a Windows 7 64Bit install and a ubuntu 64 Bit install. Sayth

2 1

64bit infrastructure
by Sayth Renshaw 17 Aug '12

17 Aug '12

Hi I was just wondering if the current absence of 64bit builds was as a result of an infrastructure or funding concern. I like many now have only 64 bit systems and am using the unofficial builds. If it was something we could raise funds to resolve, how much? Or is it more complex than that? Sayth

3 2

Correlated distributions (?)
by Andrea Gavana 17 Aug '12

17 Aug '12

Hi All, once again, my apologies for a (possibly) very ignorant question, my google-fu is failing me... also because I am not sure of what exactly I should look for. My problem is relatively simple. Let's assume I have two Python objects, A and B, and one of their attributes can assume a value of "True" or "False" depending on the results of a uniform random distribution sample, i.e.: probability_A = 0.95 probability_B = 0.86 A.has_failed = False B.has_failed = False if numpy.random.random() < probability_A: A.has_failed = True if numpy.random.random() < probability_B: B.has_failed = True Now, I know that there is a correlation factor between the failing/not failing of A and the failing/not failing of B. Specifically, If A fails, then B should have 80% more chance of failing, but I have been banging my head to find out how I should modify the "probability_B" number (or the extremes of the uniform distribution, if that makes sense) in order to reflect that correlation. I have been looking at correlated distributions, but it appears that most of the results I have found relate to normal distributions, there is very little about non-normal (and especially uniform) distributions. It's also very likely that I am not looking in the right direction, so I would appreciate any suggestion you may share. Thank you in advance. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/

4 3

Enhanced array destruction needed for C flexible numeric type
by Lee McCuller 17 Aug '12

17 Aug '12

I've been implementing a native C++ code with all of the functionality of the Python Uncertanties package (http://packages.python.org/uncertainties/). Automatic error propagation (and differentiation) is extremely useful, but native speed is needed to use in things like function fitting. I have created the type and successfully bound it to numpy (though still missing a lot of UFuncs), but I have a hangup in that I have no clean way to call a destructor on the numeric type (it is variable length via a held pointer). I could keep it in the array as a python object and get xdecref to do it, but I would much rather A: not have the overhead and B: I really want to keep the GIL free. All of the other PyArray_ArrFuncs which manage the type appear to allow pairing a destructor when memory is overwritten, and constructors are all well supported (like copyswap, setitem and fillwithscalar). I'd like to propose adding a flag for PyArray_Descr.flags NPY_ITEM_USES_DESTRUCTOR and a function PyArray_ArrFuncs.destructor with void destruct(void *data, npy_intp dstride, npy_intp len, void *arr) where data points into the data array to be affected, dstride is the data stride, len is number of affected elements, and arr is the array object containing data. It may be possible to change the call structure of another element when this flag is set (to preserve ABI compatibility for existing types). This flag would by inherited when NPY_FROM_FIELDS is set. Subclassing PyArray_Descr may also allow preservation of the API. The current implementation of NPY_ITEM_REFCOUNT could potentially use this more-general destructor implementation to call xdecref. I could potentialy implement this in a branch, but I'd like to know what other's think of it as well as how best to do it to minimize its effect on the rest of the code and external libraries. For anyone interested in the details of this uncertainties code - My error propagation class are a C++ numeric type, which carry around a list of partial derivatives between new values and all of the values used to compute them. This list is a C++ vector which needs its destructor called. My bindings are hand-written, but should provide a good template for binding to other C++ numeric types. For instance, it would not be hard to extend them also add numpy support for arbitrary precision math types, which would also need their C++ destructors called.

1 0

Slow divide of int64?
by Matthew Brett 17 Aug '12

17 Aug '12

Hi, A friend of mine just pointed out that dividing by int64 is considerably slower than multiplying in numpy: <script> from timeit import timeit import numpy as np import numpy.random as npr sz = (1024,) a32 = npr.randint(1, 5001, sz).astype(np.int32) b32 = npr.randint(1, 5001, sz).astype(np.int32) a64 = a32.astype(np.int64) b64 = b32.astype(np.int64) print 'Mul32', timeit('d = a32 * b32', 'from __main__ import a32, b32') print 'Div32', timeit('d = a32 / b32', 'from __main__ import a32, b32') print 'Mul64', timeit('d = a64 * b64', 'from __main__ import a64, b64') print 'Div64', timeit('d = a64 / b64', 'from __main__ import a64, b64') </script> gives (64 bit Debian Intel system, numpy trunk): Mul32 2.71295905113 Div32 6.61985301971 Mul64 2.78101611137 Div64 22.8217148781 with similar values for numpy 1.5.1. Crude testing with Matlab and Octave suggests they do not seem to have this same difference: >> divtest Mul32 4.300662 Div32 5.638622 Mul64 7.894490 Div64 18.121182 octave:2> divtest Mul32 3.960577 Div32 6.553704 Mul64 7.268324 Div64 13.670760 (files attached) Is there something specific about division in numpy that would cause this slowdown? Cheers, Matthew

3 4

Multidimensional neighbours
by Jose Gomez-Dans 17 Aug '12

17 Aug '12

Hi, I've just come across Travis Oliphant's array version of the game of life here <https://gist.github.com/3353411>. It's a really nice example of how to efficiently find neighbours using numpy and family. However, I wanted to apply these ideas to higher order arrays (eg GRID being three dimensional rather than two dimensional). My previous attempts at this include the (sorry for the ensuing horror!) neighbours = np.array( [ \ # Lower row... x[:-2, :-2, :-2], x[:-2, :-2, 1:-1], x[:-2, :-2, 2: ], x[:-2, 1:-1,:-2], x[:-2, 1:-1, 1:-1], x[:-2, 1:-1, 2:], x[:-2, 2:,:-2], x[:-2, 2:, 1:-1], x[:-2, 2:, 2:], # Middle row x[1:-1, :-2, :-2], x[1:-1, :-2, 1:-1], x[1:-1, :-2, 2: ], x[1:-1, 1:-1,:-2], x[1:-1, 1:-1, 2:], x[1:-1, 2:,:-2], x[1:-1, 2:, 1:-1], x[1:-1, 2:, 2:], # Top row x[2:, :-2, :-2], x[2:, :-2, 1:-1], x[2:, :-2, 2: ], x[2:, 1:-1,:-2], x[2:, 1:-1, 1:-1], x[2:, 1:-1, 2:], x[2:, 2:,:-2], x[2:, 2:, 1:-1], x[2:, 2:, 2:] ] ) I think this works (it's been a while! ;D), but it's a bit messy and the boundary condition is a bit crude. I was wondering whether there's some other nice way of doing it, like demonstrated in the above link. Also, what if you want to have a larger neighbourhood (say 5x5x2 instead of 3x3x3)? While I appreciate these index tricks, I also find them quite mind boggling! Thanks! Jose

3 3

ANN: FiPy 3.0
by Daniel Wheeler 16 Aug '12

16 Aug '12

We are pleased to announce the release of FiPy 3.0. http://www.ctcms.nist.gov/fipy The bump in major version number reflects more on the substantial increase in capabilities and ease of use than it does on a break in compatibility with FiPy 2.x. Few, if any, changes to your existing scripts should be necessary. The significant changes since version 2.1 are: • Coupled and vector equations are now supported. • A more robust mechanism for specifying boundary conditions is now used. • Most Meshes can be partitioned by meshing with Gmsh. • PyAMG and SciPy have been added to the solvers. • FiPy is capable of running under Python 3. • “getter” and “setter” methods have been pervasively changed to Python properties. • The test suite now runs much faster. • Tests can now be run on a full install using fipy.test(). This release addresses 66 tickets. ======================================================================== FiPy is an object oriented, partial differential equation (PDE) solver, written in Python, based on a standard finite volume (FV) approach. The framework has been developed in the Metallurgy Division and Center for Theoretical and Computational Materials Science (CTCMS), in the Material Measurement Laboratory (MML) at the National Institute of Standards and Technology (NIST). The solution of coupled sets of PDEs is ubiquitous to the numerical simulation of science problems. Numerous PDE solvers exist, using a variety of languages and numerical approaches. Many are proprietary, expensive and difficult to customize. As a result, scientists spend considerable resources repeatedly developing limited tools for specific problems. Our approach, combining the FV method and Python, provides a tool that is extensible, powerful and freely available. A significant advantage to Python is the existing suite of tools for array calculations, sparse matrices and data rendering. The FiPy framework includes terms for transient diffusion, convection and standard sources, enabling the solution of arbitrary combinations of coupled elliptic, hyperbolic and parabolic PDEs. Currently implemented models include phase field treatments of polycrystalline, dendritic, and electrochemical phase transformations as well as a level set treatment of the electrodeposition process. -- Daniel Wheeler

1 0

QQ Plot in Python and Noncentral Chisquare
by Yi, Chuang 16 Aug '12

16 Aug '12

Hello, I am a new user of Python. I have a couple of questions that would appreciate your guidance! * QQ Plot in Python: I could not find any functions in either Numpy or Scipy to do QQ Plot of two vectors of data. For example, in Matlab, one could just call qqplot(X,Y), which will generate the QQ plot of vector X against vector Y. Would you please let me know which python package has similar functionality? * Noncentral Chisquare: it looks to me that in Numpy and Scipy, the noncentral chisquare distribution only allows Integer degree of freedom. For example, the following code will produce an error message. In R, however, it also allows non-integer degree of freedom. Would you please let me know which python package has similar functionality? * data = numpy.random.noncentral_chisquare(0.5, 2, 100) * ValueError: df <= 0 Thank you very much, Regards, Chuang ---------------------------------------------------------------------- This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.

3 6

how to use numpy-vendor
by Ondřej Čertík 14 Aug '12

14 Aug '12

Hi, How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do "ldd" and all is clear. Ondrej [1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/

5 9

minor threads-related issue in numpy-release 1.6.2 and numpy-dev
by Aron Ahmadia 14 Aug '12

14 Aug '12

Hi all, Installing numpy 1.6.2 against a Python interpreter built with the --without threads currently fails due to missing references to PyGILState_Ensure and PyGILState_Release. The references appear to be coming from the following code in nditer.c.src: NPY_NO_EXPORT void NpyIter_DebugPrint(NpyIter *iter) { // <snip> PyGILState_STATE gilstate = PyGILState_Ensure(); // <snip> PyGILState_Release(gilstate); } Since this is debugging code, I'm guessing it doesn't get called very frequently, and I could probably just #ifdef it out or use the NPY macros for grabbing the GIL for a non-threaded build: (NPY_ALLOW_C_API and NPY_DISABLE_C_API), but I don't understand why it's grabbing the GIL in the first place. Where is it calling into the interpreter? Does it need the GIL for something else? I'm hesitant to touch this code and issue a pull request until I understand what it's trying to do. Heading on over to the master branch at numpy/numpy, I'm starting to notice more unprotected PyGILState references creeping into the development code. Even the Python developers seem to think that nobody is using --without-threaded, so I'm not going to make a strong case for being more careful, but I do want to point it out in case you want to keep the numpy sources correct for this case. Thanks, Aron

3 3

ANN: SciPy 0.11.0 release candidate 2
by Ralf Gommers 13 Aug '12

13 Aug '12

Hi, I am pleased to announce the availability of the second release candidate of SciPy 0.11.0. For this release many new features have been added, and over 120 tickets and pull requests have been closed. Also noteworthy is that the number of contributors for this release has risen to over 50. Some of the highlights are: - A new module, sparse.csgraph, has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.11.0rc2/, release notes are copied below. For this release candidate all known issues (with the exception of one Qhull issue on Debian, s390x platform) have been solved. In the meantime also OS X 10.8 was released, this RC contains a few build fixes for that platform. If no more serious issues are reported, the final release will be in one week. Cheers, Ralf ========================== SciPy 0.11.0 Release Notes ========================== .. note:: Scipy 0.11.0 is not released yet! .. contents:: SciPy 0.11.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. Highlights of this release are: - A new module has been added which provides a number of common sparse graph algorithms. - New unified interfaces to the existing optimization and root finding functions have been added. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Our development attention will now shift to bug-fix releases on the 0.11.x branch, and on adding new features on the master branch. This release requires Python 2.4-2.7 or 3.1-3.2 and NumPy 1.5.1 or greater. New features ============ Sparse Graph Submodule ---------------------- The new submodule :mod:`scipy.sparse.csgraph` implements a number of efficient graph algorithms for graphs stored as sparse adjacency matrices. Available routines are: - :func:`connected_components` - determine connected components of a graph - :func:`laplacian` - compute the laplacian of a graph - :func:`shortest_path` - compute the shortest path between points on a positive graph - :func:`dijkstra` - use Dijkstra's algorithm for shortest path - :func:`floyd_warshall` - use the Floyd-Warshall algorithm for shortest path - :func:`breadth_first_order` - compute a breadth-first order of nodes - :func:`depth_first_order` - compute a depth-first order of nodes - :func:`breadth_first_tree` - construct the breadth-first tree from a given node - :func:`depth_first_tree` - construct a depth-first tree from a given node - :func:`minimum_spanning_tree` - construct the minimum spanning tree of a graph ``scipy.optimize`` improvements ------------------------------- The optimize module has received a lot of attention this release. In addition to added tests, documentation improvements, bug fixes and code clean-up, the following improvements were made: - A unified interface to minimizers of univariate and multivariate functions has been added. - A unified interface to root finding algorithms for multivariate functions has been added. - The L-BFGS-B algorithm has been updated to version 3.0. Unified interfaces to minimizers ```````````````````````````````` Two new functions ``scipy.optimize.minimize`` and ``scipy.optimize.minimize_scalar`` were added to provide a common interface to minimizers of multivariate and univariate functions respectively. For multivariate functions, ``scipy.optimize.minimize`` provides an interface to methods for unconstrained optimization (`fmin`, `fmin_powell`, `fmin_cg`, `fmin_ncg`, `fmin_bfgs` and `anneal`) or constrained optimization (`fmin_l_bfgs_b`, `fmin_tnc`, `fmin_cobyla` and `fmin_slsqp`). For univariate functions, ``scipy.optimize.minimize_scalar`` provides an interface to methods for unconstrained and bounded optimization (`brent`, `golden`, `fminbound`). This allows for easier comparing and switching between solvers. Unified interface to root finding algorithms ```````````````````````````````````````````` The new function ``scipy.optimize.root`` provides a common interface to root finding algorithms for multivariate functions, embeding `fsolve`, `leastsq` and `nonlin` solvers. ``scipy.linalg`` improvements ----------------------------- New matrix equation solvers ``````````````````````````` Solvers for the Sylvester equation (``scipy.linalg.solve_sylvester``, discrete and continuous Lyapunov equations (``scipy.linalg.solve_lyapunov``, ``scipy.linalg.solve_discrete_lyapunov``) and discrete and continuous algebraic Riccati equations (``scipy.linalg.solve_continuous_are``, ``scipy.linalg.solve_discrete_are``) have been added to ``scipy.linalg``. These solvers are often used in the field of linear control theory. QZ and QR Decomposition ```````````````````````` It is now possible to calculate the QZ, or Generalized Schur, decomposition using ``scipy.linalg.qz``. This function wraps the LAPACK routines sgges, dgges, cgges, and zgges. The function ``scipy.linalg.qr_multiply``, which allows efficient computation of the matrix product of Q (from a QR decompostion) and a vector, has been added. Pascal matrices ``````````````` A function for creating Pascal matrices, ``scipy.linalg.pascal``, was added. Sparse matrix construction and operations ----------------------------------------- Two new functions, ``scipy.sparse.diags`` and ``scipy.sparse.block_diag``, were added to easily construct diagonal and block-diagonal sparse matrices respectively. ``scipy.sparse.csc_matrix`` and ``csr_matrix`` now support the operations ``sin``, ``tan``, ``arcsin``, ``arctan``, ``sinh``, ``tanh``, ``arcsinh``, ``arctanh``, ``rint``, ``sign``, ``expm1``, ``log1p``, ``deg2rad``, ``rad2deg``, ``floor``, ``ceil`` and ``trunc``. Previously, these operations had to be performed by operating on the matrices' ``data`` attribute. LSMR iterative solver --------------------- LSMR, an iterative method for solving (sparse) linear and linear least-squares systems, was added as ``scipy.sparse.linalg.lsmr``. Discrete Sine Transform ----------------------- Bindings for the discrete sine transform functions have been added to ``scipy.fftpack``. ``scipy.interpolate`` improvements ---------------------------------- For interpolation in spherical coordinates, the three classes ``scipy.interpolate.SmoothSphereBivariateSpline``, ``scipy.interpolate.LSQSphereBivariateSpline``, and ``scipy.interpolate.RectSphereBivariateSpline`` have been added. Binned statistics (``scipy.stats``) ----------------------------------- The stats module has gained functions to do binned statistics, which are a generalization of histograms, in 1-D, 2-D and multiple dimensions: ``scipy.stats.binned_statistic``, ``scipy.stats.binned_statistic_2d`` and ``scipy.stats.binned_statistic_dd``. Deprecated features =================== ``scipy.sparse.cs_graph_components`` has been made a part of the sparse graph submodule, and renamed to ``scipy.sparse.csgraph.connected_components``. Calling the former routine will result in a deprecation warning. ``scipy.misc.radon`` has been deprecated. A more full-featured radon transform can be found in scikits-image. ``scipy.io.save_as_module`` has been deprecated. A better way to save multiple Numpy arrays is the ``numpy.savez`` function. The `xa` and `xb` parameters for all distributions in ``scipy.stats.distributions`` already weren't used; they have now been deprecated. Backwards incompatible changes ============================== Removal of ``scipy.maxentropy`` ------------------------------- The ``scipy.maxentropy`` module, which was deprecated in the 0.10.0 release, has been removed. Logistic regression in scikits.learn is a good and modern alternative for this functionality. Minor change in behavior of ``splev`` ------------------------------------- The spline evaluation function now behaves similarly to ``interp1d`` for size-1 arrays. Previous behavior:: >>> from scipy.interpolate import splev, splrep, interp1d >>> x = [1,2,3,4,5] >>> y = [4,5,6,7,8] >>> tck = splrep(x, y) >>> splev([1], tck) 4. >>> splev(1, tck) 4. Corrected behavior:: >>> splev([1], tck) array([ 4.]) >>> splev(1, tck) array(4.) This affects also the ``UnivariateSpline`` classes. Behavior of ``scipy.integrate.complex_ode`` ------------------------------------------- The behavior of the ``y`` attribute of ``complex_ode`` is changed. Previously, it expressed the complex-valued solution in the form:: z = ode.y[::2] + 1j * ode.y[1::2] Now, it is directly the complex-valued solution:: z = ode.y Minor change in behavior of T-tests ----------------------------------- The T-tests ``scipy.stats.ttest_ind``, ``scipy.stats.ttest_rel`` and ``scipy.stats.ttest_1samp`` have been changed so that 0 / 0 now returns NaN instead of 1. Other changes ============= The SuperLU sources in ``scipy.sparse.linalg`` have been updated to version 4.3 from upstream. The function ``scipy.signal.bode``, which calculates magnitude and phase data for a continuous-time system, has been added. The two-sample T-test ``scipy.stats.ttest_ind`` gained an option to compare samples with unequal variances, i.e. Welch's T-test. ``scipy.misc.logsumexp`` now takes an optional ``axis`` keyword argument. Authors ======= This release contains work by the following people (contributed at least one patch to this release, names in alphabetical order): * Jeff Armstrong * Chad Baker * Brandon Beacher + * behrisch + * borishim + * Matthew Brett * Lars Buitinck * Luis Pedro Coelho + * Johann Cohen-Tanugi * David Cournapeau * dougal + * Ali Ebrahim + * endolith + * Bjørn Forsman + * Robert Gantner + * Sebastian Gassner + * Christoph Gohlke * Ralf Gommers * Yaroslav Halchenko * Charles Harris * Jonathan Helmus + * Andreas Hilboll + * Marc Honnorat + * Jonathan Hunt + * Maxim Ivanov + * Thouis (Ray) Jones * Christopher Kuster + * Josh Lawrence + * Denis Laxalde + * Travis Oliphant * Joonas Paalasmaa + * Fabian Pedregosa * Josef Perktold * Gavin Price + * Jim Radford + * Andrew Schein + * Skipper Seabold * Jacob Silterra + * Scott Sinclair * Alexis Tabary + * Martin Teichmann * Matt Terry + * Nicky van Foreest + * Jacob Vanderplas * Patrick Varilly + * Pauli Virtanen * Nils Wagner + * Darryl Wally + * Stefan van der Walt * Liming Wang + * David Warde-Farley + * Warren Weckesser * Sebastian Werk + * Mike Wimmer + * Tony S Yu + A total of 55 people contributed to this release. People with a "+" by their names contributed a patch for the first time.

1 0

Vagrant VM for building NumPy (1.7.x) Windows binaries
by Ondřej Čertík 13 Aug '12

13 Aug '12

Hi, I've created this repository: https://github.com/certik/numpy-vendor which uses Vagrant and Fabric to fully automate the setup & creation of NumPy binaries for Windows. The setup is especially tricky, I've thought several times already that I nailed it, and then always new things pop up. One can of course install things directly in Ubuntu, but it's tricky, there are a lot of things that can go wrong. The above approach should be 100% reproducible. So hopefully this repository will be useful for somebody new (like I am) to numpy releases. Also my hope is that more people can help out with the release just by running it on their machines and/or sending PRs against this repository. All essential logic is in these two files: https://github.com/certik/numpy-vendor/blob/master/fabfile.py https://github.com/certik/numpy-vendor/blob/master/setup-wine.sh The setup-wine.sh can be used directly in Ubuntu as well (it erases ~/.wine and reinstalls it). Times: on my computer and internet speed it takes a little less than 1h to prepare the VM (it needs to download the 300MB Ubuntu, then about 600MB of Latex, and few MB of other stuff, and then about 12 minutes of configuring), and about 0.5 for each binary, so about 6h total (the docs only take a few minutes). After doing all this, I just realized that I forgot to build numpy against atlas. So I will need to fix that. But apart from that, it should be ok. Travis, should I upload the binaries to sourceforge and do the beta release with this? Or should we wait until I fix the atlas. Also I need to do the Mac binaries. I'll have the Win binaries in about 6h from now, I am running one final build from scratch now. My apologies that it took me longer than expected. Ondrej P.S. Here is a PR for review (you need to apply it manually if you want the above to work): https://github.com/numpy/numpy/pull/384

3 4

Iterating over a given axis with buffering
by Sergio Pascual 13 Aug '12

13 Aug '12

Hi, I have a 3d array of bitswapped data, dimensions 2000x2000x20. I want to apply a function over the third axis to obtain a 2000x2000 array. This is what numpy.apply_along_axis does, but I need a faster version, so I'm writing it in C. I started by writing a generic ufunc version of the code. It is not what I need, because my particular function has several free parameters, but with this version I could get a hint of the performance. The ufunc was nice, because it handled the bitswapped data, and naturally the innermost loop was over the third axis. So I tried to write the final version using NpyIter. The only way I found to do the innermost loop over the third axis was passing the flag NPY_ITER_MULTI_INDEX, but then I can't use NPY_ITER_BUFFERED to handle my bitswapped, at least in numpy 1.6 they couldn't be used simultaneously. Any hint on how I can do this? Regards, Sergio

1 0

Re: [Numpy-discussion] Second try: possible bug in assignment to complex array
by Mark Bakker 11 Aug '12

11 Aug '12

Shall I file a bug report? Or is this fairly easy to fix? Mark > > On Fri, Aug 10, 2012 at 11:41 AM, <josef.pktd(a)gmail.com> wrote: > > > > > > > On Fri, Aug 10, 2012 at 10:00 AM, Travis Oliphant <travis(a)continuum.io > >wrote: > > > >> > >> On Aug 10, 2012, at 5:37 AM, Paul Anton Letnes wrote: > >> > >> > > >> > > >> > On 10. aug. 2012, at 09:54, Mark Bakker wrote: > >> > > >> >> I am giving this a second try. Can anybody help me out? > >> >> > >> >> I think there is a problem with assigning a 1D complex array of > length > >> one > >> >> to a position in another complex array. > >> >> > >> >> Example: > >> >> > >> >> a = ones(1,'D') > >> >> b = ones(1,'D') > >> >> a[0] = b > >> >> > >> > --------------------------------------------------------------------------- > >> >> TypeError Traceback (most recent call > >> last) > >> >> <ipython-input-37-0c4fc6d780e3> in <module>() > >> >> ----> 1 a[0] = b > >> >> > >> >> TypeError: can't convert complex to float > >> >> > >> >> This works correctly when a and b are real arrays: > >> >> > >> >> a = ones(1) > >> >> b = ones(1) > >> >> a[0] = b > >> >> > >> >> Bug or feature? > >> > > >> > The exact same thing happens on OS X 10.7.4, python 2.7.3, numpy > 1.6.1. > >> > > >> > Looks like a bug to me - or at least very surprising behavior. > >> > >> This is definitely an inconsistency. The error seems more correct > >> (though the error message needs improvement). > >> > >> Can someone try this on NumPy 1.5 and see if this inconsistency existed > >> there as well. > >> > > > > >>> np.__version__ > > '1.5.1' > > > > >>> a = np.ones(1,'D') > > >>> b = np.ones(1,'D') > > >>> a[0] = b > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > > > TypeError: can't convert complex to float > > >>> a = np.ones(1) > > >>> b = np.ones(1) > > >>> a[0] = b > > > > and > > >>> a = np.ones(1,'D') > >>> b = 2*np.ones(1) > >>> a[0] = b > >>> a > array([ 2.+0.j]) > >>> c = 3*np.ones(1, int) > >>> a[0] = c > >>> a > array([ 3.+0.j]) > > > > > > > Josef > > > > > >> Thanks, > >> > >> -Travis > >> > >> > > >> > Paul > >> > _______________________________________________ > >> > NumPy-Discussion mailing list > >> > NumPy-Discussion(a)scipy.org > >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion(a)scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > >

1 0