NumPy-Discussion April 2009

numpy-discussion@python.org

112 participants
126 discussions

suggestion for generalizing numpy functions
by Darren Dale 13 Sep '09

13 Sep '09

I spent some time over the weekend fixing a few bugs in numpy that were exposed when attempting to use ufuncs with ndarray subclasses. It got me thinking that, with relatively little work, numpy's functions could be made to be more general. For example, the numpy.ma module redefines many of the standard ufuncs in order to do some preprocessing before the builtin ufunc is called. Likewise, in the units/quantities package I have been working on, I would like to perform a dimensional analysis to make sure an operation is allowed before I call a ufunc that might change data in place. Imagine an ndarray subclass with methods like __gfunc_pre__ and __gfunc_post__. __gfunc_pre__ could accept the context that is currently provided to __array_wrap__ (the inputs and the function called), perform whatever preprocessing is desired, and maybe return a dictionary containing metadata. Numpy functions could then be wrapped with a decorator that 1) calls __gfunc_pre__ and obtain any metadata that is returned 2) calls the wrapped functions, and then 3) calls __gfunc_post__, which might be very similar to __array_wrap__ except that it would also accept the metadata created by __gfunc_pre__. In cases where the routines to be called by __gfunc_pre__ and _post__ depend on what function is called, the the subclass could implement routines and store them in a dictionary-like object that is keyed using the function called. I have been exploring this approach with Quantities and it seems to work well. For example: def __gfunc_pre__(self, gfunc, *args): try: return gfunc_pre_registry[gfunc](*args) except KeyError: return {} I think such an approach for generalizing numpy's functions could be implemented without being disruptive to the existing __array_wrap__ framework. The decorator would attempt to identify an input or output array to use to call __gfunc_pre__ and _post__. If it finds them, it uses them. If it doesnt find them, no harm done, the existing __array_wrap__ mechanisms are still in place if the wrapped function is a ufunc. One other nice feature: the metadata that is returned by __gfunc_pre__ could contain an optional flag that the decorator attempts to pass to the wrapped function so that __gfunc_pre__ and _post are not called for any decorated internal functions. That way the subclass could specify that __gfunc_pre__ and _post should be called only for the outer-most function. Comments? Darren

5 24

Masking an array with another array
by Gökhan SEVER 12 Aug '09

12 Aug '09

Hello, Could you please give me some hints about how to mask an array using another arrays like in the following example. In [14]: a = arange(5) In [15]: a Out[15]: array([0, 1, 2, 3, 4]) and my secondary array is "b" In [16]: b = array([2,3]) What I want to do is to mask a with b values and get an array of: array([False, False, True, True, False], dtype=bool) That is just an manually created array. I still don't know how to do this programmatically in Pythonic fashion or numpy's masked array functions. Thank you. Gökhan

6 10

ANN: line_profiler 1.0b2 released
by Robert Kern 20 Jul '09

20 Jul '09

http://pypi.python.org/pypi/line_profiler/ http://packages.python.org/line_profiler/ This release fixes the "negative timings" issue on Windows. Future announcements will occur on python-announce. I just wanted to make sure my earliest users here who ran into this bug are aware of the fix. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

4 6

ANNOUNCE: ETS 3.2.0 Released
by Dave Peterson 04 Jun '09

04 Jun '09

Hello, I'm pleased to announce that Enthought Tool Suite (ETS) version 3.2.0 has been tagged and released! Source distributions (.tar.gz) have been uploaded to PyPi, and Windows binaries will be follow shortly. A full install of ETS can be done using Setuptools via a command like: easy_install -U "ets[nonets] >= 3.2.0" NOTE 1: Users of an old ETS release will need to first uninstall prior to installing the new ETS. NOTE 2: If you get a 'SandboxViolation' error, simply re-run the command again -- it may take multiple invocations to get everything installed. (This error appears to be a long-standing incompatibility between numpy.distutils and setuptools.) Please see below for a list of what's new in this release. What Is ETS? =========== The Enthought Tool Suite (ETS) is a collection of components developed by Enthought and the open-source community, which we use every day to construct custom scientific applications. It includes a wide variety of components, including: * an extensible application framework * application building blocks * 2-D and 3-D graphics libraries * scientific and math libraries * developer tools The cornerstone on which these tools rest is the Traits package, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information on ETS is available from the development home page: http://code.enthought.com/projects/index.php Changelog ========= ETS 3.2.0 is a feature-added update to ETS 3.1.0, including numerous bug-fixes. Some of the notable changes include: Chaco ----- * Domain limits - Mappers now can declare the "limits" of their valid domain. PanTool and ZoomTool respect these limits. (pwang) * Adding "hide_grids" parameter to Plot.img_plot() and Plot.contour_plot() so users can override the default behavior of hiding grids. (pwang) * Refactored examples to declare a Demo object so they can be be run with the demo.py example launcher. (vibha) * Adding chaco.overlays package with some canned SVG overlays. (bhendrix) * DragZoom now can scale both X and Y axes independently corresponding to the mouse cursor motion along the X and Y axes (similar to the zoom behavior in Matplotlib). (pwang) * New Examples: * world map (bhendrix) * more financial plots (pwang) * scatter_toggle (pwang) * stacked_axis (pwang) * Fixing the chaco.scales TimeFormatter to use the built-in localtime() instead of the one in the safetime.py module due to Daylight Savings Time issues with timedelta. (r23231, pwang) * Improved behavior of ScatterPlot when it doesn't get the type of metadata it expects in its "selections" and "selection_masks" metadata keys (r23121, pwang) * Setting the .range2d attribute on GridMapper now properly sets the two DataRange1D instances of its sub-mappers. (r23119, pwang) * ScatterPlot.map_index() now respects the index_only flag (r23060, pwang) * Fixed occasional traceback/bug in LinePlot that occurred when data was completely outside the visible range (r23059, pwang) * Implementing is_in() on legends to account for padding and alignment (caused by tools that move the legend) (r23052, bhendrix) * Legend behaves properly when there are no plots to display (r23012, judah) * Fixed LogScale in the chaco.scales package to correctly handle the case when the length of the interval is less than a decade (r22907, warren.weckesser) * Fixed traceback when calling copy_traits() on a DataView (r22894, vibha) * Scatter plots generated by Plot.plot() now properly use the "auto" coloring feature of Plot. (r22727, pwang) * Reduced the size of screenshots in the user manual. (r22720, rkern) Mayavi ------ * 17, 18 March, 2009 (PR): * NEW: A simple example to show how one can use TVTK’s visual module with mlab. [23250] * BUG: The size trait was being overridden and was different from the parent causing a bug with resizing the viewer. [23243] * 15 March, 2009 (GV): * ENH: Add a volume factory to mlab that knows how to set color, vmin and vmax for the volume module [23221]. * 14 March, 2009 (PR): * API/TEST: Added a new testing entry point: ‘mayavi -t’ now runs tests in separate process, for isolation. Added enthought.mayavi.api.test to allow for simple testing from the interpreter [23195]...[23200], [23213], [23214], [23223]. * BUG: The volume module was directly importing the wx_gradient_editor leading to an import error when no wxPython is available. This has been tested and fixed. Thanks to Christoph Bohme for reporting this issue. [23191] * 14 March, 2009 (GV): * BUG: [mlab]: fix positioning for titles [23194], and opacity for titles and text [23193]. * ENH: Add the mlab_source attribute on all objects created by mlab, when possible [23201], [23209]. * ENH: Add a message to help the first-time user, using the new banner feature of the IPython shell view [23208]. * 13 March, 2009 (PR): * NEW/API: Adding a powerful TCP/UDP server for scripting mayavi via the network. This is available in enthought.mayavi.tools.server and is fully documented. It uses twisted and currently only works with wxPython. It is completely insecure though since it allows a remote user to do practically anything from mayavi. * 13 March, 2009 (GV) * API: rename mlab.orientationaxes to mlab.orientation_axes [23184] * 11 March, 2009 (GV) * API: Expose ‘traverse’ in mlab.pipeline [23181] * 10 March, 2009 (PR) * BUG: Fixed a subtle bug that affected the ImagePlaneWidget. This happened because the scalar_type of the output data from the VTKDataSource was not being set correctly. Getting the range of any input scalars also seems to silence warnings from VTK. This should hopefully fix issues with the use of the IPW with multiple scalars. I’ve added two tests for this, one is an integration test since those errors really show up only when the display is used. The other is a traditional unittest. [23166] * 08 March, 2009 (GV) * ENH: Raises an error when the user passes to mlab an array with infinite values [23150] * 07 March, 2009 (PR) * BUG: A subtle bug with a really gross error in the GridPlane component, I was using the extents when I should really have been looking at the dimensions. The extract grid filter was also not flushing the data changes downstream leading to errors that are also fixed now. These errors would manifest when you use an ExtractGrid to select a VOI or a sample rate and then used a grid plane down stream causing very wierd and incorrect rendering of the grid plane (thanks to conflation of extents and dimensions). This bug was seen at NAL for a while and also reported by Fred with a nice CME. The CME was then converted to a nice unittest by Suyog and then improved. Thanks to them all. [23146] * 28 February, 2009 (PR) * BUG: Fixed some issues reported by Ondrej Certik regarding the use Of mlab.options.offscreen, mlab.options.backend = ‘test’, removed cruft from earlier ‘null’ backend, fixed bug with incorrect imports, add_dataset set no longer adds one new null engine each time figure=False is passed, added test case for the options.backend test. [23088] * 23 February, 2009 (PR) * ENH: Updating show so that it supports a stop keyword argument that pops up a little UI that lets the user stop the mainloop temporarily and continue using Python [23049] * 21 February, 2009 (GV) * ENH: Add a richer view for the pipeline to the MayaviScene [23035] * ENH: Add safegards to capture wrong triangle array sizes in mlab.triangular_mesh_source. [23037] * 21 February, 2009 (PR) * ENH: Making the transform data filter recordable. [23033] * NEW: A simple animator class to make it relatively to create animations. [23036] [23039] * 20 February, 2009 (PR) * ENH: Added readers for various image file formats, poly data readers and unstructured grid readers. These include DICOM, GESigna, DEM, MetaImage (mha,mhd) MINC, AVSucd, GAMBIT, Exodus, STL, Points, Particle, PLY, PDB, SLC, OBJ, Facet and BYU files. Also added several tests for most of this functionality along with small data files. These are additions from PR’s project staff, Suyog Jain and Sreekanth Ravindran. [23013] * ENH: We now change the default so the ImagePlaneWidget does not control the LUT. Also made the IPW recordable. [23011] * 18 February, 2009 (GV) * ENH: Add a preference manager view for editing preferences outside envisage [22998] * 08 February, 2009 (GV) * ENH: Center the glyphs created by barchart on the data points, as mentioned by Rauli Ruohonen [22906] * 29 January, 2009 (GV) * ENH: Make it possible to avoid redraws with mlab by using mlab.gcf().scene.disable_render = True [22869] * 28 January, 2009 (PR and GV) * ENH: Make the mlab.pipeline.user_defined factory function usable to add arbitrary filters on the pipeline. [22867], [22865] * 11 January, 2009 (GV) * ENH: Make mlab.imshow use the ImageActor. Enhance the ImageActor to map scalars to colors when needed. [22816] Traits ------ * Fixed a bug whereby faulty error handling in the PyProtocols Pyrex speedup code keeps references to tracebacks that have been handled. In so doing, clean up the same code such that it can be used with a modern Pyrex release (a bare raise can no longer be used outside of an except: clause). * RangeEditor factory now supports a 'logslider' mode: Thanks to Matthew Turk for the patch * TabularEditor factory now supports editing of all columns: Thanks to Didrik Pinte for the patch * DateEditor factory in 'custom' style now supports multi-select feature. * DateEditor and TimeEditor now support the 'readonly' style. * Fixed a bug in the ArrayEditor factory that was causing multiple trait change events to get fired when the underlying array is changed externally to the editor: Thanks to Matthew Turk for he patch. * Fixed a circular import error in Color, Font and RGBColor traits * Fixed a bug in the factory for ArrayViewEditor so it now calls the toolkit backend-specific editor TraitsBackendWX --------------- * RangeEditor now supports a 'logslider' mode: Thanks to Matthew Turk for the patch * TabularEditor now supports editing of all columns: Thanks to Didrik Pinte for the patch * DateEditor in 'custom' style now supports multi-select feature. * DateEditor and TimeEditor now support the 'readonly' style. * Added a trait to the wx pyface workbench View to indicate if the view dock window should be closeable. * Fixed the DirectoryEditor to popup the correct file dialog (thanks to Luca Fasano and Phil Thompson) * Fixed a circular import error in Color, Font and RGBColor traits * Fixed a bug in the ColorEditor that was causing the revert action to not work correctly. * Fixed a bug that caused a traceback when trying to undock a pyface dock window * Fixed a bug in the 'livemodal' view that caused the UI to become unresponsive if the 'updated' event was fired on the contained view. * Fixed bugs in ListEditor (notebook style) that caused a loss of sync between the 'selected' trait and the activated dock window. TraitsBackendQt --------------- * RangeEditor now supports a 'logslider' mode: Thanks to Matthew Turk for the patch * Fixed the DirectoryEditor to popup the correct file dialog (thanks to Luca Fasano and Phil Thompson)

3 2

setmember1d_nu
by Robert Cimrman 01 Jun '09

01 Jun '09

Hi all, I have added to the ticket [1] a script that compares the proposed setmember1d_nu() implementations of Neil and Kim. Comments are welcome! [1] http://projects.scipy.org/numpy/ticket/1036 r.

2 2

Leopard install
by Stuart Edwards 16 May '09

16 May '09

Hi I am trying to install numpy 1.3.0 on Leopard 10.5.6 and at the point in the install process where I select a destination, my boot disc is excluded with the message: " You cannot install numpy 1.3.0 on this volume. numpy requires System Python 2.5 to install." I'm not sure what 'System Python 2.5' is as compared to 'Python 2.5' but in the terminal when I type 'python' I get: "Python 2.5.1 (r251:54863, Jan 13 2009, 10:26:13) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin" so the Python 2.5 requirement seems to be met. Any ideas on what is happening here and why the installer can't see my python installation? (I notice that this issue was raised 3/28 also, but no resolution yet) Thanks for any assistance Stu

5 4

Distance Formula on an Array
by Ian Mallett 12 May '09

12 May '09

Hi, I have an array sized n*3. Each three-component is a 3D position. Given another 3D position, how is the distance between it and every three-component in the array found with NumPy? So, for example, if the array is: [[0,0,0],[0,1,0],[0,0,3]] And the position is: [0,4,0] I need this array out: [4,3,5] (Just a simple Pythagorean Distance Formula) Ideas? Thanks, Ian

6 22

Re: [Numpy-discussion] Numpy Trac site redirecting in a loop?
by Ken Basye 08 May '09

08 May '09

Hi Pauli, and thanks. I tried clearing the cache (no luck), and then logging in from a completely different machine with a completely re-initialized browser. I can get to the site, but as soon as I try to log in, the same thing (redirect loop) happens. I should point out that just before this happened I had the site reset my password. The URL it seems to be trying to get to after the login dialog is dismissed is: http://projects.scipy.org/numpy/prefs/account Back on the original machine, I was able to get the main page loading again by clearing all the cookies and stored passwords from scipy.org, but the same thing happened when I tried to log into the site. Thanks again, Ken > > Just now when I tried to access http://projects.scipy.org/numpy, > > Firefox complains (after a bit) that there's a redirect loop. Is anyone > > else seeing this problem, and/or does anyone know how I can fix my side > > of things to get me to the site? I don't see this. Can you try emptying your browser's cache? -- Pauli Virtanen Wed, 15 Apr 2009 17:21:59 -0400, Ken Basye wrote:

4 6

Porting strategy for py3k
by David Cournapeau 04 May '09

04 May '09

Hi, I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed - some exceptions constructions are not allowed I think a first step would be to add a compatibility module to remove any use of print/reduce in the numpy codebase (should not be too difficult). For reduce, we could import functools from python code (functools is not available in python 2.4). Unless someone has an objection, I will import functools code, make it compile for python 2.4 and above, and remove all builtin reduce calls in numpy. Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years). cheers, David

9 16

Timing array construction
by Mark Janikas 30 Apr '09

30 Apr '09

Hello All, I was exploring some different ways to concatenate arrays, and using "c_" is the fastest by far. Is there a difference I am missing that can account for the huge disparity? Obviously the "zip" function makes the "as array" and "array" calls slower, but the same arguments (xCoords, yCoords) are being passed to the methods... so if there is no difference in the outputs (there doesn't appear to be) then what reason would I have to use "array" or "as array" in this context? Thanks so much ahead of time.. MJ ############## Snippet ################### import numpy as NUM def useAsArray(xCoords, yCoords): return NUM.asarray(zip(xCoords, yCoords)) def useArray(xCoords, yCoords): return NUM.array(zip(xCoords, yCoords)) def useC(xCoords, yCoords): return NUM.c_[xCoords, yCoords] if __name__ == "__main__": from timeit import Timer import numpy.random as RAND import collections as COLL resAsArray = COLL.defaultdict(float) resArray = COLL.defaultdict(float) resMat = COLL.defaultdict(float) numTests = 0.0 sameTests = 0.0 N = [100, 200, 400, 800, 1600, 3200, 6400, 12800] for i in N: print "Time Join List into Array for N = " + str(i) xCoords = RAND.normal(10, 1, i) yCoords = RAND.normal(10, 1, i) statement = 'from __main__ import xCoords, yCoords, useAsArray' t1 = Timer('useAsArray(xCoords, yCoords)', statement) resAsArray[i] = t1.timeit(10) statement = 'from __main__ import xCoords, yCoords, useArray' t2 = Timer('useArray(xCoords, yCoords)', statement) resArray[i] = t2.timeit(10) statement = 'from __main__ import xCoords, yCoords, useC' t3 = Timer('useC(xCoords, yCoords)', statement) resMat[i] = t3.timeit(10) for n in N: print "%i, %0.4f, %0.4f, %0.4f" % (n, resAsArray[n], resArray[n], resMat[n]) ############################################################### RESULT N, useAsArray, useArray, useC 100, 0.0066, 0.0065, 0.0007 200, 0.0137, 0.0140, 0.0008 400, 0.0277, 0.0288, 0.0007 800, 0.0579, 0.0577, 0.0008 1600, 0.1175, 0.1289, 0.0009 3200, 0.2291, 0.2309, 0.0012 6400, 0.4561, 0.4564, 0.0013 12800, 0.9218, 0.9122, 0.0019 Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas(a)esri.com<mailto:mjanikas@esri.com>

4 5

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion April 2009