Mailman 3 July 2009 - NumPy-Discussion

__eq__ method for recarray returns recarray
by Dan Yamins 04 Jul '09

04 Jul '09

If I have two recarrays with the same len and column headers, the __eq__ method returns the rich comparison, which is great. E.g. In [20]: x = np.rec.fromrecords([(1,2,'dd',.3),(33,2,'y',2.2),(2,3,'a',21.4),(3,4,'b',33.2)],names=['A','B','C','D']) In [21]: y = np.rec.fromrecords([(1,2,'dd',.3),(33,2,'y',2.2),(2,3,'a',21.4),(3,4,'b',33.2)],names=['A','B','C','D']) In [22]: x == y Out[22]: rec.array([ True, True, True, True], dtype=bool) But notice that the returned object is a recarray, not an array of bools. Why is this, and what is the purpose of having it this way? Similarly, if I subclass recarray, and say, in my subclass attach some attributes to x, then these attributes are attached to the result of the rich comparison. E.g. suppose I have a subclass of recarray called "NewRecarray" to which I attach an attribute .info. Then x = NewRecarray(data) y = NewRecarray(data) z = x == y Then z is a NewRecarray object and z.info = x.info. Is this the expected / proper behavior? Is there something wrong with the way I've subclassed recarray? Thanks, Dan

1 0

Using loadtxt to read in mixed data types
by Peter Kelley 03 Jul '09

03 Jul '09

Hey Everyone, I am reading in a file of columns with mixed data types, and the number of columns can vary and their format is inputted by the user. So I came up with this: dt=dtype({'names': [x for x in event_fields], 'formats': [b for b in event_format]}) eventArray = loadtxt(behEventFile,dt) where event_format is ['long', 'int', 'int', 'int', 'int', 'int', 'str', 'str', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int'] I get TypeError: data type not understood, and I think it is because the event format is a list of strings not data types. Does anyone have know how to convert the list of strings into the data types for dtype. -Peter

4 4

ndarray from column data
by Elaine Angelino 03 Jul '09

03 Jul '09

Hi there -- Is there a fast way to make a numpy ndarray from column data? For example, suppose I want to make an ndarray with 2 rows and 3 columns of different data types based on the following column data: C0 = [1,2] C1 = ['a','b'] C2 = [3.3,4.4] I could create an empty ndarray and fill the columns one by one: X = numpy.core.ndarray((2,), dtype='<i4,|S1,<f8') X['f0'] = C0 X['f1'] = C1 X['f2'] = C2 The result is the same as: X = numpy.array([(1,'a',3.3), (2,'b',4.4)], dtype='<i4,|S1,<f8') but I would like to make X directly from the column data. [ I know that numpy.core.records.fromarrays will produce a numpy recarray from column data, but this of course is a recarray and not a ndarray! For ex: X = numpy.numpy.core.records.fromarrays([C0,C1,C2]) ] Thanks for any help, Elaine

4 4

Numpy complex types, packing and C99
by David Cournapeau 03 Jul '09

03 Jul '09

Hi, I would like to add an explicit configuration test to check that our complex type is compatible with the C99 complex type (when available). Is this ok ? As currently defined, complex c types (npy_cfloat, etc...) are not defined in a way such as they are binary compatible with the C99 complex type. Strictly speaking, packing the complex type is an ABI break, but we already make the assumption in ufunc, so we would have wrong results/crashes currently if it were not packed, so I believe the check is harmless by itself. The rationale is that I would like to add support for complex math in npy_math (cabs, cexp, etc...). As I would like to use the C99 complex type (when available), this requires that numpy complex type is ABI compatible with C99 type. cheers, David

4 10

Multi thread loading data
by Mag Gam 02 Jul '09

02 Jul '09

Is it possible to use loadtxt in a mult thread way? Basically, I want to process a very large CSV file (100+ million records) and instead of loading thousand elements into a buffer process and then load another 1 thousand elements and process and so on... I was wondering if there is a technique where I can use multiple processors to do this faster. TIA

3 5

pep-3118 extended struct format parser
by Sebastien Binet 02 Jul '09

02 Jul '09

hi there, at last scipy'08 sprint, somebody (apologies for my brain fade) was working on being able to parse the extended struct format string so one could do: Nested structure :: struct { int ival; struct { unsigned short sval; unsigned char bval; unsigned char cval; } sub; } """i:ival: T{ H:sval: B:bval: B:cval: }:sub: """ Nested array :: struct { int ival; double data[16*4]; } """i:ival: (16,4)d:data: """ is this available somewhere ? (being it in pure python or fast-C implemented, I don't really care for the moment :) ) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay #########################################

1 0

make html
by Nils Wagner 01 Jul '09

01 Jul '09

Hi all, I am using >>> numpy.__version__ '1.4.0.dev7094' make html yields /home/nwagner/svn/numpy/doc/source/reference/generated/numpy.trunc.rst:: WARNING: document isn't included in any toctree done preparing documents... done Exception occurred: 2%] reference/generalized_ufuncs ures File "/home/nwagner/local/lib64/python2.6/site-packages/Sphinx-0.6.1-py2.6.egg/sphinx/environment.py", line 921, in get_toc_for toc = self.tocs[docname].deepcopy() KeyError: 'reference/generalized_ufuncs' The full traceback has been saved in /tmp/sphinx-err-wlA8vA.log, if you want to report the issue to the author. Please also report this if it was a user error, so that a better error message can be provided next time. Send reports to sphinx-dev(a)googlegroups.com. Thanks! make: *** [html] Fehler 1 Nils

2 1

np.memmap and memory usage
by Emmanuelle Gouillart 01 Jul '09

01 Jul '09

Hello, I'm using numpy.memmap to open big 3-D arrays of Xray tomography data. After I have created a new array using memmap, I modify the contrast of every Z-slice (along the first dimension) inside a for loop, for a better visualization of the data. Although I call memmap.flush after each modification of a Z-slice, the memory used by Ipython keeps increasing at every new iteration. At the end of the loop, the memory used by Ipython is of the order of magnitude of the size of the data file (1.8Go !). I would have expected that the maximum amount of memory used would corresponde to only one Z-slice of the 3D array. See the code snapshots below for more details. Is this an expected behaviour? How can I reduce the amount of memory used by Ipython and still process my data? import numpy as np imin, imax = -2, 2 data = np.memmap(filename = 'data.vol', mode='r+', dtype=float32, shape=(512,981, 981)) for sl in data: sl[sl<imin] = imin sl[sl>imax] = imax data.flush() # tried with and without this line Thanks in advance, Emmanuelle

4 6

permutation symbol
by Nils Wagner 01 Jul '09

01 Jul '09

Hi all, How can I build the following product with numpy q_i = \varepsilon_{ijk} q_{kj} where \varepsilon_{ijk} denotes the permutation symbol. Nils http://mathworld.wolfram.com/PermutationSymbol.html

2 9

Tutorial topics for SciPy'09 Conference
by Fernando Perez 01 Jul '09

01 Jul '09

Hi all, The time for the Scipy'09 conference is rapidly approaching, and we would like to both announce the plan for tutorials and solicit feedback from everyone on topics of interest. Broadly speaking, the plan is something along the lines of what we had last year: one continuous 2-day tutorial aimed at introductory users, starting from the very basics, and in parallel a set of 'advanced' tutorials, consisting of a series of 2-hour sessions on specific topics. We will request that the presenters for the advanced tutorials keep the 'tutorial' word very much in mind, so that the sessions really contain hands-on learning work and not simply a 2-hour long slide presentation. We will thus require that all the tutorials will be based on tools that the attendees can install at least 2 weeks in advance on all platforms (no "I released it last night" software). With that in mind, we'd like feedback from all of you on possible topics for the advanced tutorials. We have space for 8 slots total, and here are in no particular order some possible topics. At this point there are no guarantees yet that we can get presentations for these, but we'd like to establish a first list of preferred topics to try and secure the presentations as soon as possible. This is simply a list of candiate topics that various people have informally suggested so far: - Mayavi/TVTK - Advanced topics in matplotlib - Statistics with Scipy - The TimeSeries scikit - Designing scientific interfaces with Traits - Advanced numpy - Sparse Linear Algebra with Scipy - Structured and record arrays in numpy - Cython - Sage - general tutorial - Sage - specific topics, suggestions welcome - Using GPUs with PyCUDA - Testing strategies for scientific codes - Parallel processing and mpi4py - Graph theory with Networkx - Design patterns for efficient iterator-based scientific codes. - Symbolic computing with sympy We'd like to hear from any ideas on other possible topics of interest, and we'll then run a doodle poll to gather quantitative feedback with the final list of candidates. Many thanks, f

1 2

NumPy-Discussion July 2009