Mailman 3 September 2011 - NumPy-Discussion

3D array -> 2D array issue...
by Fred 06 Sep '11

06 Sep '11

Hi all, Let's say C1 is a 3D array, and q0 and k are 2D array. dim C1 = nx*ny*nz dim q0 = nx*ny = dim k I have to do the following: q0[0, 0] = C1[0, 0, k[0, 0]] q0[1, 1] = C1[1, 1, k[1, 1]] ... q0[i, j] = C1[i, j, k[i, j]] ... I tried q0 = C1[:, :, k] but this obviously does not work. How could I do this ala NumPy? TIA Cheers, -- Fred

5 11

Smoothing binary volumes
by Thiago Franco Moraes 06 Sep '11

06 Sep '11

Hi all, I've been implementing the algorithm from this paper "Reducing Aliasing Artifacts in Iso-Surfaces of Binary Volumes" from Ross T. Whitaker. Because I develop a opensource software which works with segmentation of CT and MRI medical images, and the results of segmentation is a binary volume image. The problem is that the surface generate from binary images are rough and have those staircase artifacts. Bellow some examples after and before using the algorithm: Binary ball - http://i.imgur.com/DlIwP.png Smoothed ball - http://i.imgur.com/G04zN.png Binary CT - http://i.imgur.com/Ah1LB.png Smoothed CT - http://i.imgur.com/ps1Nz.png The algorithm works in the volumetric image. After the algorithm I use the Marching Cubes (from VTK) to generate the surface. I could use a gaussian filter, but some fine features could be lost. The source-code is here [1]. The first attempt I use python, but it used a lot of memory. Now I'm using Cython, it's not completed. Maybe I will try to code directly in C. I'm sending this email to this mail-list because someone could have some interest in this, and maybe it could be inserted in some python library, like mahotas. Thanks! [1] - https://github.com/tfmoraes/binvol

1 0

Efficient way to load a 1Gb file?
by Russell E. Owen 06 Sep '11

06 Sep '11

A coworker is trying to load a 1Gb text data file into a numpy array using numpy.loadtxt, but he says it is using up all of his machine's 6Gb of RAM. Is there a more efficient way to read such text data files? -- Russell

7 13

Eigenvalues did not converge
by Rick Muller 04 Sep '11

04 Sep '11

I'm bumping into the old "Eigenvalues did not converge" error using numpy.linalg.eigh() on several different linux builds of numpy (1.4.1). The matrix is 166x166. I can compute the eigenvalues on a Macintosh build of numpy, and I can confirm that there aren't degenerate eigenvalues, and that the matrix appears to be negative definite. I've seen this before (though not for several years), and what I normally do is to build lapack with -O0. This trick did not work in the current instance. Does anyone have any tricks to getting eigh to work? Other weird things that I've noticed about this case: I can compute the eigenvalues using eigvals and eigvalsh, and can compute the eigenvals/vecs using eig(). The matrix is real symmetric, and I've tested that it's symmetric enough by forcibly symmetrizing it. Thanks in advance for any help you can offer. -- Rick Muller rpmuller(a)gmail.com 505-750-7557

3 5

converting standard array to record array
by Bryce Ready 03 Sep '11

03 Sep '11

Hello all, So i'm using numpy 1.6.0, and trying to convert a (4,4) numpy array of dtype 'f8' into a record array of this dtype: dt = np.dtype([('mat','(4,4)f8')]) > Here is the code snippet: In [21]: a = np.random.randn(4,4) > > In [22]: a.view(dt) > and the resulting error: ValueError: new type not compatible with array. > Can anyone shed some light for me on why this conversion is not possible? It is certainly technically possible, since the memory layout of the two arrays should be the same. Can anyone recommend a better way to do this conversion? Thanks in advance! -Bryce Ready

2 2

Hiring a junior developer on the scikit-learn
by Gael Varoquaux 02 Sep '11

02 Sep '11

Hi list, Once again, we are looking for a junior developer to work on the scikit-learn. Below is the official job posting. Unofficially I would like to stress that this is a unique opportunity to be payed for two years to work on learning and improving the scientific Python toolstack. Gael ________________________________________________________________________________ **Job Description** INRIA is looking to hire a young graduate on a 2-year position to help with the community-driven development of the open source machine learning in Python library, scikit-learn. The scikit-learn is one of the major machine-learning libraries in Python. It aims to be state-of-the-art on mid-size to large datasets by harnessing the power of the scientific Python toolstack. Speaking French is not a requirement, as it is an international team. **Requirements** * Programming skills in Python and C/C++ * Understanding of quality assurance in software development: test-driven programming, version control, technical documentation. * Some knowledge of Linux/Unix * Software design skills * Knowledge of open-source development and community-driven environments * Good technical English level * An experience in statistical learning or a mathematical-oriented mindset is a plus * We can only hire a young-graduate that has received a masters or equivalent degree at most a year ago. **About the company** INRIA is the French computer science research institute. It recognized word-wide as one of the leading research institutions and has a strong expertise in machine learning. You will be working in the `Parietal team <https://parietal.saclay.inria.fr/>`_ that makes a heavy use of Python for brain imaging analysis. Parietal is a small research team (around 10 people) with an excellent technical knowledge of scientific and numerical computing in Python as well as a fine understanding of algorithmic issues in machine learning and statistics. Parietal is committed to investing in scikit-learn. Working at Parietal is a unique opportunity to improve your skills in machine learning and numerical computing in Python. In addition, working full time on the scikit-learn, a very active open-source project, will give you premium experience of open source community management and collaborative project development. **Contact Info:** * **Technical Contact**: Bertand Thirion * **E-mail contact**: bertrand.thirion(a)inria.fr * **HR Contact**: Marie Domingues * **E-mail Contact**: marie.domingues(a)inria.fr * **No telecommuting**

1 0

Pull Request Review: R-like sample function
by Christopher Jordan-Squire 02 Sep '11

02 Sep '11

Hi--I've just submitted a numpy 2.0 pull request for a function sample in np.random. It's essentially an implementation of R's sample function. It allows possibly non-uniform, possibly without-replacement sampling from a given 1-D array-like. This is very useful for quickly and cleanly creating samples from, for example, a list of strings or a list of non-contiguous, non-evenly spaced integers. Both occur in data analysis with categorical data. It is, essentially, a convenience function that wraps a number of existing ways to take a random sample. I think it belongs in numpy.random rather than scipy.stats because it's just a random sampler, rather than a probability distribution. It isn't possible to define a scipy.stats discrete random variable on strings--it would have to instead be done on the indices of the list containing the possible samples. And (as far as I can tell) the scipy.stats distributions can't be used for sampling without replacement. https://github.com/numpy/numpy/pull/151 -Chris JS

4 12

PyBUF_SIMPLE/PyBUF_FORMAT: casts to unsigned bytes
by Stefan Krah 02 Sep '11

02 Sep '11

Hello, PEP-3118 presumably intended that a PyBUF_SIMPLE request should cast the original buffer's data type to 'B' (unsigned bytes). Here is a one-dimensional example that currently occurs in Lib/test/test_multiprocessing: >>> import array, io >>> a = array.array('i', [1,2,3,4,5]) >>> m = memoryview(a) >>> m.format 'i' >>> buf = io.BytesIO(bytearray(5*8)) >>> buf.readinto(m) buf.readinto() calls PyObject_AsWriteBuffer(), which requests a simple buffer from the memoryview, thus casting the 'i' data type to the implied type 'B'. The consumer can see that a cast has occurred because the new buffer's format field is NULL. This seems fine for the one-dimensional case. Numpy currently also allows such casts for multidimensional contiguous and non-contiguous arrays. See below for the examples; I don't want to distract from the main point of the post, which is this: I'm seeking a clear specification for the Python documentation that determines under what circumstances casts to 'B' should succeed. I'll formulate the points as statements for clarity, but in fact they are also questions: 1) An exporter of a C-contiguous array with ndim <= 1 MUST honor a PyBUF_SIMPLE request, setting format, shape and strides to NULL and itemsize to 1. As a corner case, an array with ndim = 0, format = "L" (or other) would also morph into a buffer of unsigned bytes. test_ctypes currently makes use of this. 2) An exporter of a C-contiguous buffer with ndim > 1 MUST honor a PyBUF_SIMPLE request, setting format, shape, and strides to NULL and itemsize to 1. 3) An exporter of a buffer that is not C-contiguous MUST raise BufferError in response to a PyBUF_SIMPLE request. Why am I looking for such rigid rules? The problem with memoryview is that it has to act as a re-exporter itself. For several reasons (performance of chained memoryviews, garbage collection, early release, etc.) it has been decided that the new memoryview object has a managed buffer that takes a snapshot of the original exporter's buffer (See: http://bugs.python.org/issue10181). Now, since getbuffer requests to the memoryview object cannot be redirected to the original object, strict rules are needed for memory_getbuf(). Could you agree with these rules? Point 2) isn't clear from the PEP itself. I assumed it because Numpy currently allows it, and it appears harmless. Stefan Krah Examples: ========= Cast a multidimensional contiguous array: ----------------------------------------- I think itemsize in the result should be 1. [_testbuffer.ndarray is from http://hg.python.org/features/pep-3118#memoryview] >>> from _testbuffer import * >>> from numpy import * >>> from _testbuffer import ndarray as pyarray >>> >>> exporter = ndarray(shape=[3,4], dtype="L") # Issue a PyBUF_SIMPLE request to 'exporter' and act as a re-exporter: >>> x = pyarray(exporter, getbuf=PyBUF_SIMPLE) >>> x.len 96 >>> x.shape () >>> x.strides () >>> x.format '' >>> x.itemsize # I think this should be 1, not 8. 8 Cast a multidimensional non-contiguous array: --------------------------------------------- This is clearly not right, since y.buf points to a location that the consumer cannot handle without shape and strides. >>> nd = ndarray(buffer=bytearray(96), shape=[3,4], dtype="L") [182658 refs] >>> exporter = nd[::-1, ::-2] [182661 refs] >>> exporter array([[0, 0], [0, 0], [0, 0]], dtype=uint64) [182659 refs] >>> y = pyarray(exporter, getbuf=PyBUF_SIMPLE) [182665 refs] >>> y.len 48 [182666 refs] >>> y.strides () [182666 refs] >>> y.shape () [182666 refs] >>> y.format '' [182666 refs] >>> y.itemsize 8 [182666 refs]

2 3

question about subtraction and shape
by Jonas Wallin 01 Sep '11

01 Sep '11

Hello, I implemented the following line of code: Gami[~index0].shape ----> (100,) sigma.shape ----> (1,1) Gami[~index0] = Gam[~index0] - sigma**2 I get the error message: *** ValueError: array is not broadcastable to correct shape apparently *temp* = Gam[~index0] - sigma**2 *temp*.shape --> (1,100) Which seems strange to me is there any reason why *temp* gets this shape ? mvh Jonas Wallin

2 1

A question about dtype syntax
by Marquette Jean-Baptiste 01 Sep '11

01 Sep '11

Hi all, I have this piece of code: Stats = [CatBase, round(stats.mean(Data.Ra), 5), round(stats.mean(Data.Dec), 5), len(Sep), round(stats.mean(Sep),4), round(stats.stdev(Sep),4)] print Stats if First: StatsAll = np.array(np.asarray(Stats), dtype=('a11, f8, f8, i4, f8, f8')) First = False else: StatsAll = np.vstack((StatsAll, np.asarray(Stats))) print len(StatsAll) This yields the error: ['bs3000k.cat', 280.60341, -7.09118, 9480, 0.2057, 0.14] Traceback (most recent call last): File "/Users/marquett/workspace/Distort/src/StatsSep.py", line 40, in <module> StatsAll = np.array(np.asarray(Stats), dtype=('a11, f8, f8, i4, f8, f8')) ValueError: could not convert string to float: bs3000k.cat What's wrong ? Thanks for your help Cheers JB

4 9