NumPy-Discussion January 2009

numpy-discussion@python.org

116 participants
133 discussions

formatting issues, locale and co
by David Cournapeau 12 Jan '09

12 Jan '09

Hi, While looking at the last failures of numpy trunk on windows for python 2.5 and 2.6, I got into floating point number formatting issues; I got deeper and deeper, and now I am lost. We have several problems: - we are not consistent between platforms, nor are we consistent with python - str(np.float32(a)) is locale dependent, but python str method is not (locale.str is) - formatting of long double does not work on windows because of the broken long double support in mingw. 1 consistency problem: ---------------------- python -c "a = 1e20; print a" -> 1e+020 python26 -c "a = 1e20; print a" -> 1e+20 In numpy, we use PyOS_snprintf for formatting, but python itself uses PyOS_ascii_formatd - which has different behavior on different versions of python. The above behavior can be simply reproduced in C: #include <Python.h> int main() { double x = 1e20; char c[200]; PyOS_ascii_format(c, sizeof(c), "%.12g", x); printf("%s\n", c); printf("%g\n", x); return 0; } On 2.5, this will print: 1e+020 1e+020 But on 2.6, this will print: 1e+20 1e+020 2 locale dependency: -------------------- Another issue is that our own formatting is local dependent, whereas python isn't: import numpy as np import locale locale.setlocale(locale.LC_NUMERIC, 'fr_FR') a = 1.2 print "str(a)", str(a) print "locale.str(a)", locale.str(a) print "str(np.float32(a))", str(np.float32(a)) print "locale.str(np.float32(a))", locale.str(np.float32(a)) Returns: str(a) 1.2 locale.str(a) 1,2 str(np.float32(a)) 1,2 locale.str(np.float32(a)) 1,20000004768 I thought about copying the way python does the formatting in the trunk (where discrepancies between platforms have been fixed), but this is not so easy, because it uses a lot of code from different places - and the code needs to be adapted to float and long double. The other solution would be to do our own formatting, but this does not sound easy: formatting in C is hard. I am not sure about what we should do, if anyone else has any idea ? cheers, David

6 19

Is there anyway to seed random numbers without global state?
by Tom Denniston 12 Jan '09

12 Jan '09

I know how to seed and generate random numbers using: numpy.random.seed and numpy.random.rand The problem is the seeding of the random numbers is global which I would think would make it non-thread safe as well as having all the other annoyances of global state like having so set the seed and set it back when done. I would think the ideal would be to be able to build random number objects where each has a seed so that each object's seed is independent. Does such a thing exist in numpy or scipy. Does this even make sense to people? Is there something about random number generation that makes it inherently a global thing? --Tom

3 2

Re: [Numpy-discussion] coding style: citations
by jh＠physics.ucf.edu 12 Jan '09

12 Jan '09

Well, best to have the full functionality. These things are intended to be both book components and stand-alone pages. Citation formats are a religious war anyway. --jh-- Date: Mon, 12 Jan 2009 12:41:17 -0500 From: Alan G Isaac <aisaac(a)american.edu> Subject: Re: [Numpy-discussion] coding style: citations To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <496B80BD.8050102(a)american.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 1/12/2009 9:08 AM jh(a)physics.ucf.edu apparently wrote: > For citation keys, what's wrong with good old author-year format? > Most scientific journals use it (Abt 1985). > Abt, H. 1985. Harold Abt used to publish surveys of things like > citations when he was ApJ editor in the 1980s but I'm making this > one up just to demonstrate the format except he'd never allow an > article title in a citation because he wanted to save ink which > everyone hated so other journals keep the titles and we should too. > ApJ 123:1-23. There are two problems. 1. The only real problem is that it is not a reST citation, so links to the citation will not be generated, and you cannot control the citation formatting separately. You can still of course use normal internal targets: See `(Abt 1985)`_. Some more text, and later ... .. _(Abt 1985): Abt, H. 1985. Harold Abt used to publish surveys of things like citations when he was ApJ editor in the 1980s but I'm making this one up just to demonstrate the format except he'd never allow an article title in a citation because he wanted to save ink which everyone hated so other journals keep the titles and we should too. ApJ 123:1-23. 2. The failure to use citation keys implies that there will be neither a citation database nor consistent formatting. (Say, if the docs become a book.) I do not suppose this is *currently* a real problem. Cheers, Alan Isaac

1 0

Re: [Numpy-discussion] coding style: citations
by jh＠physics.ucf.edu 12 Jan '09

12 Jan '09

> ... citations ... BibTeX Please ensure that, whatever you come up with, it is *very intuitive* for the writer to enter the citation data, or we will quickly lose citations, and possibly writers. For example, I would not have them type in BibTeX entries, but rather provide boxes where they can enter the title, author, etc. and have the system generate the BibTeX. Not everyone uses BibTeX, and the rules about braces, what fields are required, etc., are not totally trivial. Perhaps there's a web site out there that does this already, if BibTex is to be our internal format (which I'm not at all opposed to; I use it myself). If a large majority of our current writers thinks BibTeX is ok, then I'll change my mind. We should poll them on the acceptability of any proposed system before implementing it. For citation keys, what's wrong with good old author-year format? Most scientific journals use it (Abt 1985). --jh-- Abt, H. 1985. Harold Abt used to publish surveys of things like citations when he was ApJ editor in the 1980s but I'm making this one up just to demonstrate the format except he'd never allow an article title in a citation because he wanted to save ink which everyone hated so other journals keep the titles and we should too. ApJ 123:1-23.

2 1

Fast function application on list of 2D points?
by Eric LEBIGOT 12 Jan '09

12 Jan '09

Hello, What is the fastest way of applying a function on a list of 2D points? More specifically, I have a list of 2D points, and some do not meet some criteria and must be rejected. Even more specifically, the filter only lets through points whose x coordinate satisfies some condition, _and_ whose y coordinates satisfies another condition (maybe is there room for optimization, here?). Currently, I use points = numpy.apply_along_axis(filter, axis = 1, arr = points) but this creates a bottleneck in my program (array arr may contains 1 million points, for instance). Is there anything that could be faster? Any suggestion would be much appreciated! EOL

2 2

memmap from fd?
by Neal Becker 12 Jan '09

12 Jan '09

I'm working on interfacing to a custom FPGA board. The kernel driver exposes the FPGA memory via mmap. It might be nice to use numpy memmap to read/write data. One issue is that I think I will need to create the memmap array from a fd, not a file name. The reason is I wrote the driver to only allow 1 exclusive open, and I already have it open for other reasons. Any chance to create a memmap array from a fd?

2 9

TypeError in dtype.__eq__()
by Bryan Cole 11 Jan '09

11 Jan '09

Dtype objects throw an exception if compared for equality against other objects. e.g. >>> import numpy >>> numpy.dtype('uint32')==1 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: data type not understood >>> After some googling, I think python wisdom (given in the Python docs) says that equality testing should never throw an exception, but return the NotImplemented singleton where the comparison does not make "sense". Throwing an exception breaks membership testing for lists/tuples (using the 'in' operator calls implicitly calls __eq__()). My real-world example is serialisation of numpy arrays using PyYAML. This module should serialise any picklable object including ndarrays. However, it performs a test on each object: if data in [None, ()]: blah... This fails on the dtype object of the array. What's the consensus on this? Is the current dtype behaviour broken? cheers, BC

2 4

memmap problem
by Neal Becker 10 Jan '09

10 Jan '09

I modified memmap.py to avoid the issues with needed to read. It is working, but I am seeing these: m Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([255, 255, 255], dtype=uint8)> ignored Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0], dtype=uint8)> ignored Out[22]: eos_memmap([ 0, 0, 0, ..., 255, 255, 255], dtype=uint8) print m[0:4] Exception exceptions.EnvironmentError: (22, 'Invalid argument') in <bound method eos_memmap.__del__ of eos_memmap([0, 0, 0, 0], dtype=uint8)> ignored [0 0 0 0] What's that about?

2 2

help with typemapping a C function to use numpy arrays
by Rich E 09 Jan '09

09 Jan '09

Hi list, My question has to do with the Numpy/SWIG typemapping system. I recently got the typemaps in numpy.i to work on most of my C functions that are wrapped using SWIG, if they have arguments of the form (int sizeArray, float *pArray). Now I am trying to figure out how to wrap function that aren't of the form, such as the following function: /*! \brief compute magnitude spectrum of a DFT * * \param sizeMag size of output Magnitude (half of input real FFT) * \param pFReal pointer to input FFT real array (real/imag floats) * \param pFMAg pointer to float array of magnitude spectrum */ void sms_spectrumMag( int sizeMag, float *pInRect, float *pOutMag) { int i, it2; float fReal, fImag; for (i=0; i<sizeMag; i++) { it2 = i << 1; fReal = pInRect[it2]; fImag = pInRect[it2+1]; pOutMag[i] = sqrtf(fReal * fReal + fImag * fImag); } } There are two arrays, one is half the size of the other. But, SWIG doesn't know this, according to the type map it will think *pInRect is of size sizeMag and will not know anything about *pOutMag. Ideally in python, I would like to call the function as sms_spectrumMag(nArray1, nArray2), where nArray1 is twice the size of nArray2, and nArray2 is of size sizeMag. I think in order to do this (although if someone has a better suggestion, I am open to it), I will have to modify the typemap in order to tell SWIG how to call the C function properly. I do not want to have to edit the wrapped C file every time it is regenerated from the interface file. Here is a start I made with the existing typemap code in numpy.i (not working): /* Typemap suite for (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) */ %typecheck(SWIG_TYPECHECK_DOUBLE_ARRAY, fragment="NumPy_Macros") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) { $1 = is_array($input) && PyArray_EquivTypenums(array_type($input), DATA_TYPECODE); } %typemap(in, fragment="NumPy_Fragments") (DIM_TYPE DIM1, DATA_TYPE* INPLACE_ARRAY1) (PyArrayObject* array=NULL, int i=0) { array = obj_to_array_no_conversion($input, DATA_TYPECODE); if (!array || !require_dimensions(array,1) || !require_contiguous(array) || !require_native(array)) SWIG_fail; $1 = 1; for (i=0; i < array_numdims(array); ++i) $1 *= array_size(array,i); $2 = (DATA_TYPE*) array_data(array); } and try to alter it to allow for a conversion of type: (DIM_TYPE DIM1, DATA_TYPE* ARRAY1, DATA_TYPE* ARRAY2) where ARRAY1 is size DIM1 * 2 and ARRAY2 is size DIM1. Then I can %apply this to my function that I mentioned in the last post. So here are my first two questions: 1) where is DIM1 used to declare the array size? I don't see where it is used at all, and I need to somewhere multiply it by 2 to declare the size of ARRAY1 2) I am not understanding where $input comes from, so I do not understand how to distinguish between ARRAY1 and ARRAY2. In the attempt I have already tried, I think I just use the pointer to ARRAY1 twice. If anyone has suggestions on how to solve this problem, thanks! regards, Rich

3 11

2-D function and meshgrid
by Paulo J. S. Silva 09 Jan '09

09 Jan '09

Hello, I have a function that receives a array of shape (2,) and returns a number (a function from R^2 -> R). It basically looks like this: def weirdDistance2(x): return dot(dot(weirdMatrix, x), x) (weirdMatrix is a "global" (2,2) array) I want to see its level sets in the box [0, 1] x [0, 1], hence I have to create a meshgrid and then compute it at each point of the mesh: x = linspace(0, 1, 200) y = x.copy() X, Y = meshgrid(x, y) My problem is how to actually compute the function at each point of the mesh. I have come out with two solutions. One very short and clear, but slow, and another longer and more convoluted (it has a loop, I hate loops in numpy code), but faster. Does anyone know a "no explicit-loops" and fast solution? Solution1: def myDistance(a, b): return weirdDistance(np.array((a, b))) vecDistance = np.vectorize(myDistance) return vecDistance(X, Y) Solution 2: nPoints = X.size result = np.zeros(nPoints) points = np.array( [X.ravel(), Y.ravel()] ).T for i in xrange(nPoints): result[i] = weirdDistance(points[i]) result = result.reshape(X.shape) Of course, the first one is slow because the myDistance function creates an array at each call. The second one, even with a loop, avoids the array creations. Best, Paulo

2 2

← Newer
1
...
7
8
9
10
11
12
13
14
Older →

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion January 2009