Mailman 3 October 2004 - NumPy-Discussion

location of tutorial
by Alan G Isaac 10 Oct '04

10 Oct '04

p.29 of the Numeric manual refers to http://www.python.org/doc/tut/functional.html which no longer exists. I suggest substituting http://docs.python.org/tut/tut.html fwiw, Alan Isaac

1 0

numarray.random_array number generation in C code
by Faheem Mitha 07 Oct '04

07 Oct '04

Dear People, I want to write some C++ code to link with Python, using the Boost.Python interface. I need to generate random numbers in the C++ code, and I was wondering as to the best way of doing this. Note that it is important that the random number generation interoperate seamlessly with Python, in the sense that the behavior of the calls to the RNG is the same whether calls are made at the C level or the Python level. I hope the reasons why this is important are obvious. I was thinking that the method should go like this. 1) When C/C++ code called, reads seed from python random state. 2) Does its stuff. 3) Writes seed back to python level when it exits. After doing a little investigation of the numarray.random_array python library and associated extension modules, it seems possible that the answer is simpler than I had supposed. However, I would appreciate it if someone would tell me if my understanding is incorrect in some places. Summary: It seems that I can just call all the C entry point routines defined in ranlib.h, without worrying about getting or setting seeds. Rationale: The structure of this random number facility has three parts, all files in Packages/RandomArray2/Src. 1) low-level C routines: Packages/RandomArray2/Src/com.c and Packages/RandomArray2/Src/ranlib.c. com.c: basic RNG stuff; getting and setting seeds etc. ranlib.c: Random number generator algorithms for different distributions etc. 2) Python to C interface: Packages/RandomArray2/Src/ranlibmodule.c. This interfaces the stuff in com.c and ranlib.c. 3) Python wrapper: Packages/RandomArray2/Lib/RandomArray2.py. This wraps the C interface. In most cases it does not do much else besides some basic argument error checking. >From my perspective, the important thing is that the random number seed is only defined at C level as a static object, all the RNG stuff happens at C level, and the Python code just calls the C code as necessary. (I'm sketchy about the details of what is defined as the seed etc.) This is in contrast with the R RNG facility (the only other RNG facility I am familiar with), which uses macros SetRNGstate() and GetRNGstate() to read and write the seed, which is defined at R level. Therefore, the upshot is that the C routines in ranlib.h read and write the same seed as the python level functions do, so no special action is necessary with regard to the seed. Is this correct? In any case, it would be nice if something like the above was documented, so lost souls like myself don't have to go trawling through the source code to figure out what is going on. Of course it is nice that the source code is available, otherwise even that would be impossible. R documents this stuff in the "Writing R Extensions" manual, online at http://cran.r-project.org/doc/manuals/R-exts.pdf. Perhaps the Numarray manual could have a small section about this too. Regards, Faheem.

3 3

Re: [Numpy-discussion] numarray.random_array number generation in C code
by Bruce Southey 06 Oct '04

06 Oct '04

Hi, My understanding is that you can use the Ranlib, R math, and GNU Scientific libraries in the manner you suggest or directly include the random number generator in your code. Usually you define the seed that should provide the same psuedo-random number stream every time these are used. If you don't use a seed then it is usually impossible to get the same stream of psuedo-random numbers. So I do not understand what you need to keep the same random number state. Not to mention that the common generators do repeat, some sooner than others. In your response to Perry, you indicate that you do not need an array of random numbers but rather the stream of random numbers. This is very different and I think you need to refine your algorithm to identify what parts need to be C/C++ and what need to be in Python/numarray. Since you currently have Python code, I would profile it to see what parts actually need extending - some times Python is rather surprising on how quick some things can be done (like using dictionaries). Providing those parts may be more fruitful to you than my vague responses. Regards Bruce ---- Original message ---- >Date: Tue, 5 Oct 2004 18:43:48 -0400 (EDT) >From: Faheem Mitha <faheem(a)email.unc.edu> >Subject: Re: [Numpy-discussion] numarray.random_array number generation in C code >To: Bruce Southey <southey(a)uiuc.edu> >Cc: numpy-discussion <numpy-discussion(a)lists.sourceforge.net> > > > >On Tue, 5 Oct 2004, Bruce Southey wrote: > >> Hi, >> It is rather hard to suggest anything without more detail on what you want to >> actually do. > >I could give you more details if you were interested. > >> As you describe it, why do you need the 'seed' returned? It would only >> make sense if you were going in and out of Python multiple times - a >> somewhat undesirable situation due to the overhead costs. > >Not really. One might (and I frequently do) want to run the same function >(which in this case might be all in C++ code), interactively with >different parameters. The kind of thing that I'm doing is akin to >exploratory data analysis, and the specific code in question is a >stochastic search algorithm. Doing all this in C++ would not be very >interactive. Also, one often wants to postprocess data output using Python >scripts. This involves multiple calls to C++ code, and would be impossible >to do using C++, since one has to call other Python libraries. > > > I see at least three options: > >> 1) Do everything in Python/numarray. > >That's my current situation. > >> 2) Do parts in Python and the other in C/C++. >> For example, pass a matrix of random numbers to your code from Python. The >> 'seed' never needs to leave Python. > >This doesn't work very well unless you know in advance how many random >numbers are needed (not the case, for example, for stochastic search >algorithms), and in any case is a rather clumsy way to do things. No >offense intended. > >> 3) Do it all in C/C++ - pass the 'seed' into your code that includes the >> random number generator(s) - there is C/C++ code around for this. Do you stuff >> and then return the 'seed' back with whatever else is required. > >Yes, but part of the point of mixed programming is that you have an >interpreted front end which can easily hook into other routines. Also, in >this case, you would not be passing the seed in, since there is nothing to >pass it in from. One would simply call system time or something similar to >obtain the seed. > >> You can email me privately if you want. > >I'll keep sending this to the list unless someone objects, since I think >this is of some general interest. > >Really, my main question was to whether my understanding of how to use the >Numarray random number facilities in C was correct or not. > > Faheem.

1 0

Numeric to numarray experiences
by Raik Grünberg 05 Oct '04

05 Oct '04

Hi there, I've just translated a package for molecular modelling, which makes extensive use of Numeric, from Numeric to numarray. The outcome is somewhat negative - for now we are basically going to postpone the transition - the reasons might be interesting for the list and the numarray developpers out there (who are doing a brave job!). Speed: A typical task in our package is the least-square fitting of a large array of coordinate frames ( N1 x N2 x 3) onto a set of reference or average coordinates (using a sub-set of coordinates for the matching). The example I looked at (500 x 876 x 3 items) took 1.3 s with Numeric and 4.7 s with numarray. The main culprits for the slow-down were: * compress() - factor 10 * average() - factor 7 (average() is missing from Numeric and I hence had to write a little function myself) * LinearAlgebra.singular_value_decomposition() - factor 10 but a lot of extra time is also spent in uufunc.py and various numarraycore.py routines. Memory efficiency: I hoped numarray would solve some of the Out-of-memory problems that I get with Numeric but it turns out that it is rather less memory efficient for my kind of applications. Slicing an array that takes up 800MB on disc just about runs through with Numeric (and heavy swapping) but gives an Out-of-memory with numarray. Suggestions: OK, it's easy to make clever comments without contributing any real work... - compress(), take(), etc, really need some optimization - a C-coded average() routine would be helpful - faster LinearAlgebra routines are necessary Our sysadmin noted that unlike Numeric, numarray is not using any external math libraries (like LAPACK) that have been speed-optimized for decades and are available in CPU-optimized variants (e.g. ATLAS). It's probably difficult to match this efficiency with any new code ... Greetings Raik PS: I didn't find any useful HowTo for the translation from Numeric to numarray. The practical issues were the different nonzero() return value, the more restrictive boolean comparison, that take doesn't support 'O' arrays any longer, and the missing average(). -- ----------------------------------------------------- Raik Grünberg | Bioinformatique Structurale | Institut Pasteur | Paris, France -----------------------------------------------------

5 4

Re: [Numpy-discussion] Numeric to numarray experiences
by Stephen Walton 05 Oct '04

05 Oct '04

On Tue, 2004-10-05 at 16:00, Flavio Coelho wrote: > I wrote > > env USE_LAPACK=1 python setup.py build > > > > after editing addons.py appropriately. You should see a major speed > > improvement. > > > > > If that is the case, why is it not the default?, at least when LAPACK > is installed? Well, I won't pretend to speak for the developers on this one. But I strongly suspect it is just too hard to find all possible LAPACK distributions; the default numarray setup should be self contained even if somewhat slower. The current version of Numeric also defaults to its own built-in BLAS and requires editing setup.py to use a different one. -- Stephen Walton <stephen.walton(a)csun.edu> Dept. of Physics & Astronomy, Cal State Northridge

2 1

Re: [Numpy-discussion] numarray.random_array number generation in C code
by Bruce Southey 05 Oct '04

05 Oct '04

Hi, It is rather hard to suggest anything without more detail on what you want to actually do. As you describe it, why do you need the 'seed' returned? It would only make sense if you were going in and out of Python multiple times - a somewhat undesirable situation due to the overhead costs. I see at least three options: 1) Do everything in Python/numarray. 2) Do parts in Python and the other in C/C++. For example, pass a matrix of random numbers to your code from Python. The 'seed' never needs to leave Python. 3) Do it all in C/C++ - pass the 'seed' into your code that includes the random number generator(s) - there is C/C++ code around for this. Do you stuff and then return the 'seed' back with whatever else is required. You can email me privately if you want. Bruce ---- Original message ---- >Date: Sat, 2 Oct 2004 01:23:21 -0400 (EDT) >From: Faheem Mitha <faheem(a)email.unc.edu> >Subject: [Numpy-discussion] numarray.random_array number generation in C code >To: numpy-discussion <numpy-discussion(a)lists.sourceforge.net> > > >Dear People, > >I want to write some C++ code to link with Python, using the >Boost.Python interface. I need to generate random numbers in the C++ >code, and I was wondering as to the best way of doing this. > >Note that it is important that the random number generation interoperate >seamlessly with Python, in the sense that the behavior of the calls to >the RNG is the same whether calls are made at the C level or the Python >level. I hope the reasons why this is important are obvious. > >I was thinking that the method should go like this. > >1) When C/C++ code called, reads seed from python random state. > >2) Does its stuff. > >3) Writes seed back to python level when it exits. > >After doing a little investigation of the numarray.random_array python >library and associated extension modules, it seems possible that the >answer is simpler than I had supposed. However, I would appreciate it if >someone would tell me if my understanding is incorrect in some places. > >Summary: It seems that I can just call all the C entry point routines >defined in ranlib.h, without worrying about getting or setting seeds. > >Rationale: > >The structure of this random number facility has three parts, all files in >Packages/RandomArray2/Src. > >1) low-level C routines: Packages/RandomArray2/Src/com.c and >Packages/RandomArray2/Src/ranlib.c. > >com.c: basic RNG stuff; getting and setting seeds etc. >ranlib.c: Random number generator algorithms for different distributions >etc. > >2) Python to C interface: Packages/RandomArray2/Src/ranlibmodule.c. > >This interfaces the stuff in com.c and ranlib.c. > >3) Python wrapper: Packages/RandomArray2/Lib/RandomArray2.py. > >This wraps the C interface. In most cases it does not do much else besides >some basic argument error checking. > >From my perspective, the important thing is that the random number seed is >only defined at C level as a static object, all the RNG stuff happens at C >level, and the Python code just calls the C code as necessary. (I'm >sketchy about the details of what is defined as the seed etc.) > >This is in contrast with the R RNG facility (the only other RNG facility I >am familiar with), which uses macros SetRNGstate() and GetRNGstate() to >read and write the seed, which is defined at R level. > >Therefore, the upshot is that the C routines in ranlib.h read and write >the same seed as the python level functions do, so no special action is >necessary with regard to the seed. > >Is this correct? > >In any case, it would be nice if something like the above was documented, >so lost souls like myself don't have to go trawling through the source >code to figure out what is going on. Of course it is nice that the source >code is available, otherwise even that would be impossible. > >R documents this stuff in the "Writing R Extensions" manual, online at >http://cran.r-project.org/doc/manuals/R-exts.pdf. Perhaps the Numarray >manual could have a small section about this too. > > Regards, Faheem. > > > >------------------------------------------------------- >This SF.net email is sponsored by: IT Product Guide on ITManagersJournal >Use IT products in your business? Tell us what you think of them. Give us >Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more >http://productguide.itmanagersjournal.com/guidepromo.tmpl >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion(a)lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion

2 1

Small bug in MA with arrays of rank > 1
by Fernando Perez 04 Oct '04

04 Oct '04

Hi all, a while back I noticed a small problem with MA for rank 2 (and larger) arrays. Here's a simple example: In [1]: a=RA.random((3,3)) In [2]: a Out[2]: array([[ 0.002542, 0.70301 , 0.705466], [ 0.467305, 0.381492, 0.655857], [ 0.103372, 0.776988, 0.466528]]) In [3]: import MA In [4]: a Out[4]: [[ 0.002542, 0.70301 , 0.705466,] [ 0.467305, 0.381492, 0.655857,] [ 0.103372, 0.776988, 0.466528,]] The bug is that the commas at the end of each line are coming _before_ the closing bracket, instead of after. This seemingly trivial problem turns out to be pretty serious for me, because I use this string representation to export python arrays into Mathematica files, by simply replacing [] with {} (and playing some other tricks). Unfortunately, this bug means I can't use MA, which is otherwise great because of the way it gracefully handles the case where you accidentally say A when A is some monster array. With MA, instead of your CPU getting killed for 10 minutes, you get a nice summary of A's dimensions and typecode. Anyway, it would be great if one of the gurus had a chance to fix this one. Best, f

1 0

Bug Compiling Numeric on amd64
by Flavio Coelho 04 Oct '04

04 Oct '04

Hi, look at this: >>> from RandomArray import * >>> normal(2,2,10) array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]) This is Numeric 23.1 compiled on my AMD64!!! I ran the same tests on a 32bit P4 and it ran fine. Has anyone else seen this before? luckily: >>> from numarray.random_array import * >>> normal(2,2,10) array([-0.04525638, 4.31467819, -0.17468357, 5.29377031, 0.84202135, 5.29593539, 4.69651532, 1.61354655, 1.10839236, 1.7743317 ]) Both modules were compiled on my gentoo box with: gcc version 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6) any comments? Flavio -- I use Linux daily to UP my productivity -- Microsoft, UP yours!

2 1

[Matplotlib-users] warning: Numeric and amd64
by Flávio Codeço Coelho 01 Oct '04

01 Oct '04

Hi, look at this: >>> from RandomArray import * >>> normal(2,2,10) array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]) This is Numeric 23.1 compiled on my AMD64!!! I ran the same tests on a 32bit P4 and it ran fine. Has anyone else seen this before? For those that didn't understand, the normal function as called above, is supposed to give me ten samples form a normal distribution with mean = 2 and standard deviation = 2 luckily: >>> from numarray.random_array import * >>> normal(2,2,10) array([-0.04525638, 4.31467819, -0.17468357, 5.29377031, 0.84202135, 5.29593539, 4.69651532, 1.61354655, 1.10839236, 1.7743317 ]) If anybody still needed a reason for switching to numarray, there you go! I anybody here subscribes the numeric or numarray mailing lists (i.e. if they even exist) could you please forward this message to them? Flavio ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Matplotlib-users mailing list Matplotlib-users(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users --=-f+ARSKyzBPwKnxDSn4zh--

1 0

[Fwd: [Matplotlib-users] warning: Numeric and amd64]
by Todd Miller 01 Oct '04

01 Oct '04

--

4 6

NumPy-Discussion October 2004