Mailman 3 October 2009 - NumPy-Discussion

Using matplotlib's prctile on masked arrays
by Gökhan Sever Oct. 28, 2009

Oct. 28, 2009

Hello, Consider this sample two columns of data: 999999.9999 999999.9999 999999.9999 999999.9999 999999.9999 999999.9999 999999.9999 1693.9069 999999.9999 1676.1059 999999.9999 1621.5875 651.8040 1542.1373 691.0138 1650.4214 678.5558 1710.7311 621.5777 999999.9999 644.8341 999999.9999 696.2080 999999.9999 Putting into this data into a file say "sample.data" and loading with: a,b = np.loadtxt('sample.data', dtype="float").T I[16]: a O[16]: array([ 1.00000000e+06, 1.00000000e+06, 1.00000000e+06, 1.00000000e+06, 1.00000000e+06, 1.00000000e+06, 6.51804000e+02, 6.91013800e+02, 6.78555800e+02, 6.21577700e+02, 6.44834100e+02, 6.96208000e+02]) I[17]: b O[17]: array([ 999999.9999, 999999.9999, 999999.9999, 1693.9069, 1676.1059, 1621.5875, 1542.1373, 1650.4214, 1710.7311, 999999.9999, 999999.9999, 999999.9999]) ### interestingly, the second column is loaded as it is but a values reformed a little. Why this could be happening? Any idea? Anyways, back to masked arrays: I[24]: am = ma.masked_values(a, value=999999.9999) I[25]: am O[25]: masked_array(data = [-- -- -- -- -- -- 651.804 691.0138 678.5558 621.5777 644.8341 696.208], mask = [ True True True True True True False False False False False False], fill_value = 999999.9999) I[30]: bm = ma.masked_values(b, value=999999.9999) I[31]: am O[31]: masked_array(data = [-- -- -- -- -- -- 651.804 691.0138 678.5558 621.5777 644.8341 696.208], mask = [ True True True True True True False False False False False False], fill_value = 999999.9999) So far so good. A few basic checks: I[33]: am/bm O[33]: masked_array(data = [-- -- -- -- -- -- 0.422662755126 0.418689311712 0.39664667346 -- -- --], mask = [ True True True True True True False False False True True True], fill_value = 999999.9999) I[34]: mean(am/bm) O[34]: 0.41266624676580849 Unfortunately, matplotlib.mlab's prctile cannot handle this division: I[54]: prctile(am/bm, p=[5,25,50,75,95]) O[54]: array([ 3.96646673e-01, 6.21577700e+02, 1.00000000e+06, 1.00000000e+06, 1.00000000e+06]) This also results with wrong looking box-and-whisker plots. Testing further with scipy.stats functions yields expected correct results: I[55]: stats.scoreatpercentile(am/bm, per=5) O[55]: 0.40877012449846228 I[49]: stats.scoreatpercentile(am/bm, per=25) O[49]: masked_array(data = --, mask = True, fill_value = 1e+20) I[56]: stats.scoreatpercentile(am/bm, per=95) O[56]: masked_array(data = --, mask = True, fill_value = 1e+20) Any confirmation? -- Gökhan

3 5

Re: [Numpy-discussion] [Cython] Syntax highlighting for Cython and NumPy
by Sturla Molden Oct. 27, 2009

Oct. 27, 2009

Lisandro Dalcin skrev: > Is there any specific naming convention for these XML files to work > with KATE? Would it be fine to call it 'cython-mode-kate.xml' to push > it to the repo? Will it still work (I mean, with that name) when > placed appropriately in KATE config dirs or whatever? ... Just > concerned that 'cython.xml' is a bit too generic filename... > > You can name it anything you want. The file has an entry like this: <language name="Cython with NumPy" version="1.1" kateversion="2.4" section="Sources" extensions="*.pyx;*.pxi;*.pxd" that tells KDE what this is and where to put it. I was thinking of changing section to "Scientific", because that I where I want to put "Python with NumPy" as well. It makes sence to have NumPy highlighting alongside Matlab and Octave. S.M.

1 0

Syntax highlighting for Cython and NumPy
by Sturla Molden Oct. 27, 2009

Oct. 27, 2009

Here is an XML for Cython syntax highlighting in katepart (e.g. KATE and KDevelop). I made this because KATE is my faviourite text edior (feel free to call me a heretic for not using emacs). Unfortunately, the Python highlighting for KDE contains several bugs. And the Pyrex/Cython version that circulates on the web builds on this and introduces a couple more. I have tried to clean it up. Note that this will also highlight numpy.* or np.*, if * is a type or function you get from "cimport numpy" or "import numpy". This works on Windows as well, if you have installed KDE for Windows. Just copy the XML to: ~/.kde/share/apps/katepart/syntax/ C:\kde\share\apps\katepart\syntax (or whereever you have KDE installed) and "Cython with NumPy" shows up under Sources. Anyway, this is the syntax high-lighter I use to write Cython. Feel free to use it as you wish. P.S. I am also cleaning up Python high-lighting for KDE. Not done yet, but I will post a "Python with NumPy" highlighter later on if this is interesting. P.P.S. This also covers Pyrex, but add in some Cython stuff. Sturla Molden

1 1

Multiplicity of an entry
by Nils Wagner Oct. 27, 2009

Oct. 27, 2009

Hi all, how can I obtain the multiplicity of an entry in a list a = ['abc','def','abc','ghij'] The multiplicity of 'abc' is 2. 'def' is 1. 'ghij' is 1. Nils

7 8

Astype and strings
by Eli Bressert Oct. 27, 2009

Oct. 27, 2009

Hi Everyone, Is Numpy supposed to behave this like this when converting an array of numbers to an array of strings with astype? print(arange(20).astype(np.str)) ['0' '1' '2' '3' '4' '5' '6' '7' '8' '9' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1'] When I do the following it works fine, print(arange(20).astype('|S2')) ['0' '1' '2' '3' '4' '5' '6' '7' '8' '9' '10' '11' '12' '13' '14' '15' '16' '17' '18' '19'] I would have thought that astype would be more intelligent with strings rather than just resorting to the first character for each element. Is this a bug or or is it how astype works? Thanks, Eli

2 1

C-API: How is data filling done in PyArray_SimpleNewFromData ?
by Raspaud Martin Oct. 27, 2009

Oct. 27, 2009

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I’m using numpy v1.2.0, and I have the following codes that provide different results : - --------------------- cal = (PyArrayObject *)PyArray_SimpleNew(2,dims,NPY_FLOAT); for(i=0;i<dims[0];i++) for(j=0;j<dims[1];j++) { *((npy_float *)PyArray_GETPTR2(cal,i,j))=(npy_float)in[i][j]; } - --------------------- and - --------------------- cal = (PyArrayObject *)PyArray_SimpleNewFromData(2,dims,NPY_FLOAT,in); - --------------------- As you probably guessed, "in" is a 2D array of floats of dimensions "dims". My questions are thus: - - Why do the two methods provide different results ? - - How do I get the second to behave like the first ? Thanks, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJK5usJAAoJEBdvyODiyJI4YWwH/1VyfpeVVfI0/VfcVJ4L064z cWbqI4WOpDeYZmFIKLQfuVWwlEkQzbe7AQTxDiy3kR1iL93d6af0+YiLCaURdx9X yv3P7VE+i5clr6jqpESLngkl15LT8aJBzlI/UJL6UnrVBiKYAQ0XSFVdAWChOLE/ VtnEMNxbb5ZHqAwBhmjD/fldG9OwSODpq0yOudVgk1VWTQd1M4AESypY7QGFPod2 Exu83zm3ZVLzOvHjBE8yVOjVTlgm2hPBWMDySazz6owMh3XBCMMMz9rN3H4XQ0CJ 2R3bN22sJm363D2DlauHoTAMuAz8958fPzMFYVFxdSXZBH2V6Ck8Q3fi4MA/fnI= =h7J3 -----END PGP SIGNATURE-----

2 1

C code coverage tool
by Michael Droettboom Oct. 27, 2009

Oct. 27, 2009

I know David Cournapeau has done some work on using gcov for coverage with Numpy. Unaware of this, (doh! -- I should have Googled first), I wrote a small C code-coverage tool built on top of valgrind's callgrind tool, so it basically only works on x86/AMD64 unixy platforms, but unlike gcov it doesn't require any recompilation headaches (though compiling unoptimized helps). It's about 200 lines of Python code that parses callgrind's output and generates text and HTML. It has specialized support for the code generation used in Numpy -- each line is marked not only by *if* it ran, but in which version of the function it ran. I've put an example of its output from the numpy unit tests up here (temporary address): http://www.droettboom.com/c_coverage/ A particularly interesting file is this one: http://www.droettboom.com/c_coverage/numpy_core_src_multiarray_arraytypes.c… Is this something we want to add to the SVN tree, maybe under tools? Usage instructions are below. Mike =============== C coverage tool =============== This is a tool to generate C code-coverage reports using valgrind's callgrind tool. Prerequisites ------------- * `Valgrind <http://www.valgrind.org/>`_ (3.5.0 tested, earlier versions may work) * `Pygments <http://www.pygments.org/>`_ (0.11 or later required) C code-coverage --------------- Generating C code coverage reports requires two steps: * Collecting coverage results (from valgrind) * Generating a report from one or more sets of results For most cases, it is good enough to do:: > c_coverage_collect.sh python -c "import numpy; numpy.test()" > c_coverage_report.py callgrind.out.pid which will run all of the Numpy unit tests, create a directory called `coverage` and write the coverage report there. In a more advanced scenario, you may wish to run individual unit tests (since running under valgrind slows things down) and combine multiple results files together in a single report. Collecting results `````````````````` To collect coverage results, you merely run the python interpreter under valgrind's callgrind tool. The `c_coverage_collect.sh` helper script will pass all of the required arguments to valgrind. For example, in typical usage, you may want to run all of the Numpy unit tests:: > c_coverage_collect.sh python -c "import numpy; numpy.test()" This will output a file ``callgrind.out.pid`` containing the results of the run, where ``pid`` is the process id of the run. Generating a report ``````````````````` To generate a report, you pass the ``callgrind.out.pid`` output file to the `c_coverage_report.py` script:: > c_coverage_report.py callgrind.out.pid To combine multiple results files together, simply list them on the commandline or use wildcards:: > c_coverage_report.py callgrind.out.* Options ''''''' * ``--directory``: Specify a different output directory * ``--pattern``: Specify a regex pattern to match for source files. The default is `numpy`, so it will only include source files whose path contains the string `numpy`. If, for instance, you wanted to include all source files covered (that are available on your system), pass ``--pattern=.``. * ``--format``: Specify the output format(s) to generate. May be either ``text`` or ``html``. If ``--format`` is not provided, both formats will be output. Reading a report ---------------- The C code coverage report is a flat directory of files, containing text and/or html files. The files are named based on their path in the original source tree with slashes converted to underscores. Text reports ```````````` The text reports add a prefix to each line of source code: - '>' indicates the line of code was run - '!' indicates the line of code was not run HTML reports ```````````` The HTML report highlights the code that was run in green. The HTML report has special support for the "generated" functions in Numpy. Each run line of code also contains a number in square brackets indicating the number of different generated functions the line was run in. Hovering the mouse over the line will display a list of the versions of the function in which the line was run. These numbers can be used to see if a particular line was run in all versions of the function. Caveats ------- The coverage results occasionally misses lines that clearly must have been run. This usually can be traced back to the compiler optimizer removing lines because they are tautologically impossible or to combine lines together. Compiling Numpy without optimizations helps, but not completely. Even despite this flaw, this tool is still helpful in identifying large missed blocks or functions. -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA

2 2

fftpack_lite question
by Ralf Gommers Oct. 26, 2009

Oct. 26, 2009

Hi all, Can anyone tell me if fftpack_lite is an exact C translation of the fftpack Fortran code? Or at least close enough that the signature, parameter descriptions and algorithm are the same? If so, I can use the fftpack Fortran sources (which have useful comments) to write docs for fftpack_lite funcs (rfft* and cfft*). Thanks, Ralf

3 5

500 internal server error from docs.scipy.org
by Neal Becker Oct. 26, 2009

Oct. 26, 2009

This link: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.var.html#sc… gives 500 internal server error

2 1

Procedure for doing documentation reviews
by Christopher Burns Oct. 26, 2009

Oct. 26, 2009

When a documents status is "Needs Review" and when reviewing it we feel it needs edits, should we add comments regarding the edits, or should we feel free to edit it directly? Chris

3 2