NumPy-Discussion September 2007

numpy-discussion@python.org

76 participants
71 discussions

Documentation
by Charles R Harris 13 Sep '07

13 Sep '07

Hi All, While doing documentation, I've run into the need for a simple word to express the fact that a variable can be an array, or anything that can be converted to an array using asarray(). I have been using array_like for this kind of variable, but I am open to suggestions. Also, the current version of epydoc doesn't like None as a default keyword when doing module introspection. On my machine this yields [{'bogomips': '5066.01', 'cache size': '4096 KB', 'clflush siz... as the value. I suspect this can be fixed in epydoc. There are a few other minor glitches as well. Chuck

1 0

pycdf probs
by Mathew Yeates 13 Sep '07

13 Sep '07

Anybody know how to contact the pycdf author? His name is Gosselin I think. There are hardcoded values that cause pycdf to segfault when using large strings. Mathew

2 1

ANN: Reminder - Texas Python Regional Unconference
by Travis Vaught 12 Sep '07

12 Sep '07

Greetings, Just a reminder for those in the area... http://pycamp.python.org/Texas/HomePage The Unconference is to be held this weekend (Saturday and Sunday, September 15, 16) at the Texas Learning & Computing Center at the University of Houston main campus. It's free. Sign up by adding your name to the wiki page. Travis

1 0

Optimizing similarity matrix algorithm
by Kurdt Bane 12 Sep '07

12 Sep '07

Hi to all! For reverse engineering purposes, I need to find where every possible chunk of bytes in file A is contained in file B. Obviously, if a chunk of length n is contained in B, I dont' want my script to recognize also all the subchunks of size < n contained in the chunk. I coded a naive implementation of a similarity algorithm: scan the similarity algorithm and find all the diagonals. Here's the code: file_a = open(argv[1]) file_b = open(argv[2]) a = numpy.fromstring(file_a.read(),'c') b = numpy.fromstring(file_b.read(),'c') tolerance = int(argv[3]) chunks = [] valid = True count_xcent = 0 for x in xrange(len(a)): for y in xrange(len(b)): count = 0 if (a[x] == b[y]): x_cnt, y_cnt = x,y if (a[x_cnt] == b[y_cnt]): try: while (a[x_cnt+1] == b[y_cnt+1]): count += 1 x_cnt += 1 y_cnt += 1 except IndexError: pass if ((count > tolerance) or (count == tolerance)): for tuple in chunks: if (((x >= tuple[0]) and (x_cnt <= tuple[1])) and ((y >= tuple[2]) and (y_cnt <= tuple[3]))): valid = False if __debug__: print "Found an already processed subchunk" break if (valid): chunks.append ((x, x_cnt, y, y_cnt)) print "Corresponding chunk found. List 1: from " + str(x) + " to " + str(x_cnt) +". List 2: from " + str(y) + " to " + str(y_cnt) print "with length of " + str (x_cnt + 1 - x) else: valid = True It simply scans the similarity matrix, finds the diagonals in wich a[x] == a[y] and interprets the diagonal as a chunk. Then, it stores the chunk in a list and determines if it's a subchunk of another greater chunk already found. The problem is: this implementation is very slow, imho because of three factors: 1. I use a nested for loop to scan the matrix 2. When the start of a diagonal is found, the program scans the diagonal with another additional loop. Maybe it would be faster to use a function such as diagonal() (but I can't actually _create_ the boolean similarity matrix, as it gets way too big for files big enough (we're talking about ~ 100 Kb files) - I am forced to compute it "on the way". 3. When I find a chunk, I compute all its subchunks with this approach, and compare them with the "big" chunk I stored in the list. Maybe there's a better algorithmical way. What do you think about these issues? Is there a way to optimize them? And are there other issues I didn't take in account? Thanks in advance, regards, Chris.

1 1

how to include numpy headers when building an extension?
by Christopher Barker 12 Sep '07

12 Sep '07

Hi all, I'm porting an extension from Numeric. At the top, I've changed: #include <Numeric/arrayobject.h> to #include <numpy/arrayobject.h> But distutils can't find numpy/arrayobject.h -- how do I tell distutils where to look for it? I've gotten it to work by hard-coding the entire path, but that's not very portable. It looks like Numeric used to put its headers in a standard location, but that numpy buries them deeper. thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

6 10

odd behavior for numpy.ndarray index?
by Orest Kozyar 10 Sep '07

10 Sep '07

In the following output (see below), why would x[1,None] work, but x[1,None,2] or even x[1,2,None] not work? Incidentally, I would be very interested in a solution that allows me to index numpy arrays using a list/iterable that might contain None values. Is there a straightforward way to do so, or should I just use list comprehensions (i.e. [x[a] for a in indices if a])? Thanks! Orest In [122]: x = arange(5) In [123]: x[1] Out[123]: 1 In [124]: x[None] Out[124]: array([[0, 1, 2, 3, 4]]) In [125]: x[1,None] Out[125]: array([1]) In [126]: x[1,None,2] --------------------------------------------------------------------------- <type 'exceptions.IndexError'> Traceback (most recent call last) c:\documents\research\programs\python\epl\src\<ipython console> in <module>() <type 'exceptions.IndexError'>: invalid inde

2 1

von mises distribution in numpy.random biased
by killian koepsell 10 Sep '07

10 Sep '07

hi, the von mises distribution in numpy.random seems to be biased towards a higher concentration (kappa). given a concentration of 2, it produces data that has a concentration of 2.36. i compared the distribution to the one produced by the CircStats[1] package of R[2] using RPy [3] and created a figure here: http://redwood.berkeley.edu/kilian/vonmises.png the script i used is attached to this email. i don't know what algorithm NumPy uses, so i can't tell if it is a real bug or some sort of rounding error. the CircStats package uses the algorithm by Best and Fisher [4]. kilian [1] http://cran.r-project.org/src/contrib/Descriptions/CircStats.html [2] http://www.r-project.org/ [3] http://rpy.sf.net/ [4] Best, D. and Fisher, N. (1979). Efficient simulation of the von Mises distribution. Applied Statistics, 24, 152-157. ---------------------- vonmises.py ---------------------- from numpy import array,pi,exp,mod,random from pylab import hist,linspace,figure,clf,plot,subplot,xlim,title,legend from rpy import r as R R.library("CircStats") # compute von Mises distribuion fo two different values of kappa size = 1e5 kappa1 = 1 x1 = random.vonmises(0,kappa1,size=size) y1 = mod(R.rvm(size,0,kappa1),2*pi) kappa2 = 2 x2 = random.vonmises(0,kappa2,size=size) y2 = mod(R.rvm(size,0,kappa2),2*pi) def phasedist(x,kappa): ph = linspace(-pi,pi,50) vM = R.dvm(ph,0,kappa) kest = R.A1inv(abs(exp(1j*array(x)).mean())) vMest = R.dvm(ph,0,kest) plot(ph,vM,'r',linewidth=2) plot(ph,vMest,'k:',linewidth=2) legend(['$\\kappa=%d$'%kappa,'$\\hat{\\kappa}=%1.2f$'%kest]) hist(x,ph,normed=1) xlim((-pi,pi)) # plot figure fig=figure(1) clf() ax1=subplot(2,2,1) title('NumPy.random') phasedist(x1,kappa1) ax2=subplot(2,2,3) phasedist(x2,kappa2) subplot(2,2,2,sharey=ax1) title('RPy.CircStats') phasedist(y1,kappa1) subplot(2,2,4,sharey=ax2) phasedist(y2,kappa2)

2 2

why system_info.check_libs does not look for dll on windows ?
by David Cournapeau 08 Sep '07

08 Sep '07

Hi, I would like to know if there is a reason why system_info does not look for dll on windows ? I think it would make sense to look for dll when you want to use an external lib through ctypes, for example. cheers, David

2 2

Importing data from html tables
by ale 07 Sep '07

07 Sep '07

Hi, I'm trying to import into array the data contained in a html table. I use BeautifulSoup as html parser html = open('T0015.html','r') bs = BeautifulSoup(html) for tr in bs.findAll('tr')[1:]: table.append([td.p.string for td in tr.findAll('td')]) and I get this: print table [[u'1925', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'105.0'] [u'1926', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'136.0'] [u'1927', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'--', u'51.0'] [u'1928', u'--', u'--', u'--', u'nn', u'--', u'--', u'--', u'--', u'104.0'] ,.......and so on] How to put this list of list of strings in a numpy array, and set '--' and 'nn' as NaN? Thank you Alessio

2 1

corrcoef
by John Hunter 06 Sep '07

06 Sep '07

Is it desirable that numpy.corrcoef for two arrays returns a 2x2 array rather than a scalar In [10]: npy.corrcoef(npy.random.rand(10), npy.random.rand(10)) Out[10]: array([[ 1. , -0.16088728], [-0.16088728, 1. ]]) I always end up extracting the 0,1 element anyway. What is the advantage, aside from backwards compatibility, for returning a 2x2? JDH

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2007