[SciPy-user] conversion of c++ vector in weave
Michael Sorich
mike_sorich at hotmail.com
Mon Oct 14 20:57:21 EDT 2002
distMatrices will usually be 3x3 or 4x4 (perhaps 5x5).
Number of distMatrices will be between 1 and 20000 per function call in
most cases, though in some cases there will be around 500 000
distMatrices.
Presently I am only using ~20 bins, however I would like to be able to
scale this to ~1000 bins per run for later work.
I am interested to see how you will do this!
Michael
-----Original Message-----
From: scipy-user-admin at scipy.net [mailto:scipy-user-admin at scipy.net] On
Behalf Of eric jones
Sent: Tuesday, 15 October 2002 5:17 AM
To: scipy-user at scipy.net
Subject: RE: [SciPy-user] conversion of c++ vector in weave
I think Numeric will work fine for you (with perhaps a wee bit of
weave). You'll need to use a mixture of arrays for "fixed" sized data
blocks and list for things that are changing size all the time. Give me
a little more info on your needs and I'll send back my take on the
algorithm:
In a typical run:
1. Are the distMatrix ever very large, or are they always 3x3?
2. Do you have a zillion distMatrices in a typical run, or just a few?
3. Do you have zillions of bins or just a few?
I think if 1 is true, then you'll need to weavify much of the code, but
if only 2 and 3 are true, then you only need a few lines of weave.
Eric
----------------------------------------------
eric jones 515 Congress Ave
www.enthought.com Suite 1614
512 536-1057 Austin, Tx 78701
> -----Original Message-----
> From: scipy-user-admin at scipy.net [mailto:scipy-user-admin at scipy.net]
On
> Behalf Of Michael Sorich
> Sent: Sunday, October 13, 2002 8:36 PM
> To: scipy-user at scipy.net
> Subject: RE: [SciPy-user] conversion of c++ vector in weave
>
> Thanks for the help, Eric. Sorry about the lack of details.
>
> The part that I want to speed up takes a distance matrix (array) and a
> list of bins (which can be overlapping) and returns all the
combinations
> of bins that the values in the distance matrix upper triangle fall
> within, as a list of bin index lists.
>
> Example:
> distMatrix = [[0.,1.,2.],[1.,0.,3.],[2.,3.,0.]]
> bins = [[0.9,2.1],[1.9,3.1],[2.9,4.1]], where the inner lists contain
> the min and max values of the bin.
> In this example there are 3 distances to be binned (1.,2.,3.).
distance
> 1 can be binned into bin 0, dist 2 into bins 0 and 1 and dist 3 into
> bins 1 and 2. The combinations include [0,0,1],[0,0,2],[0,1,1],[0,1,2]
>
> It is a little more complicated than above, in that the function takes
> multiple dist matrices (these are stored in the third dimension of the
> distMatrix variable).
>
> I have implemented this in python using lists to store the
combinations,
> because I could not see any way to determine the size of the array at
> the time of construction. I figured that vectors in C++ would be the
> best replacement for the lists, however I was unsure what would be the
> most efficient method to return the results stored in the vector.
Thanks
> for the examples of how to do this. What I have ended up doing is
> creating a PyArray (since at the end the size of the vector is known)
> and copying the data across and then returning the PyArray. It wasn't
as
> complicated as I had feared.
>
> I have attached the python and c versions of the function, if you are
> interested. The algorithm I made is a quick hack. If you know of a
> better way please let me know!
>
> Thanks again,
>
> Michael
>
> -----Original Message-----
> From: scipy-user-admin at scipy.net [mailto:scipy-user-admin at scipy.net]
On
> Behalf Of eric jones
> Sent: Friday, 11 October 2002 5:49 PM
> To: scipy-user at scipy.net
> Subject: RE: [SciPy-user] conversion of c++ vector in weave
>
> Hey Michael,
>
> Can you give me a little more information about your application --
> especially what your data structure looks like. This'll help us
> give more informed ideas on how to speed up the code and whether
> weave is going to help or not.
>
> If you are working with a list of numbers, then the following example
> might give you some clues. Note, though, that the fastest versions
> assume
> you are using a homogeneous list of integers and doesn't do any error
> checking at all. This'll lead to bad things if you pass in a list
with
> a
> string in it. Further, if your data looks like this, your data would
> fit
> nicely into Numeric arrays, so I'm betting this isn't the ticket.
>
> YOU'LL NEED THE LATEST WEAVE FROM CVS TO RUN THE EXAMPLES !!
>
> I've made quite a few changes in the last week. I'm finally to the
> point
> of updating the documentation. When that is done, I'll release weave
> 0.3.
>
> regards,
> eric
>
>
#-----------------------------------------------------------------------
> -----
> # Example converting list to std::vector
> # Typical run:
> # h:\tmp>python std_vec.py
> # a initially: [1, 2, 3]
> # a on return: [101, 102, 103]
> # time it: N = 1000000
> # python-loop (sec): 0.56299996376
> # python-list-comprehension (sec): 0.827999949455
> # simple weave (sec): 0.139999985695
> # fast weave(sec): 0.0470000505447
> # weave-no-vector(sec): 0.0460000038147
>
#-----------------------------------------------------------------------
> -----
>
> import time
> import weave
>
> code = """
> const int Na = a.length();
> int i;
> // convert to std::vector
> std::vector<int> v_a(Na);
> for(i = 0; i < Na; i++)
> v_a[i] = a[i];
>
> // do something with the vector
> for(i = 0; i < Na; i++)
> v_a[i] = v_a[i] + 100;
>
> // put results back into a
> for(i = 0; i < Na; i++)
> a[i] = v_a[i];
> """
> a = [1,2,3]
> print "a initially:", a
> weave.inline(code,['a'],headers=["<vector>"])
> print "a on return:", a
>
> N = 1000000
> print "time it: N =", N
>
>
#-----------------------------------------------------------------------
> -----
> # Python
>
#-----------------------------------------------------------------------
> -----
> a = range(N)
> t1 = time.time()
> for i in xrange(N):
> a[i] = a[i] + 100
> t2 = time.time()
> print "python-loop (sec):", t2 - t1
>
>
#-----------------------------------------------------------------------
> -----
> # Python List Comprehension
>
#-----------------------------------------------------------------------
> -----
> a = range(N)
> t1 = time.time()
> a = [x + 100 for x in a]
> t2 = time.time()
> print "python-list-comprehension (sec):", t2 - t1
>
>
#-----------------------------------------------------------------------
> -----
> # Simple weave
>
#-----------------------------------------------------------------------
> -----
> a = range(N)
> t1 = time.time()
> weave.inline(code,['a'],headers=["<vector>"])
> t2 = time.time()
> print "simple weave (sec):", t2 - t1
>
>
#-----------------------------------------------------------------------
> -----
> # Python API
>
#-----------------------------------------------------------------------
> -----
> api_code = """
> const int Na = a.length();
> int i;
> // convert to std::vector
> std::vector<int> v_a(Na);
> for(i = 0; i < Na; i++)
> v_a[i] = PyInt_AS_LONG(PyList_GET_ITEM(py_a,i));
>
> // do something with the vector
> for(i = 0; i < Na; i++)
> v_a[i] = v_a[i] + 100;
>
> // put results back into a
> for(i = 0; i < Na; i++)
> PyList_SET_ITEM(py_a,i,PyInt_FromLong(v_a[i]));
> """
> a = [1,2,3]
> weave.inline(api_code,['a'],headers=["<vector>"])
>
> a = range(N)
> t1 = time.time()
> weave.inline(api_code,['a'],headers=["<vector>"])
> t2 = time.time()
> print "fast weave(sec):", t2 - t1
>
>
#-----------------------------------------------------------------------
> -----
> # Python API -- no std::vector
>
#-----------------------------------------------------------------------
> -----
> api_code = """
> const int Na = a.length();
> int i, val;
> // convert to std::vector
> for(i = 0; i < Na; i++)
> {
> val = PyInt_AS_LONG(PyList_GET_ITEM(py_a,i));
> PyList_SET_ITEM(py_a,i,PyInt_FromLong(val+100));
> }
> """
> a = [1,2,3]
> weave.inline(api_code,['a'],headers=["<vector>"])
>
> a = range(N)
> t1 = time.time()
> weave.inline(api_code,['a'],headers=["<vector>"])
> t2 = time.time()
> print "weave-no-vector(sec):", t2 - t1
>
> > -----Original Message-----
> > From: scipy-user-admin at scipy.net [mailto:scipy-user-admin at scipy.net]
> On
> > Behalf Of Michael Sorich
> > Sent: Friday, October 11, 2002 2:02 AM
> > To: scipy-user at scipy.net
> > Subject: [SciPy-user] conversion of c++ vector in weave
> >
> > Hi
> >
> > I wish to convert a small part of my code to C++ for speed. The code
> is
> > not amenable to numeric arrays and uses lists extensively. I am
> thinking
> > about using c++ vectors in place of python lists (I presume that
this
> > will be a lot faster than using Py::List). What little experience I
> have
> > in extending python with C/C++ is limited to using weave.
> >
> > If I use vectors, is there any method to automatically convert my
> result
> > (vector< vector<int> >) to something that can be returned to python
> > (tuple/list/array). I believe that Boost can do this (at least the
> > upcoming v2, see
> >
>
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/li
> > bs/python/doc/v2/faq.html), however I am not sure how I would be
able
> to
> > integrate this with weave.inline. It does not appear obvious that
> there
> > is anything in CXX that will do this.
> >
> > Any help would be appreciated.
> >
> > Thanks
> >
> > Michael Sorich
> > PhD Student
> > School of Pharmaceutical, Molecular and Biomedical Sciences
> > University of South Australia
> > Email: michael.sorich at postgrads.unisa.edu.au
> > mike_sorich at hotmail.com
> >
> >
> >
> > ---
> > Outgoing mail is certified Virus Free.
> > Checked by AVG anti-virus system (http://www.grisoft.com).
> > Version: 6.0.393 / Virus Database: 223 - Release Date: 30/09/2002
> >
> > _______________________________________________
> > SciPy-user mailing list
> > SciPy-user at scipy.net
> > http://www.scipy.net/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.net
> http://www.scipy.net/mailman/listinfo/scipy-user
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.393 / Virus Database: 223 - Release Date: 30/09/2002
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.393 / Virus Database: 223 - Release Date: 30/09/2002
>
>
_______________________________________________
SciPy-user mailing list
SciPy-user at scipy.net
http://www.scipy.net/mailman/listinfo/scipy-user
---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.393 / Virus Database: 223 - Release Date: 30/09/2002
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.393 / Virus Database: 223 - Release Date: 30/09/2002
More information about the SciPy-User
mailing list