REPOST: Re: affects on extended modules
cjensen at bioeng.ucsd.edu
Fri Dec 28 02:05:10 CET 2001
Pedro <pedro_rodriguez at club-internet.fr> wrote in message news:<pan.2001.12.06.12.45.41.172197.2456 at club-internet.fr>...
> "Curtis Jensen" <cjensen at bioeng.ucsd.edu> wrote:
> > Kragen Sitaker wrote:
> >> Curtis Jensen <cjensen at bioeng.ucsd.edu> writes:
> >> > We have created a python interface to some core libraries of our own
> >> > making. We also have a C interface to these same libraries. However,
> >> > the the python interface seems to affect the speed of the extended
> >> > libraries. ie. some library routines have their own benchmark code,
> >> > and the time of exection from the start of the library routine to the
> >> > end of the library routine (not including any python code execution),
> >> > takes longer than it's C counterpart.
> >> In the Python version, the code is in a Python extension module, right?
> >> A .so or .dll file? Is it also in the C counterpart? (If that's not
> >> it, can you provide more details on how you compiled and linked the
> >> two?)
> >> In general, referring to dynamically loaded things through symbols ---
> >> even from within the same file --- tends to be slower than referring to
> >> things that aren't dynamically loaded.
> >> What architecture are you on? If you're on the x86, maybe Numeric is
> >> being stupid and allocating things that aren't maximally aligned. But
> >> you'd probably notice a pretty drastic difference in that case.
> >> ... or maybe Numeric is being stupid and allocating things in a way
> >> that causes cache-line contention.
> >> Hope this helps.
> > Thanks for the responce. The C counterpart is directly linked together
> > into one large binary (yes, the python is using a dynamicaly linked
> > object file, a .so). So, That might be the source of the problem. I
> > can try and make a dynamicaly linked version of the C counterpart and
> > see how that affects the speed. We are running on IRIX 6.5 machines
> > (mips).
> > Thanks.
> Don't know if this helps but I had a similar problem on Linux.
> The context was : a python script was calling an external program and
> parsing output (with popen) many times. I decided to optimize this
> by turning the external program into a dynamicaly linked library with
> python bindings. I expected to gain the extra system calls to fork and
> start a new process, but it turned out that this solution was slower.
> The problem was caused by multithreading stuff. When using the library
> straight from a C program, I didn't link with multithreaded libraries
> and so all system calls weren't protected (they don't need to lock and
> unlock their resources).
> Unfortunately, the library was reading files with fgetc (character by
> character :( ). Since the Python version I used was compiled with
> multi-threading enabled, it turned out that the fgetc function used in
> this case lock/unlock features, which cause the extra waste of time.
> To find this, I compiled my library with profiling (I think I needed to
> use some system call to activate profiling from the library, since I
> couldn't rebuild Python).
> OT : at the end I fixed the library (fgetc replaced by fgets), and didn't
> gain anything by turning the external program into a python extension.
> Since it seemed that Linux disk cache was good, I removed the python
> extension thus keeping a pure Python program, and implemented a cache
> for the results of the external program. This was much simpler and more
> efficient in this case.
Is this a problem with i/o only? Our the code sections that we
benchmarked has no i/o in it.
========= WAS CANCELLED BY =======:
From: cjensen at bioeng.ucsd.edu (Curtis Jensen)
Subject: cmsg cancel <1be8d8b.0112271705.50940295 at posting.google.com>
Control: cancel <1be8d8b.0112271705.50940295 at posting.google.com>
Date: Mon, 31 Dec 2001 02:20:07 GMT
Organization: A poorly-installed InterNetNews site
Message-ID: <cancel.1be8d8b.0112271705.50940295 at posting.google.com>
X-Trace: news2.kornet.net 1009775461 27193 18.104.22.168 (31 Dec 2001 05:11:01 GMT)
X-Complaints-To: usenet at news2.kornet.net
NNTP-Posting-Date: Mon, 31 Dec 2001 05:11:01 +0000 (UTC)
X-Commentary: I love NewsAgent 1.10 and the Sandblaster Cancel Engine Build 74 (19 March 1999)
This message was cancelled from within Mozilla.
More information about the Python-list