What psyco is goot at [Was: Rookie Speaks]

[Samuel Walters]

|Thus Spake Delaney, Timothy C (Timothy) On the now historical date of
Fri, 09 Jan 2004 09:51:07 +1100|
> As an example, check these pystone results (Python 2.3.3, psyco 1.1.1,
> Windows 2K SP4). Note I've upped the number of passes ...
> 
> The benchmarks with `from psyco.classes import __metaclass__` are
> interesting - they're counter to what one would expect from the docs -
> increased memory usage and decreased performance. Is this a known thing
> Armin, or is something going screwy on my box? It's happening
> consistently.
> 
> Note that psyco.profile() depends heavily on how your code works. I've got
> one piece of code that performs approx 5 times better with psyco.full(),
> but actually performs worse with psyco.profile() than without psyco at
> all!

No matter how badly we wish optimization were a theoretical science, I
think it will always be mostly empirical.  The bottlenecks always seem to
show up in the most unintuitive places.  You can make some educated
guesses based on the Big O nature of your algorithms, but in the end, you
just have to field test.

> 
> Unfortunately, I can't make it available, but it's a single pass over a
> file (line by line), matching and extracting data using various regexes,
> writing lots of other files, then iterating over the other files to
> produce even more files. The datasets tend to be very large - 500MB+.
> 
> The main functions only get called one time each (which is probably the
> killer) and the bulk of the work is done inside while loops in those
> functions. I suspect that if I moved the contents of the main while loops
> to their own functions psyco.profile() would do a lot better.

It seems you've got two forces pulling in opposite directions here:
Function calls are expensive in Python and
Psyco only optimizes functions as a whole.

I've heard of a tool, I believe it's called HotSpot, that will help you
figure out which lines of code, not functions, are the most expensive for
you.  You could then see if wrapping those lines in a function and binding
it with psyco would give you a gain in performance.  You might look into
it.  I haven't researched it, know nothing about it, and it's pretty low
on my to-do list ATM, but it might help you with your needs.


Sam Walters.

-- 
Never forget the halloween documents.
http://www.opensource.org/halloween/
""" Where will Microsoft try to drag you today?
    Do you really want to go there?"""