Can you make this faster?

[Roy Smith]

In article <889cbba0.0406271022.fd1f9ac at posting.google.com>,
 klachemin at home.com (Kamilche) wrote:

> I have a routine that I really need, but it slows down processing
> significantly. Can you spot any ineffeciencies in the code?
> 
> This code makes a critical function of mine run about 7x slower than
> using a prebuilt format string. For maximum flexibility, it would be
> best to calculate the format string using this method, so I'd dearly
> love to keep it.
> 
> def fmtstring(args):
>     delim = '\0'
>     fmt = []
>     fmt.append('<')
>     for arg in args:
>         t = type(arg)
>         if t == types.StringType:
>             l = len(arg)
>             fmt.append(str(l) + 's')
>         elif t == types.IntType:
>             fmt.append('i')
>         elif t == types.LongType:
>             fmt.append('q')
>         elif t == types.BooleanType:
>             fmt.append('c')
>         elif t == types.FloatType:
>             fmt.append('d')
>         else:
>             raise Exception("Can't pack argument of type %s!" % t)
>     s = ''.join(fmt)
>     s = s + '\0'
>     return s

The first thing is to profile your code and make sure this really is 
significant in the overall scheme of things.  From your description, it 
sounds like you've already done that, so I'll go with that assumption.

What's the most likely value for type(arg)?  If 90% of your args are 
floats, for example, most of the time you work your way all the way down 
the if/else tree before getting to the right spot.  Putting the tests in 
order of probability of occurance will help you a little.

Doing "from types import *" and then being able to just say "IntType" 
instead of "types.IntType" will be a little faster since there will be 
one less name lookup.

Another thing you can try is doing dictionary lookups instead of an 
if/else tree.  Something like:

map = {IntType: 'i',
       LongType: 'q',
       BooleanType: 'c',
       FloatType: 'd'}
[...]
try:
   fmt.append (map[t])
except KeyError:
   raise Exception ("Can't pack %s" % t)

You still need to special-case StringType, however.

Lastly, is an idea which is really quite gross.  If you're desperate for 
speed, the majority of your args are strings, and you have no pride, it 
might be worth trying, however.  Instead of saying:

t = type(arg)
if t == types.StringType:
    l = len(arg)

you could do something like:

try:
   l = len (arg)
   fmt.append(str(l) + 's')
except TypeError:
   process all the non-string types

The idea here is to not waste time getting and testing the type, just 
assume it's a string and deal with the consequences later if your 
assumption was wrong.  In real life, this plays out as, "It's easier to 
ask forgiveness than permission".

Of the types you mentioned, string is the only one which which has a 
length; all the others will raise TypeError when passed to len().  
Exception processing is relatively slow, but if it only happens a small 
percentage of the time, you might be ahead of the game this way.

Some people would call this elegant, others would call it a gross hack.  
I guess it depends on your perspective, and how desperate you are to 
make your code run faster :-)

Of course, you could put several of these ideas together.  Try the 
exception-catching idea for strings first, and then sort out the other 4 
types using a dictionary lookup.

My gut feeling is that even with all these tricks, the best you'll be 
seeing is a factor of 2 or 3 speedup.  I'd be amazed if you could get 
anywhere near the 7x you're looking for.