[Python-Dev] Performance of pre-creating exceptions?

[Andrew Dalke]

On 3/2/07, Adam Olsen <rhamph at gmail.com> wrote:
> We can get more than half of the benefit simply by using a default
> __init__ rather than a python one.  If you need custom attributes but
> they're predefined you could subclass the exception and have them as
> class attributes.  Given that, is there really a need to pre-create
> exceptions?

The only real world example of (re)using pre-computed exceptions
I found was in pyparsing.  Here are two examples:

    def parseImpl( self, instring, loc, doActions=True ):
        if (instring[loc] == self.firstMatchChar and
            (self.matchLen==1 or instring.startswith(self.match,loc)) ):
            return loc+self.matchLen, self.match
        #~ raise ParseException( instring, loc, self.errmsg )
        exc = self.myException
        exc.loc = loc
        exc.pstr = instring
        raise exc

(The Token's constructor is

class Token(ParserElement):
    def __init__( self ):
        super(Token,self).__init__( savelist=False )
        self.myException = ParseException("",0,"",self)

and the exception class uses __slots__ thusly:

class ParseBaseException(Exception):
    """base exception class for all parsing runtime exceptions"""
    __slots__ = ( "loc","msg","pstr","parserElement" )
    # Performance tuning: we construct a *lot* of these, so keep this
    # constructor as small and fast as possible
    def __init__( self, pstr, loc, msg, elem=None ):
        self.loc = loc
        self.msg = msg
        self.pstr = pstr
        self.parserElement = elem


so you can see that each raised exception modifies 2 of
the 4 instance variables in a ParseException.)

-and-

    # this method gets repeatedly called during backtracking with the
same arguments -
    # we can cache these arguments and save ourselves the trouble of re-parsing
    # the contained expression
    def _parseCache( self, instring, loc, doActions=True, callPreParse=True ):
        lookup = (self,instring,loc,callPreParse)
        if lookup in ParserElement._exprArgCache:
            value = ParserElement._exprArgCache[ lookup ]
            if isinstance(value,Exception):
                if isinstance(value,ParseBaseException):
                    value.loc = loc
                raise value
            return value
        else:
            try:
                ParserElement._exprArgCache[ lookup ] = \
                    value = self._parseNoCache( instring, loc,
doActions, callPreParse )
                return value
            except ParseBaseException, pe:
                ParserElement._exprArgCache[ lookup ] = pe
                raise


The first definitely has the look of a change for better performance.
I have not asked the author nor researched to learn how much gain
there was with this code.

Because the saved exception is tweaked each time (hence not
thread-safe), your timing tests aren't directly relevant as your
solution of creating the exception using the default constructor
then tweaking the instance attributes directly would end up
doing 4 setattrs instead of 2.

% python -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try: raise
e' 'except: pass'
1000000 loops, best of 10: 1.55 usec per loop
% python -m timeit -r 10 -n 1000000 -s 'e = Exception()' 'try:
e.x=1;e.y=2;raise e' 'except: pass'
1000000 loops, best of 10: 1.98 usec per loop

so 4 attributes should be about 2.5usec, or 25% slower than 2 attributes.


There's also a timing difference between looking for the exception
class name in module scope vs. using self.myException

I've tried to find other examples but couldn't in the 20 or so packages
I have on my laptop.  I used searches like

# find module variables assigned to exception instances
egrep "^[a-z].*=.*Error\(" *.py
egrep "^[a-z].*=.*Exception\(" *.py

# find likely instances being raised
grep "^ *raise [a-z]" *.py
# find likely cases of 3-arg raises
grep "^ *raise .*,.*," *.py

to find candidates.  Nearly all false positives.

        Andrew
        dalke at dalkescientific.com