[Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)

[Jean-Paul Calderone]

On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" <amk at amk.ca> wrote:
>On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote:
>> >That feels like 6 layers too many, given that
>> > _logrun(selectable, _drdw, selectable, method, dict)
>> > return context.call({ILogContext: newCtx}, func, *args, **kw)
>> > return self.currentContext().callWithContext(ctx, func, *args, **kw)
>> > return func(*args, **kw)
>> > getattr(selectable, method())
>> > klass(number, string)
>> >
>> >are all generic calls.
>>
>> I know function calls are expensive in Python, and method calls even more
>> so... but I still don't understand this issue.  Twisted's call stack is too
>> deep?  It is fair to say it is deep, I guess, but I don't see how that is a
>> problem.  If it is, I don't see how it is specific to this discussion.
>
>It's hard to debug the resulting problem.  Which level of the *12*
>levels in the stack trace is responsible for a bug?  Which of the *6*
>generic calls is calling the wrong thing because a handler was set up
>incorrectly or the wrong object provided?  The code is so 'meta' that
>it becomes effectively undebuggable.

I've debugged plenty of Twisted applications.  So it's not undebuggable. :)

Application code tends to reside at the bottom of the call stack, so Python's
traceback order puts it right where you're looking, which makes it easy to
find.  For any bug which causes something to be set up incorrectly and only
later manifests as a traceback, I would posit that whether there is 1 frame or
12, you aren't going to get anything useful out of the traceback.  Standard
practice here is just to make exception text informative, I think, but this is
another general problem with Python programs and event loops, not one specific
to either Twisted itself or the particular APIs Twisted exposes.

As a personal anecdote, I've never once had to chase a bug through any of the
6 "generic calls" singled out.  I can't think of a case where I've helped any
one else who had to do this, either.  That part of Twisted is very old, it is
_very_ close to bug-free, and application code doesn't have very much control
over it at all.  Perhaps in order to avoid scaring people, there should be a
way to elide frames from a traceback (I don't much like this myself, I worry
about it going wrong and chopping out too much information, but I have heard
other people ask for it)?

Jean-Paul