[Async-sig] Simplifying stack traces for tasks?

Mark E. Haase mehaase at gmail.com
Fri Apr 13 11:55:38 EDT 2018

Thanks for the feedback! I put this aside for a while but I'm coming back
to it now and cleaning it up.

The approach used in this first post was obviously very clumsy. In my
latest version I am using module instance directly (as shown in Nathaniel's
reply) and using the qualified package name (as suggested by Roger). I
created an explicit blacklist (incomplete--still needs more testing) of
functions to hide in my custom backtraces and refactored a bit so I can
write tests for it. Code below.

One interesting thing I learned while working on this is that the
backtraces change depending on the asyncio debug mode, because in debug
mode couroutines are wrapped in CoroWrapper[1], which adds a frame
everytime a coroutine sends, throws, etc. So I am now thinking that my
custom excepthook is probably most useful in debug mode, but probably not
good to enable in production.

I'm working on a more general asyncio task group library that will include
this excepthook. I'll release the whole thing on PyPI when it's done.



def _async_excepthook(type_, exc, tb):
    An ``excepthook`` that hides event loop internals and displays
    task group information.

    :param type type_: the exception type
    :param Exception exc: the exception itself
    :param tb tb: a traceback of the exception
    print(_async_excepthook_format(type_, exc, tb))

def _async_excepthook_format(type_, exc, tb):
    This helper function is used for testing.

    :param type type_: the exception type
    :param Exception exc: the exception itself
    :param tracetack tb: a traceback of the exception
    :return: the formatted traceback as a string
    format_str = ''
    cause_exc = None
    cause_str = None

    if exc.__cause__ is not None:
        cause_exc = exc.__cause__
        cause_str = 'The above exception was the direct cause ' \
                    'of the following exception:'
    elif exc.__context__ is not None and not exc.__suppress_context__:
        cause_exc = exc.__context__
        cause_str = '\nDuring handling of the above exception, ' \
                    'another exception occurred:'

    if cause_exc:
        format_str += _async_excepthook_format(type(cause_exc), cause_exc,

    if cause_str:
        format_str += '\n{}\n\n'.format(cause_str)

    format_str += 'Async Traceback (most recent call last):\n'

    # Need file, line, function, text
    for frame, line_no in traceback.walk_tb(tb):
        if _async_excepthook_exclude(frame):
            format_str += '  ---\n'
            code = frame.f_code
            filename = code.co_filename
            line = linecache.getline(filename, line_no).strip()
            format_str += '  File "{}", line {}, in {}\n' \
                .format(filename, line_no, code.co_name)
            format_str += '    {}\n'.format(line)

    format_str += '{}: {}'.format(type_.__name__, exc)
    return format_str

    'asyncio.base_events': ('_run_once', 'call_later', 'call_soon'),
    'asyncio.coroutines': ('__next__', 'send', 'throw'),
    'asyncio.events': ('__init__', '_run'),
    'asyncio.tasks': ('_step', '_wakeup'),
    'traceback': ('extract', 'extract_stack'),

def _async_excepthook_exclude(frame):
    ''' Return True if ``frame`` should be excluded from tracebacks. '''
    module = frame.f_globals['__name__']
    function = frame.f_code.co_name
    return module in _ASYNC_EXCEPTHOOK_BLACKLIST and \
         function in _ASYNC_EXCEPTHOOK_BLACKLIST[module]

On Tue, Nov 14, 2017 at 7:15 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Nov 14, 2017 at 2:00 PM, Roger Pate <rogerpate at gmail.com> wrote:
> > On Tue, Nov 14, 2017 at 9:54 AM, Mark E. Haase <mehaase at gmail.com>
> wrote:
> > ...
> >>         print('Async Traceback (most recent call last):')
> >>         for frame in traceback.extract_tb(tb):
> >>             head, tail = os.path.split(frame.filename)
> >>             if (head.endswith('asyncio') or tail == 'traceback.py') and
> \
> >>                 frame.name.startswith('_'):
> > ...
> >> The meat of it is towards the bottom, "if head.endswith('asyncio')..."
> There
> >> are a lot of debatable details and this implementation is pretty hacky
> and
> >> clumsy, but I have found it valuable in my own usage, and I haven't yet
> >> missed the omitted stack frames.
> >
> > It would be better to determine if the qualified module name is
> > "traceback" or starts with "asyncio." (or topmost package is
> > "asyncio", etc.) rather than allow false positives for
> > random_package.asyncio.module._any_function or
> > random_package.traceback._any_function.  I don't see an easy way to
> > get the module object at this point in your hook; however:
> You can't get the module from the cooked data that extract_tb returns,
> but it's there in the tb object itself. This walks the traceback and
> prints each frame's module:
> current = tb
> while current is not None:
>     print("Next module", current.tb_frame.f_globals.get("__name__"))
>     current = current.tb_next
> -n
> --
> Nathaniel J. Smith -- https://vorpus.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180413/5a8de716/attachment.html>

More information about the Async-sig mailing list