[Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore())

[Oscar Benjamin]

On 17 October 2013 20:01, Guido van Rossum <guido at python.org> wrote:
> On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin
> <oscar.j.benjamin at gmail.com> wrote:
>>
>> On 17 October 2013 19:40, Xavier Morel <python-dev at masklinn.net> wrote:
>> > I think there's already a significant split between context managers
>> > which handle the lifecycle of a local resource (file, transaction) and
>> > those which purport to locally alter global-ish state (cwd,
>> > decimal.localcontext, logging.captureWarnings, redirect_stdout).
>> >
>> > And the latter worries me (much more than the very localized behavior of
>> > suppress) because I don't see any way to implement them safely and
>> > correctly when mixing it with coroutines in today's Python (some of them
>> > aren't even thread-safe), all of that while I expect coroutines will see
>> > significantly more use in the very near future with yield from and
>> > tulip's promotion of coroutine-style async.
>>
>> I maybe misunderstanding how the  coroutine-style async works but I
>> would have thought that it would be as simple as: don't use
>> global-state-restoring-context-managers around statements that yield
>> control (it would be simpler if there was a good term for describing
>> that kind of CM). That's simpler to implement and computationally
>> cheaper than e.g. the thread-local state used by the decimal module.
>
> Context managers that actually save and restore *global* state are already
> not thread-safe, so concluding they are also not coroutine-safe (or
> task-safe?) seems a small step.
>
> I'd be more worried about context manager that use thread-local state --
> there is no similar concept in Tulip.

It's unnecessary in Tulip. The need for thread-local state in e.g.
decimal contexts is driven by the fact that multi-threaded execution
switches in an uncontrollable way. Tulip specifically makes it
possible to control the points at which a switch occurs making this
safe (even if localcontext() wasn't thread-safe):

with decimal.localcontext() as ctx:
    ctx.prec = 100
    c = a + b
    # more synchronous decimal calculations

# State is restored before allowing other code to execute
yield from save_in_database(c)

So it's fine to use global/thread-local state modifying/restoring
context managers in Tulip as long as you don't yield control to other
code within the with block. (unless I misunderstand - I lost track of
Tulip some time ago).

The issue with decimal.localcontext() and yield arises when using
generators as much as coroutines e.g.:

def exact_sum(nums):
    start = Decimal(0)
    with decimal.localcontext() as ctx:
        ctx.traps[decimal.Inexact] = True
        for num in nums:
            try:
                total += Decimal(num)
            except decimal.Inexact:
                ctx.prec *= 2
    return total

The above is fine for computing the sum of a list of
Decimals/ints/floats. However it fails if you pass in a generator that
carelessly modifies the arithmetic context around yield calls:

def compute():
    with decimal.localcontext() as ctx:
        ctx.prec = 15
        ctx.traps[decimal.Inexact] = False
        yield a + b
        yield b - c
        # etc.

exact_sum(compute())

There needs to be a convention that either functions like exact_sum()
mustn't assume continuity of the context between iterations or a
function like compute() must restore before yielding. IMO the only
sane approach for async coroutines is to say that if you yield or
yield from then it is your responsibility to restore any temporarily
altered global/thread-local state first.


Oscar