________________________________________ Frá: Ronald Oussoren [ronaldoussoren@mac.com]

This appears to be simular to the mechanism in PEP 377 http://www.python.org/dev/peps/pep-0377/ which was rejected. Indeed, it is similar, but simpler. (I was unaware of Nick's PEP). The PEP contains the fallacy that in case of __enter__ raising the SkipStatement (or in my case, ContextManagerExit) exception, that something needs to be assigned to the "as" body. The proposal I'm making is to make it possible to do in a single context manager, what it is possible to do with two context managers currently: @contextmanager errordude: 1 // 0 yield @contextmanager handler: try: yield except ZeroDivisionError: pass with handler as a, errordude as b: do_stuff(a, b)

do_stuff will be silently skipped, and b won't be assigned to. I'm proposing a mechanism where the same could be done with a single context manager: with combined as a, b: do_stuff(a, b) Currently, you have programmatic capabilities (silent skipping of the statement) with two context managers that you don't have with a single one. That's just plain odd. I don't think the original objectsions to PEP 377 are valid. My approach introduces very little added complexity. The use case is "correctly being able to combine context managers". It is a "completeness" argument too, that you can do with a single context manager what you can do with two nested ones.

2) The mechanism used in implementing ContextManagerExit above is easily extended to allowing a special context manager: None. This is useful for having _optional_ context managers. E.g. code like this: with performance_timer(): do_work()

def performance_timer(): if profiling: return accumulator return None

None becomes the trivial context manager and its __enter__ and __exit__ calls are skipped, along with their overhead.

How bad is the overhead of a trivial contextmanager (that is, one with empty bodies for both __enter__ and __exit__)? Valid question. Let's try:

#testnone.py import timeit class trivial: def __enter__(self): pass def __exit__(self, a, b, c): pass trivial = trivial() print("trivial") print(timeit.timeit("with trivial: pass", setup = "from __main__ import trivial")) print("none") print(timeit.timeit("with None: pass", setup = "from __main__ import trivial")) print("trivial + sum") print(timeit.timeit("with trivial: 1+1", setup = "from __main__ import trivial")) print("none + sum") print(timeit.timeit("with None: 1+1", setup = "from __main__ import trivial")) yields: trivial 0.786079668918435 none 0.07240212102423305 trivial + sum 0.9069962424632848 none + sum 0.11295328499933865 As you can see, the function call overhead is tremendously high. Two complete function calls with bells and whistles are omitted by using the None context manager. Ronald

-----Original Message-----

From: Ronald Oussoren [mailto:ronaldoussoren@mac.com]

Skipping the body makes it possible to introduce flow control in a context

manager, which could make it harder to read code. That is, currently the

body is either executed unless an exception is raised that raises an

exception, while with your proposal the body might be skipped entirely. I

haven't formed an opinion yet on whether or not that is a problem, but PEP

343 (introduction of the with statement) references

<http://blogs.msdn.com/b/oldnewthing/archive/2005/01/06/347666.aspx>

which claims that hiding flow control in macros can be bad.

Perhaps my example was not clear enough. The body will be skipped entirely if the __enter__() method raises an exception. An outer context manager can then suppress this exception. However, you cannot create a single context manager that does both. This, to me, was the most serious problem with the old nested() context manager: Nesting of two context managers could not be correctly done for arbitrary context managers I am not advocating that people do create such context manager, but pointing out that this omission means that you cannot always combine two context managers into one and preserve the semantics because it is possible to do something with two nested context managers that you cannot achieve with a single one. And this is my completeness argument. Imagine if there were code that could only be written using two nested functions, and that the nested function could not be folded into the outer one without sacrificing semantic correctness? I.e., you could do: foo = bar(spam(value)) but this: def ham(value): return bar(spam(value)) foo = ham(value) would not be correct for all possible "bar" and "spam"? As an example, here is what you can currently do in python (again, not recommending it but providing as an example) class ContextManagerExit(): pass @contextmanager def if_a(): try: yield except ContextManagerExit: pass @contextmanager def if_b(condition): if not condition: raise ContextManagerExit yield with if_a(), if_b(condition): execute_code() #this line is executed only if "condition" is True In current python, it is impossible to create a combined context manager that does this: if_c = nested(if_a(), if_b(condition)) with if_c: execute_code() #A single context manager cannot both raise an exception from __enter__() _and_ have it supressed. With my patch, "if_b()" is all that is needed, because ContextManagerExit is a special exception that is caught and suppressed by the interpreter. And so, it becomes possible to nest arbitrary context managers without sacrificing semantic correctness. Ø The overhead is fairly high, although not enough that I'd worry a lot in my code. But then again, I don't build gaming backends or cloud-scale software :-) Ø That said, using None as the "do nothing" context manager is problematic as this could hide problems in users code. I'm sure that I'm not the only person that sometimes forgets to actually return a value from a function. Using that (missing, and hence implicitly None) return value as a context manager currently causes an exception that clearly points out a bug in my code and could silently do the wrong thing with this proposal, e.g: Ø def get_lock(obj): Ø lck = LockManager.get_lock_for_object(obj) Ø # oops forgot to return lck Ø with get_lock(42): Ø ... Ø Using a different singleton instead of None would avoid this drawback. I think something like this is important for context managers that are used as "diagnostic" tools, e.g. timing related context managers that may be disabled in production without changing code. But I agree that None is problematic for the reason you demonstrate, and did consider that. I'm suggesting it here as a demonstration of the concept, and also to reduce the need for yet another built-in. Perhaps the ellipsis could be used, that's everyone's favourite singleton :) K

On 13 Aug 2013 08:27, "Kristján Valur Jónsson" <kristjan@ccpgames.com> wrote:

Perhaps my example was not clear enough. The body will be skipped entirely if the __enter__() method raises an exception. An outer context manager can then suppress this exception.

However, you cannot create a single context manager that does both. This, to me, was the most serious problem with the old nested() context manager: Nesting of two context managers could not be correctly done for arbitrary context managers

nested() was deprecated and removed because it didn't handle files (or any other CM that does resource acquisition in __init__) correctly. The fact you can't factor out arbitrary context managers had nothing to do with it.

I am not advocating that people do create such context manager, but pointing out that this omission means that you cannot always combine two context managers into one and preserve the semantics because it is possible to do something with two nested context managers that you cannot achieve with a single one. And this is my completeness argument.

At the moment, a CM cannot prevent execution of the body - it must be paired with an if statement or an inner call that may raise an exception, keeping the flow control at least somewhat visible at the point of execution. The following is also an illegal context manager: @contextmanager def bad_cm(broken=False): if not broken: yield It's illegal for exactly the same reason this is illegal (this is the expanded form of your nested CM example): @contextmanager def bad_cm2(broken=False): class Skip(Exception): pass try: if broken: raise Skip yield except Skip: pass

Imagine if there were code that could only be written using two nested functions, and that the nested function could not be folded into the outer one without sacrificing semantic correctness? I.e., you could do:

foo = bar(spam(value))

but this:

def ham(value): return bar(spam(value))

foo = ham(value)

would not be correct for all possible “bar” and “spam”?

But that's not what you're asking about. You're asking for the ability to collapse two independent try statements into one. There are already things you can't factor out as functions - that's why we have generators and context managers. It's also a fact that there are things you can't factor out as single context managers. This is why we have nested context managers and also still have explicit try/except/else/finally statements.

As an example, here is what you can currently do in python (again, not recommending it but providing as an example)

class ContextManagerExit(): pass

@contextmanager def if_a(): try: yield except ContextManagerExit: pass

@contextmanager def if_b(condition): if not condition: raise ContextManagerExit yield

with if_a(), if_b(condition): execute_code() #this line is executed only if “condition” is True

Expand it out to the underlying constructs and you will see this code is outright buggy, because the exception handler is too broad: try: if not condition: raise ContextManagerExit execute_code() # ContextManagerExit will be eaten here except ContextManagerExit: pass

In current python, it is impossible to create a combined context manager that does this:

if_c = nested(if_a(), if_b(condition))

with if_c: execute_code() #A single context manager cannot both raise an exception from __enter__() _and_ have it supressed.

This is a feature, not a bug: the with statement body will *always* execute, unless __enter__ raises an exception. Don't be misled by the ability to avoid repeating the with keyword when specifying multiple context managers in the same statement: semantically, that's equivalent to multiple nested with statements, so the outer one always executes, and the inner ones can only skip the body by raising an exception from __enter__.

With my patch, “if_b()” is all that is needed, because ContextManagerExit is a special exception that is caught and suppressed by the interpreter. And so, it becomes possible to nest arbitrary context managers without sacrificing semantic correctness.

I'd be open to adding the following context manager to contextlib: @contextmanager def skip(keep=False): class Skip(Exception): pass Skip.caught = None try: yield Skip except Skip as exc: if keep: Skip.caught = exc This would allow certain currently awkward constructs to be expressed more easily without needing to drop back to a try/except block (note that 3.4 already adds contextlib.ignored to easily suppress selected exceptions in a block of code). Creating a custom exception type each time helps avoid various problems with overly broad exception handlers.

But I agree that None is problematic for the reason you demonstrate, and did consider that. I’m suggesting it here as a demonstration of the concept, and also to reduce the need for yet another built-in. Perhaps the ellipsis could be used, that’s everyone’s favourite singleton :)

An empty contextlib.ExitStack() instance is already a perfectly serviceable "do nothing" context manager, we don't need (and won't get) another one. Cheers, Nick.

On 14 August 2013 17:05, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:

Phew, it is a bit awkward to discuss this in two separate places. Those interested are invited to take a peek at the issue as well: http://bugs.python.org/issue18677

I put my real reply on the tracker issue (since that's a better historical record), but the short version is that you've persuaded *me* to go back to wanting to fix this (since your approach is significantly less horrible than what I came up with back in 2009 for PEP 377), which means Guido is the one you need to convince to let you reopen the PEP (I added him to the nosy list on the issue). If Guido still doesn't like it, then it will stay broken :( Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2013-08-13, at 16:48 , Serhiy Storchaka wrote:

3. With ExitStack:

import contextlib with contextlib.ExitStack() as cm: if ...: file = open(...) cm.enter_context(file) else: file = None if ...: file2 = open(...) cm.enter_context(file2) else: file2 = None process(file, file2, …)

Please note that enter_context will return the result of object.__enter__, so: with contextlib.ExitStack() as cm: file = cm.enter_context(open(path1)) if condition1 else None file2 = cm.enter_context(open(path2)) if condition2 else None process(file, file2, ...) or if you really like conditional statements: with contextlib.ExitStack() as cm file = None file2 = None if condition1: file = cm.enter_context(open(path1)) if condition2: file2 = cm.enter_context(open(path2)) process(file, file2, ...)

Furthermore: On 2013-08-13, at 16:48 , Serhiy Storchaka wrote:

And when the with statement will support None as a "null-manager":

I think that is a terrible idea, and will be a source of woe and bugs more than a solution to anything. If you want an optional context manager, why not create an OptionalContextManager(object | None) which delegates to the underlying non-None CM? That's simple, that's clear, that's explicit, and that's not a hack.

13.08.13 18:29, Masklinn написав(ла):

If you want an optional context manager, why not create an OptionalContextManager(object | None) which delegates to the underlying non-None CM? That's simple, that's clear, that's explicit, and that's not a hack.

Because OptionalContextManager doesn't support an interface of the underlying object (e.g. doesn't have the read() method). And OptionalContextManager(None) is not None.

On 8/13/2013 10:48 AM, Serhiy Storchaka wrote:

...

None becomes the trivial context manager and its __enter__ and __exit__ calls are skipped, along with their overhead.

And when the with statement will support None as a "null-manager":

As I understand the proposal, it is not to actually make None a context manager with __enter__ and __exit__ methods (bad), but to make it a signal to not do the normal calls. Such a use of None as a 'skip' signal is similar to its use as function argument. While such use is fine when None is a default argument (the normal case), its use in filter for a required argument is problematical (see recent thread here). I think using None as a signal object here would engender the same confusion here. "How can None be a context manager if it does not have the ... methods?" Given the alternative of an empty stack and a possible addition to contextlib, I do not think we should go for this generalization of None usage. -- Terry Jan Reedy

13.08.13 22:23, Serhiy Storchaka написав(ла):

13.08.13 20:45, Terry Reedy написав(ла):

...

As I understand the proposal, it is not to actually make None a context manager with __enter__ and __exit__ methods (bad),

Why?

...

but to make it a signal to not do the normal calls.

I agree this is a bad idea.

Sorry, I misunderstood you. Skipping the body of the with statement is a bad idea (because it converts with into other control construction). But skipping calls of __enter__() and __exit__() methods looks just as an optimization an implementation detail (at least for None).