Is this PEP-able? "with" statement inside genexps / list comprehensions
Hello, Do you think it would be nice to allow with statements inside genexps or list comprehensions? The functions __enter__ and __exit__ would be automatically called as iterables are traversed. I am thinking of drafting a PEP about this. Examples: This g = (f.read() for fn in filenames with open(fn) as f) would be equivalent to the following use of a generator function: def __gen(): for fn in filenames: with open(fn) as f: yield f.read() g = __gen() This list = [f.read() for fn in filenames with open(fn) as f] would be equivalent to the following: list = [] for fn in filenames: with open(fn) as f: list.append(f.read()) -- Rudy
On Monday, July 30, 2018 at 3:55:25 PM UTC-4, Kyle Lahnakoski wrote:
Rudy,
I think your proposal may be very specific to iterable context managers;
I don't think his proposal is specific to iterable context managers. You can have a with clause that is used in a following for clause. in which case, make a method that makes that assumption:
On 30 July 2018 at 20:15, Rudy Matela <rudy@matela.com.br> wrote:
Hello,
Hi Rudy,
Yielding from a with block should be discouraged rather than given special syntax. There is essentially a contradiction between the meaning/purpose of yield (suspend indefinitely) and with (definitely call __exit__). If I partially iterate over g as in for line in g: break then at this point g is suspended and f.__exit__ has not been called, so the file is not closed. I may choose to iterate over g later or not, so it has to remain in suspension just in case. In practice if you do this in CPython then f.__exit__ will *probably* be invoked indirectly by g.__del__ if/when the gc collects g. This defeats the main point of using with-open though which is to avoid depending on the gc for closing files. -- Oscar
On 30/07/18 20:15, Rudy Matela wrote:
I was worried until I twigged that you meant adding a clause rather than a statement per se. How does this interact with the other clauses in a comprehension? I'm thinking of monstrosities like: g = (f.read() for fn in filenames if fn.endswith(".txt") with open(fn) as f for filenames in get_directory_listing(dir) if .....) -- Rhodri James *-* Kynesim Ltd
On 30/07/18 21:15, Rudy Matela wrote:
To sail around Oscar's concern, this should rather be def __gen(): for fn in filenames: with open(fn) as f: _v = f.read() yield _v But in this case I think it'd be clearer to make it an expression rather than a generator expression term: g = (f.read() with open(fn) as f for fn in filenames) where _ = f.read() with open(fn) as f is equivalent to with open(fn) as f: _ = f.read() Currently possibly (if silly) alternative: from functools import wraps class with_one_use: def __init__(self, context): self.__context = context def __getattr__(self, name): exc1 = False obj = self.__context.__enter__() try: temp = getattr(obj, name) except: exc1 = True if not self.__context.__exit__(*sys.exc_info()): raise else: if callable(temp): @wraps(temp) def f(*args, **kwargs): exc2 = False try: return temp(*args, **kwargs) except: exc2 = True if not self.__context.__exit__(*sys.exc_info()): raise finally: if not exc2: self.__context.__exit__(None, None, None) exc1 = True return f else: return temp finally: if not exc1: self.__context.__exit__(None, None, None) g = (with_one_use(open(fn)).read() for fn in filenames) -- Thomas
On Monday, July 30, 2018 at 3:55:25 PM UTC-4, Kyle Lahnakoski wrote:
Rudy,
I think your proposal may be very specific to iterable context managers;
I don't think his proposal is specific to iterable context managers. You can have a with clause that is used in a following for clause. in which case, make a method that makes that assumption:
On 30 July 2018 at 20:15, Rudy Matela <rudy@matela.com.br> wrote:
Hello,
Hi Rudy,
Yielding from a with block should be discouraged rather than given special syntax. There is essentially a contradiction between the meaning/purpose of yield (suspend indefinitely) and with (definitely call __exit__). If I partially iterate over g as in for line in g: break then at this point g is suspended and f.__exit__ has not been called, so the file is not closed. I may choose to iterate over g later or not, so it has to remain in suspension just in case. In practice if you do this in CPython then f.__exit__ will *probably* be invoked indirectly by g.__del__ if/when the gc collects g. This defeats the main point of using with-open though which is to avoid depending on the gc for closing files. -- Oscar
On 30/07/18 20:15, Rudy Matela wrote:
I was worried until I twigged that you meant adding a clause rather than a statement per se. How does this interact with the other clauses in a comprehension? I'm thinking of monstrosities like: g = (f.read() for fn in filenames if fn.endswith(".txt") with open(fn) as f for filenames in get_directory_listing(dir) if .....) -- Rhodri James *-* Kynesim Ltd
On 30/07/18 21:15, Rudy Matela wrote:
To sail around Oscar's concern, this should rather be def __gen(): for fn in filenames: with open(fn) as f: _v = f.read() yield _v But in this case I think it'd be clearer to make it an expression rather than a generator expression term: g = (f.read() with open(fn) as f for fn in filenames) where _ = f.read() with open(fn) as f is equivalent to with open(fn) as f: _ = f.read() Currently possibly (if silly) alternative: from functools import wraps class with_one_use: def __init__(self, context): self.__context = context def __getattr__(self, name): exc1 = False obj = self.__context.__enter__() try: temp = getattr(obj, name) except: exc1 = True if not self.__context.__exit__(*sys.exc_info()): raise else: if callable(temp): @wraps(temp) def f(*args, **kwargs): exc2 = False try: return temp(*args, **kwargs) except: exc2 = True if not self.__context.__exit__(*sys.exc_info()): raise finally: if not exc2: self.__context.__exit__(None, None, None) exc1 = True return f else: return temp finally: if not exc1: self.__context.__exit__(None, None, None) g = (with_one_use(open(fn)).read() for fn in filenames) -- Thomas
participants (7)
-
Kyle Lahnakoski -
Neil Girdhar -
Oscar Benjamin -
Rhodri James -
Rudy Matela -
Serhiy Storchaka -
Thomas Jollans