Best use of "open" context manager

[dn]

On 6/07/24 22:49, Rob Cliffe via Python-list wrote:
> Consider this scenario (which I ran into in real life):
>      I want to open a text file and do a lot of processing on the lines 
> of that file.
>      If the file does not exist I want to take appropriate action, e.g. 
> print an error message and abort the program.
> I might write it like this:
> 
> try:
>      with open(FileName) as f:
>          for ln in f:
>              print("I do a lot of processing here")
>              # Many lines of code here .....
> except FileNotFoundError:
>      print(f"File {FileName} not found")
>      sys.exit()
> 
> but this violates the principle that a "try" suite should be kept small, 
> so that only targeted exceptions are trapped,

Yes!


> not to mention that having "try" and "except" far apart decreases 
> readability.

Uh-oh!

- and there's a bit of a hang-over for old-timers who had to take care 
of file-locking within the application - we try to minimise 'time' 
between opening a file and closing it (etc)!

As it seems the file is opened to read. Less relevant in this case, but 
habits and styles of coding matter...


> Or I might write it like this:
> 
> try:
>      f = open(FileName) as f:
>      FileLines = f.readlines()
> except FileNotFoundError:
>      print(f"File {FileName} not found")
>      sys.exit()
> # I forgot to put "f.close()" here -:)
> for ln in File Lines:
>          print("I do a lot of processing here")
>          # Many lines of code here .....
> 
> but this loses the benefits of using "open" as a context manager,
> and would also be unacceptable if the file was too large to read into 
> memory.

So, now there are two concerns:
1 FileNotFoundError, and
2 gradual processing to avoid memory-full

- added to remembering to close the file.


> Really I would like to write something like
> 
> try:
>      with open(FileName) as f:
> except FileNotFoundError:
>      print(f"File {FileName} not found")
>      sys.exit()
> else: # or "finally:"
>          for ln in f:
>              print("I do a lot of processing here")
>              # Many lines of code here .....
> 
> but this of course does not work because by the time we get to "for ln 
> in f:" the file has been closed so we get
> ValueError: I/O operation on closed file
> 
> I could modify the last attempt to open the file twice, which would 
> work, but seems like a kludge (subject to race condition, inefficient).
> 
> Is there a better / more Pythonic solution?

Idea 1: invert the exception handling and the context-manager by writing 
a custom context-manager class which handles FileNotFoundError 
internally. Thus, calling-code becomes:

with...
     for...
         processing

Idea 2: incorporate idea of encapsulating "processing" into a 
(well-named) function to shorten the number of lines-of-code inside the 
with-suite.

Idea 3: consider using a generator to 'produce' lines of data 
one-at-a-time. Remember that whilst context-managers and generators are 
distinct concepts within Python, they are quite similar in many ways. 
So, a custom generator could work like a context-manager or 'wrap' a 
context-manager per Idea 1.

Building a custom-class (Idea 1 or Idea 3) enables the components to be 
kept together, per the ideal. It keeps the try-except components close 
and easy to relate. It is Pythonic (in the OOP style).

-- 
Regards,
=dn