[Tutor] How to parse large files

[Peter Otten]

Danny Yoo wrote:

>> AttributeError                            Traceback (most recent call
>> last) <ipython-input-3-f1c2a78eeb9a> in <module>()
>> ----> 1 with shelve.open("diz5") as db:
>>       2     with open("tmp1.txt") as instream:
>>       3         for line in instream:
>>       4             assert line.count("\t") == 1
>>       5             key, _tab, value = line.rstrip("\n").partition("\t")
>>
>> AttributeError: DbfilenameShelf instance has no attribute '__exit__'
> 
> 
> The error that you're seeing is on this line:
> 
>     with shelve.open("diz5") as db:
> 
> so we should focus our efforts to know why this line is failing.
> 
> 
> The with statement in Python has a requirement, that the resource
> supports context management.  Context managers have to have a few
> methods, according to:
> 
> https://docs.python.org/3/reference/compound_stmts.html#the-with-statement
> 
> https://docs.python.org/3/reference/datamodel.html#context-managers
> 
> 
> However, the error message reports that it can't find a method that
> it's looking for, "__exit__".  it looks like shelves don't have the
> methods "__enter__" or "__exit__", which context managers must have.
> 
> It looks like a deficiency in 'shelve', but one that we can work
> around without too much difficulty.  

You need 3.4 or higher to use with... Quoting
<https://docs.python.org/dev/library/shelve.html#shelve.Shelf>:

"""
class shelve.Shelf(dict, protocol=None, writeback=False, 
keyencoding='utf-8')
...
Changed in version 3.4: Added context manager support.
"""

> We can use the
> contextlib.closing() function to adapt a thing that knows how to
> close(), so that it works as a context manager.
> https://docs.python.org/3/library/contextlib.html#contextlib.closing
> 
> It should be a matter of saying:
> 
>     import contextlib
>     ...
>     with contextlib.closing(shelve.open("diz5")) as db: ...

Yes, either use contextlib.closing() or close the db manually:

db = shelve.open("diz5")
try:
    ... # use db
finally:
    db.close()