[Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Ethan Furman]

On 07/09/2014 09:02 PM, Nick Coghlan wrote:
> On 9 Jul 2014 17:14, "Ethan Furman" wrote:
>>
>> I like the 'onerror' API better primarily because it gives a single
>> point to deal with the errors. [...]
>
> The "onerror" approach can also deal with readdir failing, which the
>  PEP currently glosses over.

Do we want this, though?  I can see an error handler for individual entries, but if one of the *dir commands fails that 
would seem to be fairly catastrophic.

> I'm somewhat inclined towards the current approach in the PEP, but I'd like to see an explanation of two aspects:
>
> 1. How a scandir variant with an 'onerror' option could be implemented given the version in the PEP

Here's a stab at it:

     def scandir_error(path, info=None, onerror=None):
         for entry in scandir(path):
             if info == 'type':
                 try:
                     entry.is_dir()
                 except OSError as exc:
                     if onerror is None:
                         raise
                     if not onerror(exc, entry):
                         continue
             elif info == 'lstat':
                 try:
                     entry.lstat()
                 except OSError as exc:
                     if onerror is None:
                         raise
                     if not onerror(exc, entry):
                         continue
             yield entry

Here it is again with an attempt to deal with opendir/readdir/closedir exceptions:

     def scandir_error(path, info=None, onerror=None):
         entries = scandir(path)
         try:
             entry = next(entries)
         except StopIteration:
             # pass it through
             raise
         except Exception as exc:
             if onerror is None:
                 raise
             if not onerror(exc, 'what else here?'):
                 # what do we do on False?
                 # what do we do on True?
         else:
             for entry in scandir(path):
                 if info == 'type':
                     try:
                         entry.is_dir()
                     except OSError as exc:
                         if onerror is None:
                             raise
                         if not onerror(exc, entry):
                             continue
                 elif info == 'lstat':
                     try:
                         entry.lstat()
                     except OSError as exc:
                         if onerror is None:
                             raise
                         if not onerror(exc, entry):
                             continue
                 yield entry


> 2. How the existing scandir module handles the 'onerror' parameter to its directory walking function

Here's the first third of it from the repo:

     def walk(top, topdown=True, onerror=None, followlinks=False):
         """Like os.walk(), but faster, as it uses scandir() internally."""
         # Determine which are files and which are directories
         dirs = []
         nondirs = []
         try:
             for entry in scandir(top):
                 if entry.is_dir():
                     dirs.append(entry)
                 else:
                     nondirs.append(entry)
         except OSError as error:
             if onerror is not None:
                 onerror(error)
             return
         ...

--
~Ethan~