[Python-Dev] PEP 3151 from the BDFOP

[Barry Warsaw]

On Aug 24, 2011, at 01:57 AM, Antoine Pitrou wrote:

>> One guiding principle for me is that we should keep the abstraction as thin
>> as possible.  In particular, I'm concerned about mapping multiple errnos
>> into a single Error.  For example both EPIPE and ESHUTDOWN mapping to
>> BrokePipeError, or EACESS or EPERM to PermissionError.  I think we should
>> resist this, so that one errno maps to exactly one Error.  Where grouping
>> is desired, Python already has mechanisms to deal with that,
>> e.g. superclasses and multiple inheritance.  Therefore, I think it would be
>> better to have
>> 
>> + FileSystemPermissionError
>>   + AccessError (EACCES)
>>   + PermissionError (EPERM)
>
>I'm not sure that's a good idea:

Was it the specific grouping under FileSystemPermissionError that you're
objecting to, or the "keep the abstraction thin" principle?  Let's say we
threw out the idea of FSPE superclass, would you still want to collapse EACCES
and EPERM into PermissionError, or would separate exceptions for each be okay?
It's still pretty easy to catch both in one except clause, and it won't be too
annoying if it's rare.

>Yes, FileSystemError might be removed. I thought that it would be
>useful, in some library routines, to catch all filesystem-related
>errors indistinctly, but it's not a complete catchall actually (for
>example, AccessError is outside of the FileSystemError subtree).

Reading your IRC message (sorry, I was afk) it sounds like you think
FileSystemError can be removed.  I like keeping the hierarchy flat.

>> Similarly, I think it would be helpful to have the errno name (e.g. ENOENT)
>> in the error message string.  That way, it won't get in the way for most
>> code, but would be usefully printed out for uncaught exceptions.
>
>Agreed, but I think that's a feature request quite orthogonal from the
>PEP. The errno *number* is still printed as it was before:
>
>>>> open("foo")
>Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>FileNotFoundError: [Errno 2] No such file or directory: 'foo'
>
>(see e.g. http://bugs.python.org/issue12762)

True, but since you're going to be creating a bunch of new exception classes,
it should be relatively painless to give them a better str.  Thanks for
pointing out that bug; I agree with it.

>> A second guiding principle should be that careful code that works in Python
>> 3.2 must continue to work in Python 3.3 once PEP 3151 is accepted, but also
>> for Python 2 code ported straight to Python 3.3.
>
>I don't porting straight to 3.3 would make a difference, especially now
>that the idea of deprecating old exception names has been abandoned.

Cool.

>> Do be prepared for complaints about compatibility for careless code though
>> - there's a ton of that out in the wild, and people will always complain
>> with their "working" code breaks due to an upgrade.  Be *very* explicit
>> about this in the release notes and NEWS file, and put your asbestos
>> underoos on.
>
>I'll take care about that :)

:)

>> Have you considered the impact of this PEP on other Python implementations?
>> My hazy memory of Jython tells me that errnos don't really leak into Java
>> and thus Jython much, but what about PyPy and IronPython?  E.g. step 1's
>> deprecation strategy seems pretty CPython-centric.
>
>Alternative implementations already have to implement errno codes in a
>way or another if they want to have a chance of running existing code.
>So I don't think the PEP makes much of a difference for them.
>But their implementors can give their opinion on this.

Let's give them a little more time to chime in (hopefully, they are reading
this thread).  We needn't wait too long though.

>> As for step 1 (coalescing the errors).  This makes sense and I'm generally
>> agreeable, but I'm wondering whether it's best to re-use IOError for this
>> rather than introduce a new exception.  Not that I can think of a good name
>> for that.  I'm just not totally convinced that existing code when upgrading
>> to Python 3.3 won't introduce silent failures.  If an existing error is to
>> be re-used for this, I'm torn on whether IOError or OSError is a better
>> choice.  Popularity aside, OSError *feels* more right.
>
>I don't have any personal preference. Previous discussions seemed to
>indicate people preferred IOError. But changing the implementation to
>OSError would be simple. I agree OSError feels slightly more right, as
>in more generic.

Thanks for making this change in the PEP.

>> And that anything raising an exception (e.g. via PyErr_SetFromErrno) other
>> than the new ones will raise IOError?
>
>I'm not sure I understand the question precisely.

My question mostly was about raising OSError (as the current PEP states) with
an errno that does *not* map to one of the new exceptions.  In that case, I
don't think there's anything you could raise other than exactly OSError,
right?

>The errno mapping mechanism is implemented in IOError.__new__, but it gets
>called only if the class is exactly IOError, not a subclass:
>
>>>> IOError(errno.EPERM, "foo")
>PermissionError(1, 'foo')
>>>> class MyIOError(IOError): pass
>... 
>>>> MyIOError(errno.EPERM, "foo")
>MyIOError(1, 'foo')
>
>Using IOError.__new__ is the easiest way to ensure that all code
>raising IO errors takes advantage of the errno mapping. Otherwise you
>may get APIs raising the proper subclasses, and other APIs always
>raising base IOError (it doesn't happen often, but some Python
>library code raises an IOError with an explicit errno).
>
>> I also think that rather than transforming exception when raised from
>> Python, i.e. via __new__ hackery, perhaps it should be a ValueError in its
>> own right to raise IOError with an error represented by one of the
>> subclasses.
>
>That would make it harder to keep compatibility while adding new
>subclasses in future Python versions. Imagine a lot of people lobby for
>a dedicated EBADF subclass and obtain it, then IOError(EBADF, "some
>message") would suddenly raise a ValueError. Or do I misunderstand your
>proposal?

Somewhat.  FWIW, this is the part that I'm most uncomfortable with.

So, for raising OSError with an errno mapping to one of the subclasses, it
appears to break the "explicit is better than implicit" principle, and I think
it could lead to hard-to-debug or understand code.  You'll look at code that
raises OSError, but the exception that gets printed will be one of the
subclasses.  I'm afraid that if you don't know that this is happening, you're
going to think you're going crazy.

The other half is, let's say raising FileNotFoundError with the EEXIST errno.
I'm guessing that the __init__'s for the new OSError subclasses will not have
an `errno` attribute, so there's no way you can do that, but the PEP does not
discuss this.  It probably should.

>> I found more examples of ECHILD and ESRCH than the
>> former two.  How'd you like to add those two to make your BDFOP happy? :)
>
>Wow, I didn't know ESRCH.
>How would you call the respective exceptions?
>- ChildProcessError for ECHILD?

The Linux wait(2) manpage says:

       ECHILD (for wait()) The calling process does not have any unwaited-for
              children.

       ECHILD (for waitpid() or waitid()) The process specified by pid (wait‐
              pid())  or  idtype and id (waitid()) does not exist or is not a
              child of the calling process.  (This can happen for  one's  own
              child  if  the  action for SIGCHLD is set to SIG_IGN.  See also
              the Linux Notes section about threads.)

>- ProcessLookupError for ESRCH?

The Linux kill(2) manpage says:

       ESRCH  The pid or process group does not exist.  Note that an existing
              process might be a zombie, a process  which  already  committed
              termination, but has not yet been wait(2)ed for.

So in a sense, both are lookup errors, though I think it's going too far to
multiply inherit from LookupError.  Maybe ChildWaitError or ChildLookupError
for the former?  ProcessLookupError seems good to me.

>> What if all the errno symbolic names were mapped as attributes on IOError?
>> The only advantage of that would be to eliminate the need to import errno,
>> or for the ugly `e.errno == errno.ENOENT` stuff.  That would then be
>> rewritten as `e.errno == IOError.ENOENT`.  A mild savings to be sure, but
>> still.
>
>Hmm, I guess that's explorable as an orthogonal idea.

Cool.  How should we capture that?

>> How dumb/useless/unworkable would it be to add an __future__ to switch from
>> the old hierarchy to the new one?  Probably pretty. ;)
>
>Well, the hierarchy is built-in, since it's about standard exceptions.
>Also, you usually get the exception from some library API, so a
>__future__ in your own module would not achieve much.
>
>> What about an api that applications/libraries could use to add additional
>> exceptions based on other errnos they cared about?  This could be consulted in
>> PyErr_SetFromErrno() and raised instead of IOError.  Okay, yeah, that's
>> probably pretty dumb too.
>
>The problem is that behaviour becomes inconsistent accross libraries.
>I'm not sure that's very helpful to the user.

Yeah, on further reflection, let's forget those last two ideas. ;)

Okay, so here's what's still outstanding for me:

* Should we eliminate FileSystemError? (probably "yes")
* Should we ensure one errno == one exception?
  - i.e. separate EACCES and EPERM
  - i.e. separate EPIPE and ESHUTDOWN
* Should the str of the new exception subclasses be improved (e.g. to include
  the symbolic name instead of the errno first)?
* Is the OSError.__new__() hackery a good idea?
* Should the PEP define the signature of the new exceptions (e.g. to prohibit
  passing in an incorrect errno to an OSError subclass)?
* Can we add ECHILD and ESRCH, and if so, what names should we use?
* Where can we capture the idea of putting the symbolic names on OSError class
  attributes, or is it a dumb idea that should be ditched?
* How long should we wait for other Python implementations to chime in?

Cheers,
-Barry