PEP 3151: Reworking the OS and IO exception hierarchy
Hello,
I would like to propose the following PEP for feedback and review.
Permanent link to up-to-date version with proper HTML formatting:
http://www.python.org/dev/peps/pep-3151/
Thank you,
Antoine.
PEP: 3151
Title: Reworking the OS and IO exception hierarchy
Version: $Revision: 83042 $
Last-Modified: $Date: 2010-07-21 21:16:49 +0200 (mer. 21 juil. 2010) $
Author: Antoine Pitrou
I read this while you were working on it in the sandbox - +1 in
principle, but the devil is obviously going to be in the details.
On Thu, Jul 22, 2010 at 5:34 AM, Antoine Pitrou
Step 1: coalesce exception types ================================
The first step of the resolution is to coalesce existing exception types. The extent of this step is not yet fully determined. A number of possible changes are listed hereafter:
* alias both socket.error and select.error to IOError * alias mmap.error to OSError * alias IOError to OSError * alias WindowsError to OSError
Each of these changes doesn't preserve exact compatibility, but it does preserve *useful compatibility* (see "compatibility" section above).
Not only does this first step present the user a simpler landscape, but it also allows for a better and more complete resolution of step 2 (see "Prerequisite" below).
Another idea along these lines would be to coalesce the builtin exceptions at the EnvironmentError level. That is, the top of the revised hierarchy would look like: +-- IOError +-- io.BlockingIOError +-- io.UnsupportedOperation (also inherits from ValueError) IOError would be aliased as EnvironmentError, OSError, WindowsError, socket.error, mmap.error and select.error Coalescing WindowsError like that would mean the "winerr" attribute would be present on all platforms, just set to "None" if the platform isn't Windows. (errno, filename and strerror can all already be None, as will often be the case when IOError is raised directly by Python code). select.error (now just an alias for IOError) would also grow the common IOError attributes. I'm suggesting IOError as the name based on your survey of what standard libraries currently raise (i.e. the vast majority of them use IOError rather than one of the other names). EnvironmentError would probably be more accurate, but IOError is more common and easier to type (and from the interpreter's point of view, any manipulation of the underlying OS can be viewed as a form of I/O, even if it involves accessing the process table or the environment variables or the registry rather than the filesystem or network). Also, there should be a helper function (probably in the os module) that given an errno value will create the appropriate IOError subclass. Regards, Nick. P.S. I want to let the idea kick around in my brain for a while before offering suggestions for possible useful IOError subclasses. Note that we don't need to create subclasses for *everything* - errors without a specific subclass can fall back to the basic IOError. Still, I expect many bikesheds will be painted a wide variety of colours before this discussion is done ;) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 22 Jul 2010 07:20:36 +1000
Nick Coghlan
Another idea along these lines would be to coalesce the builtin exceptions at the EnvironmentError level.
Agreed, I will add it to the PEP and process the rest of your input.
Still, I expect many bikesheds will be painted a wide variety of colours before this discussion is done ;)
Yes, this PEP offers a lot of opportunities for discussion :) Thanks, Antoine.
+1 on the general idea, always seemed awkward to me that these operations all raise the same exception. I didn't even know about the errno comparison method, though I've never looked for it. Point is that it is cryptic and as such not very pythonic.
As a user you are talking about in the PEP, i'm quite satisfied with the current implementation. The only change i see worth considering is concatenating IOError and OSError, possibly with EnvironmentError. It's worth giving EnvironmentError a better name too. I strongly oppose introducing 'fine-grained' exceptions. As an exception user, i care about: 1) brief, easy to remember exception hierarchy with clearly distinguished domains 2) intuitive, natural-looking handling 3) simple way to know which exceptions are thrown by a code IMO, the current implementation covers 2) excellently, 1) and 3) only have IOError/OSError ambiguities. 1)) OSError and IOError do intersect in the OS's PoV. The only reason why they're separate is because they 'feel' different. Historically, I/O and OSes come different ways: OSes are almost purely software and I/O technologies are primarily hardware-based. So, 'OSError' looks like something that is meaningful only to software logic and 'IOError' - like something that has a physical incarnation. Both OSError can be thought as part of IOError and vice versa so neither is likely to meet consensus to be made a subclass of the other. So we either 1) declare the above 'feelings' retrograde and fuse the types. In this case, EnvironmentError will become redundant so we'll have to fuse it in too; 2) just use EnvironmentError for all ambiguous cases and give it a better name. The current one is just wa-a-a-y t-o-o-o lo-o-o-ong for ubiquitous usage. 2)) The 'neat' handling except OSError,e: if e.errno==EEXIST: act() else: raise looks the most natural solution to me as all OSErrors are perceived as errors of the same type (errors caused by external factors and received as error codes from the OS standard API). Adding errno-specific subexceptions 1) makes some errnos privileged at the expense of others 2) introduces a list that isn't bound to any objective characteristic and is just an arbitrary "favorites" list 3) adds types that characterize single errors rather than error classes which is an unnecessary level of complexity. 3)) Builtin I/O operations throw IOError, OS function wrappers throw OSError, package functions throw package-specific ones - quite obvious where to expect which. There's EnvironmentError for any ambiguities, the only reason against it is the long and unusual name.
On Sat, Jul 24, 2010 at 5:16 PM, Ivan Pozdeev
Adding errno-specific subexceptions 1) makes some errnos privileged at the expense of others
Why is that a problem? Some errnos *are* more important than others -
they're the ones the regularly appear on the right hand side of "errno
==
2) introduces a list that isn't bound to any objective characteristic and is just an arbitrary "favorites" list
Why would you consider new classes that would be based on a survey of the errnos that developers actually check for in published code to be "arbitrary"?
3) adds types that characterize single errors rather than error classes which is an unnecessary level of complexity.
Any new IOError subclasses would likely still characterise classes of errors rather than single errno values. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 24 Jul 2010 11:16:45 +0400
Ivan Pozdeev
As a user you are talking about in the PEP, i'm quite satisfied with the current implementation.
As a user, I'm also quite satisfied with the current implementation, but think the proposal would be a major improvement in all areas.
I strongly oppose introducing 'fine-grained' exceptions.
I think we see this in two different ways. The current version of the PEP seems to be somewhere between the two views. But in particular:
Adding errno-specific subexceptions
I didn't see the PEP as calling for that. I saw it as dividing up the new combined IOError/OSError/EnvironmentError into finer-grained groups that make logical sense together. Yes, that division would depend on errno, and some of the groups on some platforms may well have only one errno - indeed, the first pass had a lot of those - but that's an implementation detail.
1) makes some errnos privileged at the expense of others
Well, some are privileged, in that they now belong to a finer grouping. I don't see how it's at the "expense" of the others: you can still catch the upper-level one and sort on errno, just like you do now.
2) introduces a list that isn't bound to any objective characteristic and is just an arbitrary "favorites" list
That I would object to. I expect the PEP to include objective rules for each subgroup that can be used to determine if an errno belongs in that subgroup.
3) adds types that characterize single errors rather than error classes which is an unnecessary level of complexity.
That actually sounds like the way most packages do things.
3)) Builtin I/O operations throw IOError, OS function wrappers throw OSError, package functions throw package-specific ones - quite obvious where to expect which.
If only practice matched theory. Package functions throw
package-specific exceptions, but they also call functions that can
throw OS or IO errors. So in practice, I find I often have to deal
with those as well as the package-specific ones. I can't even say the
package authors are wrong not to catch and map those errors into
package-specific errors.
Здравствуйте, Nick. Вы писали 24 июля 2010 г., 14:12:31:
Why is that a problem? Some errnos *are* more important than others - they're the ones the regularly appear on the right hand side of "errno ==
" checks.
Why would you consider new classes that would be based on a survey of the errnos that developers actually check for in published code to be "arbitrary"?
Since the list would be a sole opinion of some people who take part in the survey, you'll be constantly faced with demands of other people who want to have "shortcuts" for something else too. And you won't be able to explain why your choice is more preferable than theirs.
Any new IOError subclasses would likely still characterise classes of errors rather than single errno values.
The ones i see in the PEP correspond to either one or a few errnos. If the problem is you don't like the 'cryptic' errno mnemonics, it's a reason to change them instead. Current ones are just the standard POSIX names the errors are long and widely known under. -- Regards, Ivan mailto:vano@mail.mipt.ru
On Wed, Jul 21, 2010 at 12:34 PM, Antoine Pitrou
Hello,
I would like to propose the following PEP for feedback and review. Permanent link to up-to-date version with proper HTML formatting: http://www.python.org/dev/peps/pep-3151/
Thank you,
Antoine.
... ...
New exception classes ---------------------
The following tentative list of subclasses, along with a description and the list of errnos mapped to them, is submitted to discussion:
* ``FileAlreadyExists``: trying to create a file or directory which already exists (EEXIST)
* ``FileNotFound``: for all circumstances where a file and directory is requested but doesn't exist (ENOENT)
* ``IsADirectory``: file-level operation (open(), os.remove()...) requested on a directory (EISDIR)
* ``NotADirectory``: directory-level operation requested on something else (ENOTDIR)
* ``PermissionDenied``: trying to run an operation without the adequate access rights - for example filesystem permissions (EACCESS, optionally EPERM)
* ``BlockingIOError``: an operation would block on an object (e.g. socket) set for non-blocking operation (EAGAIN, EALREADY, EWOULDBLOCK, EINPROGRESS); this is the existing ``io.BlockingIOError`` with an extended role
* ``BadFileDescriptor``: operation on an invalid file descriptor (EBADF); the default error message could point out that most causes are that an existing file descriptor has been closed
* ``ConnectionAborted``: connection attempt aborted by peer (ECONNABORTED)
* ``ConnectionRefused``: connection reset by peer (ECONNREFUSED)
* ``ConnectionReset``: connection reset by peer (ECONNRESET)
* ``TimeoutError``: connection timed out (ECONNTIMEOUT); this could be re-cast as a generic timeout exception, useful for other types of timeout (for example in Lock.acquire())
This list assumes step 1 is accepted in full; the exception classes described above would all derive from the now unified exception type OSError. It will need reworking if a partial version of step 1 is accepted instead (again, see appendix A for the current distribution of errnos and exception types).
Exception attributes --------------------
In order to preserve *useful compatibility*, these subclasses should still set adequate values for the various exception attributes defined on the superclass (for example ``errno``, ``filename``, and optionally ``winerror``).
Implementation --------------
Since it is proposed that the subclasses are raised based purely on the value of ``errno``, little or no changes should be required in extension modules (either standard or third-party). As long as they use the ``PyErr_SetFromErrno()`` family of functions (or the ``PyErr_SetFromWindowsErr()`` family of functions under Windows), they should automatically benefit from the new, finer-grained exception classes.
Library modules written in Python, though, will have to be adapted where they currently use the following idiom (seen in ``Lib/tempfile.py``)::
raise IOError(_errno.EEXIST, "No usable temporary file name found")
Fortunately, such Python code is quite rare since raising OSError or IOError with an errno value normally happens when interfacing with system calls, which is usually done in C extensions.
If there is popular demand, the subroutine choosing an exception type based on the errno value could be exposed for use in pure Python.
Possible objections ===================
Namespace pollution -------------------
Making the exception hierarchy finer-grained makes the root (or builtins) namespace larger. This is to be moderated, however, as:
* only a handful of additional classes are proposed;
* while standard exception types live in the root namespace, they are visually distinguished by the fact that they use the CamelCase convention, while almost all other builtins use lowercase naming (except True, False, None, Ellipsis and NotImplemented)
An alternative would be to provide a separate module containing the finer-grained exceptions, but that would defeat the purpose of encouraging careful code over careless code, since the user would first have to import the new module instead of using names already accessible.
+1 in on this whole PEP! The EnvrionmentError hierarchy and common errno test code has bothered me for a while. While I think the namespace pollution concern is valid I would suggest adding "Error" to the end of all of the names (your initial proposal only says "Error" on the end of one of them) as that is consistent with the bulk of the existing standard exceptions and warnings. They are unlikely to conflict with anything other than exceptions people have already defined themselves in any existing code (which could likely be refactored out after we officially define these).
Earlier discussion ==================
While this is the first time such as formal proposal is made, the idea has received informal support in the past [1]_; both the introduction of finer-grained exception classes and the coalescing of OSError and IOError.
The removal of WindowsError alone has been discussed and rejected as part of another PEP [2]_, but there seemed to be a consensus that the distinction with OSError wasn't meaningful. This supports at least its aliasing with OSError.
Moratorium ==========
The moratorium in effect on language builtins means this PEP has little chance to be accepted for Python 3.2.
Possible alternative ====================
Pattern matching ----------------
Another possibility would be to introduce an advanced pattern matching syntax when catching exceptions. For example::
try: os.remove(filename) except OSError as e if e.errno == errno.ENOENT: pass
Several problems with this proposal:
* it introduces new syntax, which is perceived by the author to be a heavier change compared to reworking the exception hierarchy * it doesn't decrease typing effort significantly * it doesn't relieve the programmer from the burden of having to remember errno mnemonics
ugh. no. :) That only works well for single exceptions and encourages less explicit exception types. Exceptions are a class hierarchy, we should encourage its use rather than encouraging magic type specific attributes with conditionals. -gps
2010/7/24 Ivan Pozdeev
Why would you consider new classes that would be based on a survey of the errnos that developers actually check for in published code to be "arbitrary"?
Since the list would be a sole opinion of some people who take part in the survey, you'll be constantly faced with demands of other people who want to have "shortcuts" for something else too.
I think you misunderstood the survey methodology. It was not a survey of developers, instead large bodies of code were examined. There is nothing arbitrary or subjective in this approach. FWIW, am +1 on the PEP.
Mike Meyer wrote:
I can't even say the package authors are wrong not to catch and map those errors into package-specific errors.
I'd say they're not wrong at all. The exception hierarchy should be based on the semantics of the exceptions, not which package they happen to originate from or pass through. -- Greg
2010/7/25 Ivan Pozdeev
Здравствуйте, Nick.
Вы писали 24 июля 2010 г., 14:12:31:
Why is that a problem? Some errnos *are* more important than others - they're the ones the regularly appear on the right hand side of "errno ==
" checks. Why would you consider new classes that would be based on a survey of the errnos that developers actually check for in published code to be "arbitrary"?
Since the list would be a sole opinion of some people who take part in the survey, you'll be constantly faced with demands of other people who want to have "shortcuts" for something else too. And you won't be able to explain why your choice is more preferable than theirs.
As Alexander pointed out, the word survey has multiple meanings. One of those is the subjective approach you're objecting to (ask a bunch of people what they think), another is the more objective approach actually documented in the PEP (go and look at what is out there, as in the sense of "land survey"). Think "code survey" rather than "developer survey". (A scripted tool to gather statistics on exception handling in this space from Google code search results and direct scans of local Python code bases would actually be helpful, even if it wasn't 100% accurate) There is still a subjective step in whittling the code survey results down into a revised class heirarchy, but that's: - why it's a separate step in the PEP, independent of the consolidation step - why the PEP doesn't include a concrete proposal as yet - one of the main goals of discussion of the PEP here and across the wider Python community Language design is inherently a matter of judgment. Based on the way it has played out in practice (frequently requiring explicit errno checks and catching of multiple exception types in order to write correct code), we now think the previous judgment in relation to the EnvironmentError exception hierarchy is demonstrably flawed. That doesn't mean we throw our hands up in the air and give up - it means we knuckle down and try to come up with something better, based on what we can learn from what has gone before. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 Jul, 2010, at 23:47, Alexander Belopolsky wrote:
2010/7/24 Ivan Pozdeev
: .. Why would you consider new classes that would be based on a survey of the errnos that developers actually check for in published code to be "arbitrary"?
Since the list would be a sole opinion of some people who take part in the survey, you'll be constantly faced with demands of other people who want to have "shortcuts" for something else too.
I think you misunderstood the survey methodology. It was not a survey of developers, instead large bodies of code were examined. There is nothing arbitrary or subjective in this approach.
FWIW, am +1 on the PEP.
Same here, I'm +1 as well. The PEP is clear and solves a definite problem with a well though-out methodology. Ronald
Hello,
On Sat, 24 Jul 2010 14:31:42 -0700
"Gregory P. Smith"
The EnvrionmentError hierarchy and common errno test code has bothered me for a while. While I think the namespace pollution concern is valid I would suggest adding "Error" to the end of all of the names (your initial proposal only says "Error" on the end of one of them) as that is consistent with the bulk of the existing standard exceptions and warnings. They are unlikely to conflict with anything other than exceptions people have already defined themselves in any existing code (which could likely be refactored out after we officially define these).
The reason I haven't added "Error" to them is that the names are already quite long, and it's quite obvious that they refer to errors. I'm obviously not religious about it, though :) Regards Antoine.
Antoine Pitrou wrote:
Hello,
On Sat, 24 Jul 2010 14:31:42 -0700 "Gregory P. Smith"
wrote: The EnvrionmentError hierarchy and common errno test code has bothered me for a while. While I think the namespace pollution concern is valid I would suggest adding "Error" to the end of all of the names (your initial proposal only says "Error" on the end of one of them) as that is consistent with the bulk of the existing standard exceptions and warnings. They are unlikely to conflict with anything other than exceptions people have already defined themselves in any existing code (which could likely be refactored out after we officially define these).
The reason I haven't added "Error" to them is that the names are already quite long, and it's quite obvious that they refer to errors. I'm obviously not religious about it, though :)
Please keep the "Error" suffix on those exception class names. This is common practice and we wouldn't want to break with it just because the names get a little longer (we have editor type completion to deal with that ;-). Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 28 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 24 July 2010 22:31, Gregory P. Smith
On Wed, Jul 21, 2010 at 12:34 PM, Antoine Pitrou
wrote: Hello,
I would like to propose the following PEP for feedback and review. Permanent link to up-to-date version with proper HTML formatting: http://www.python.org/dev/peps/pep-3151/
Thank you,
Antoine.
[...]
+1 in on this whole PEP!
+1 from me too. Michael
The EnvrionmentError hierarchy and common errno test code has bothered me for a while. While I think the namespace pollution concern is valid I would suggest adding "Error" to the end of all of the names (your initial proposal only says "Error" on the end of one of them) as that is consistent with the bulk of the existing standard exceptions and warnings. They are unlikely to conflict with anything other than exceptions people have already defined themselves in any existing code (which could likely be refactored out after we officially define these).
Earlier discussion ==================
While this is the first time such as formal proposal is made, the idea has received informal support in the past [1]_; both the introduction of finer-grained exception classes and the coalescing of OSError and IOError.
The removal of WindowsError alone has been discussed and rejected as part of another PEP [2]_, but there seemed to be a consensus that the distinction with OSError wasn't meaningful. This supports at least its aliasing with OSError.
Moratorium ==========
The moratorium in effect on language builtins means this PEP has little chance to be accepted for Python 3.2.
Possible alternative ====================
Pattern matching ----------------
Another possibility would be to introduce an advanced pattern matching syntax when catching exceptions. For example::
try: os.remove(filename) except OSError as e if e.errno == errno.ENOENT: pass
Several problems with this proposal:
* it introduces new syntax, which is perceived by the author to be a heavier change compared to reworking the exception hierarchy * it doesn't decrease typing effort significantly * it doesn't relieve the programmer from the burden of having to remember errno mnemonics
ugh. no. :) That only works well for single exceptions and encourages less explicit exception types. Exceptions are a class hierarchy, we should encourage its use rather than encouraging magic type specific attributes with conditionals.
-gps
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
participants (11)
-
Alexander Belopolsky
-
Antoine Pitrou
-
Dag Odenhall
-
Greg Ewing
-
Gregory P. Smith
-
Ivan Pozdeev
-
M.-A. Lemburg
-
Michael Foord
-
Mike Meyer
-
Nick Coghlan
-
Ronald Oussoren