PEP 3151 - Reworking the OS and IO exception hierarchy (again)
Hello, Georg (who manages the 3.2 release) suggested that I push PEP 3151 (*) in the hope that it can get in before the 3.2 feature freeze. Is there any consensus that PEP 3151 should get in 3.2, or even that it should get in at all? If there's some process issue that I'm missing, I'd be glad to know about it. (*) http://www.python.org/dev/peps/pep-3151/ Thanks Antoine.
Antoine Pitrou wrote:
In its current state, I don't think it should go in 3.2: There are a couple of cases where you remove more error classes which will cause existing code to catch more errors than what the code was designed for, e.g. catching socket.error will now also catch all IOErrors, code catching OSErrors will now also catch IOErrors. By making the scope of an except clause larger, you alter the semantics of programs using the more fine-grained hierarchies in ways which are both difficult to detect and harder to work around. Since user exceptions often inherit from the stdlib ones, the situation propogates to user exceptions as well, so even finding all the places in the code that must be reviewed gets difficult. OTOH, the removal of the more fine-grained classes doesn't gain us anything, e.g. it's pretty obvious that the os module raises OSErrors, the socket module socket.errors, the mmap module mmap.errors and so on. Please also have a look at http://www.python.org/dev/peps/pep-0348/ since some of the things you raise in your PEP were already discussed and decided a while back, including whether removal of exceptions and addition of new, more fine-grained ones I'm +1 on adding more fine-grained error classes, but don't see much use in removing existing fine-grained structures. BTW: Why are you discussing the PEP on python-ideas and not on python-dev where PEPs normally go for further discussion after they been assigned a PEP number (see http://www.python.org/dev/peps/pep-0001/) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Nov 11, 2010 at 7:09 PM, M.-A. Lemburg <mal@egenix.com> wrote:
This issue is addressed directly in the PEP by the "Compatibility Strategy" section. Code which checks errno will not suddenly start catching additional exceptions due to the merger of IOError and OSError. However, I agree with you that the PEP would benefit from keeping socket.error, select.error and mmap.error as separate IOError subclasses. Existing code may legitimately be catching those in try blocks that include other unrelated operations that may raise IOError. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Am 11.11.2010 13:03, schrieb Nick Coghlan:
Agreed. Let's keep the potential incompatibilities and surprises to a minimum. After all, behavior changes in this area will only show up in the exceptional case, which depending on the context will not be triggered frequently, and its test coverage is often low due to the relative difficulty of mocking these things. cheers, Georg
On Thu, 11 Nov 2010 22:03:36 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Most socket I/O errors use different errnos (socket-specific errnos such as ECONNRESET, etc.), so merging the exception classes wouldn't change anything for careful code which checks the errno. As for mmap.error, I think it's a design mistake in the first place, since mmap'ed files are not consistently different from normal files. As for select.error, the way the select module itself is generally inconsistent in which exceptions it raises also makes me think it is a design or implementation bug. The fact that it doesn't even inherit EnvinromentError reinforces that feeling. More fundamentally, the effort to create a detailed IO exception hierarchy is much less useful if we keep islands such as socket.error, mmap.error, since those errors won't benefit. The PEP only achieves maximum usefulnesss if it can leverage a single IO exception base class. That said, it is obvious that this PEP can raise compatibility concerns for uncaring code (code which is already poorly written in the first place). There's no way to reorganize the exception hierarchy without risking such issues. Actually, even without reorganizing the exception hierarchy, it often happens that we casually change the exception type raised by a given function, because we think the old behaviour was mistaken. Nobody apparently blames us for doing that (perhaps they are not reading python-checkins carefully enough :-)). So, as a data point, carefully merging the exception classes (all of OSError, IOError, EnvinronmentError, WindowsError, VMSError, socket.error, mmap.error and select.error) doesn't produce any error in the regression test suite. You can look at the SVN branch named "pep-3151" for that. Regards Antoine.
On Thu, Nov 11, 2010 at 11:08 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Perhaps the better approach would be to explicitly list out the exceptions being merged in the compatibility section and state which case applies: "Code that cares should already be checking the errno attribute": socket.error, OSError, WindowsError, VMSError "Already cannot be reliably handled distinct from IOError due to the way the affected modules are implemented": mmap.error, select.error Then we can state that any exception clause which changes scope due to the new merger into IOError was already broken and we're just slightly changing exactly how it is broken. That said, didn't we originally punt on this PEP for 3.2 due to the moratorium rather than due to any lack of time to get it done? Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le jeudi 11 novembre 2010 à 23:50 +1000, Nick Coghlan a écrit :
Ok, I will look at that.
That said, didn't we originally punt on this PEP for 3.2 due to the moratorium rather than due to any lack of time to get it done?
I myself thought that the moratorium would get in the way, but Georg asked me if it could be implemented in time for 3.2, which is why I'm bringing it here. Getting a pronouncement would be good anyway. Regards Antoine.
Antoine Pitrou wrote:
Why ? You can use multiple inheritance to plug the existing class structure into a new one.
There's nothing wrong with code such as this: try: ... except socket.error: print 'Network problem' except IOError: print 'File I/O problem' except OSError: print 'File system problem' You probably have a different use of try-except in mind. Those exception handling blocks are used at various levels in an application and the code run between try-except may well be dealing with sockets and file I/O, but require two different sets of problem resolution or reporting implementations. The PEP breaks such code. It also doesn't make a difference whether you additionally check errno or not in the except clauses, since Python will never branch into later except clauses if the class matches the first found exception class. In the above example, the except socket.error clause would receive all errors, but it's not prepared to deal with them, since the code was written with a different hierarchy in mind. And indeed, "careful" code using a more coarse class hierarchy as basis is more likely to break because of such changes than code which works based on classes of exceptions. Simply due to the fact that the errno value loses the additional context information of how it came to be under the PEP proposal. Since this is a known fact, PEP 348 used the 2.x-3.x transition to implement those changes. The 3.x-4.x transition would be the next chance to apply such structural changes.
Such changes need to be carefully documented in the NEWS file. They may well introduce DoS security problems in applications, e.g. by causing an application to crash with a traceback due to a try-except not triggering because of the change. So again, +1 on adding the new exceptions and integrating them with the existing IOError subclasses. -1 on removing those subclasses. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, 11 Nov 2010 17:01:00 +0100 Georg Brandl <g.brandl@gmx.net> wrote:
I can understand the desire to intercept network exceptions to print out messages. However, socket.error can also indicate a programming error: for example, trying to read from an unconnected socket. So, having a catch-all for socket.error can hide bugs in the code. The real way to discriminate errors is by using errno, not the module raising the error. That's why the PEP proposes errno-specific classes such as ConnectionError. Besides, useful feedback for the user requires printing the error message (which is filled out by the OS and doesn't depend on Python's exception hierarchy), and preferably which network service failed responding; just printing "network error" is completely unhelpful. Regards Antoine.
Georg Brandl wrote:
I'm not sure where Antoine got this impression. Stdlib modules traditionally always implemented a base error class for exceptions specific to that module and sometimes also added subclasses for additional breakdown. The OSError originated in the os module's os.error and was only raised for os module exceptions. With the introduction of OSError as a builtin, other modules started using this exception as well, which is a bit unfortunate. I still don't find its use arbitrary, though. IOError, OTOH, was used for exceptions raised by C lib I/O related functions and operations. This was never tied to a single module, but almost always used for typical I/O operations on files, sockets, devices, etc. As such, both depend on the runtime environment, hence the EnvironmentError base class. I also find assumptions such as "However, the user is interested in the nature of the error, not in which part of the interpreter it comes from (since the latter is obvious from reading the traceback message or application source code)." misleading. Of course, the user is interested in which part of the interpreter an exception originated and yes, she does want to take action on the exception based on the context of origin. That was the whole point why we added per module error base classes in the first place. Antoine can't really be serious when expecting the user to run the exception by the inspect module to find that context, e.g. to determine whether it was caused by a socket, an open file, or an attempt to ask the file system for a directory listing. All of these can e.g. raise an error with errno EACCESS. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Nov 12, 2010 at 1:03 AM, M.-A. Lemburg <mal@egenix.com> wrote:
No, that's the kind of code we're saying is already subtly broken, and we're just changing the way it is broken to be far more obvious (instead of *sometimes* printing the wrong message based on exactly what went wrong, it will instead consistently print the message associated with whichever of IOError and OSError is listed first). The standard library may have been consistent about IOError vs OSError at one point in time, but that time is in the past. Any code which doesn't catch (or ignore) them both and treat them as equivalent to each other is either already broken (if the try block contains code which may raise either exception) or won't be affected by the proposal in the PEP (if the try block is guaranteed to raise only the exception type already caught by the try block). Trying to restore a consistent distinction between them is even more of compatibility problem than simply giving up and merging them. I thought the PEP already made this point, but it may not be as clear if you don't already agree with the conclusion. There is a slightly better case for keeping socket.error around since we've been fairly consistent in having the networking code raise that rather than a bare IOError. However, even there the value of the distinction is blurring, since the other layers of the IO stack will still raise IOError. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I don't follow you. Example: If I run code that e.g. uses os.fork() and get an IOError, I don't want to catch that, since this is likely a programming error of some sort which I don't want to mask. However, if I get an OSError (the error raised by os.fork()), I can report this as a resource problem to higher level code which can then take appropriate action. Your arguing that because the stdlib has blurred the distinction between IOError and OSError (as a result of making OSError builtin rather than letting it live in the os module), it is OK to now blur the distinction between error handling code that was meant to run for issues that are related to the actually used functions and possibly causing the error reporting to malfunction. Antoine is arguing that simply by relying on the errno you can write "careful" code. That's not correct. If I write code that only checks the errno without knowing in which context this was set, I cannot always make assumptions on what that errno really means, e.g. EACCES, EBADF or EPERM are used by lots of C lib and OS functions, so there's no implicit hint to the subsystem where the error originated. If we go down that route, we might as well merge TypeError into ValueError as well, since the distinction between those has been blurred long before OSError was even introduced. But we don't do that. And the reason is that we know which functions or methods return which types of errors and use that knowledge to direct the flow of error handling into the right lanes. Besides, there are lots of functions in the os module that don't do any I/O in the standard sense - why should those raise an IOError ? Most of the functions deal with OS services, so the name OSError is a lot more appropriate. Note that the same kind of blurring is likely going to happen for the newly suggested additional exceptions. Sooner or later people are going to use ConnectionError for all kinds of connections, not only socket-related ones. Same for TimeoutError and PermissionError. So in the end, you have to go and check the errno code again, just to see whether the exception you're looking at really originated from a socket or file operation which set errno, provided .errno isn't set to the E-values mentioned in the PEP per default... or we just merge everything back again into IOError and leave it to the programmers to find some other way to signal where the exception originated. Not very helpful. Perhaps it's indeed better to clearly draw the line between IOError and OSError again, like you say, instead of trying to "fix" things by further blurring information. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, 12 Nov 2010 00:31:18 +0100 "M.-A. Lemburg" <mal@egenix.com> wrote:
If you're writing code such as (which I'm sure is frowned upon in the stdlib): try: ... 99 lines of code ... os.fork() ... 99 other lines of code ... except OSError: # oh, os.fork() must have failed you're already living dangerously, and even without this PEP. OSError is not "the error raised by os.fork()", it is the error raised by *many* stdlib functions including os.fork(). It could be raised by many of the other 198 lines of code in the try ... except block above. So, the goal of the exception category is to tell you what the problem is, not where it happened. How you know where it happened is by having fine-grained try ... except blocks. (and if you're a human, you can also read the traceback)
Checking exceptions without knowing in which context they are raised *is* doomed to failure. Your OSError could come from a lot of places already, so you can't do anything meaningful with it. On the other hand, if using a fine-grained try ... except block you know that e.g. EACCES (or PermissionError) was issued by os.rename(), you know precisely what it means.
Well, no, on the contrary. The distinction between TypeError and ValueError (and KeyError and SyntaxError and MemoryError... etc.) is *precisely* the kind of distinction which is advocated by this PEP, that is: a distinction between different kinds of problems. Not a distinction based on the location where a problem happens. (otherwise we would have DictError, StrError, BytesError, IntError... instead) And, indeed, the PEP helps improve the consistency of the exception hierarchy in that respect. (if you prefer the other kind of consistency, you can propose a PEP to introduce DictError, ForkError, RenameError, etc.)
Well, you were part of the earlier discussion where this was discussed. I'll let you browse through the archives.
I'm not sure what the problem with reusing exceptions is. We do it all the time. Regards Antoine.
Antoine Pitrou wrote:
In its current state, I don't think it should go in 3.2: There are a couple of cases where you remove more error classes which will cause existing code to catch more errors than what the code was designed for, e.g. catching socket.error will now also catch all IOErrors, code catching OSErrors will now also catch IOErrors. By making the scope of an except clause larger, you alter the semantics of programs using the more fine-grained hierarchies in ways which are both difficult to detect and harder to work around. Since user exceptions often inherit from the stdlib ones, the situation propogates to user exceptions as well, so even finding all the places in the code that must be reviewed gets difficult. OTOH, the removal of the more fine-grained classes doesn't gain us anything, e.g. it's pretty obvious that the os module raises OSErrors, the socket module socket.errors, the mmap module mmap.errors and so on. Please also have a look at http://www.python.org/dev/peps/pep-0348/ since some of the things you raise in your PEP were already discussed and decided a while back, including whether removal of exceptions and addition of new, more fine-grained ones I'm +1 on adding more fine-grained error classes, but don't see much use in removing existing fine-grained structures. BTW: Why are you discussing the PEP on python-ideas and not on python-dev where PEPs normally go for further discussion after they been assigned a PEP number (see http://www.python.org/dev/peps/pep-0001/) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Nov 11, 2010 at 7:09 PM, M.-A. Lemburg <mal@egenix.com> wrote:
This issue is addressed directly in the PEP by the "Compatibility Strategy" section. Code which checks errno will not suddenly start catching additional exceptions due to the merger of IOError and OSError. However, I agree with you that the PEP would benefit from keeping socket.error, select.error and mmap.error as separate IOError subclasses. Existing code may legitimately be catching those in try blocks that include other unrelated operations that may raise IOError. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Am 11.11.2010 13:03, schrieb Nick Coghlan:
Agreed. Let's keep the potential incompatibilities and surprises to a minimum. After all, behavior changes in this area will only show up in the exceptional case, which depending on the context will not be triggered frequently, and its test coverage is often low due to the relative difficulty of mocking these things. cheers, Georg
On Thu, 11 Nov 2010 22:03:36 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Most socket I/O errors use different errnos (socket-specific errnos such as ECONNRESET, etc.), so merging the exception classes wouldn't change anything for careful code which checks the errno. As for mmap.error, I think it's a design mistake in the first place, since mmap'ed files are not consistently different from normal files. As for select.error, the way the select module itself is generally inconsistent in which exceptions it raises also makes me think it is a design or implementation bug. The fact that it doesn't even inherit EnvinromentError reinforces that feeling. More fundamentally, the effort to create a detailed IO exception hierarchy is much less useful if we keep islands such as socket.error, mmap.error, since those errors won't benefit. The PEP only achieves maximum usefulnesss if it can leverage a single IO exception base class. That said, it is obvious that this PEP can raise compatibility concerns for uncaring code (code which is already poorly written in the first place). There's no way to reorganize the exception hierarchy without risking such issues. Actually, even without reorganizing the exception hierarchy, it often happens that we casually change the exception type raised by a given function, because we think the old behaviour was mistaken. Nobody apparently blames us for doing that (perhaps they are not reading python-checkins carefully enough :-)). So, as a data point, carefully merging the exception classes (all of OSError, IOError, EnvinronmentError, WindowsError, VMSError, socket.error, mmap.error and select.error) doesn't produce any error in the regression test suite. You can look at the SVN branch named "pep-3151" for that. Regards Antoine.
On Thu, Nov 11, 2010 at 11:08 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Perhaps the better approach would be to explicitly list out the exceptions being merged in the compatibility section and state which case applies: "Code that cares should already be checking the errno attribute": socket.error, OSError, WindowsError, VMSError "Already cannot be reliably handled distinct from IOError due to the way the affected modules are implemented": mmap.error, select.error Then we can state that any exception clause which changes scope due to the new merger into IOError was already broken and we're just slightly changing exactly how it is broken. That said, didn't we originally punt on this PEP for 3.2 due to the moratorium rather than due to any lack of time to get it done? Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le jeudi 11 novembre 2010 à 23:50 +1000, Nick Coghlan a écrit :
Ok, I will look at that.
That said, didn't we originally punt on this PEP for 3.2 due to the moratorium rather than due to any lack of time to get it done?
I myself thought that the moratorium would get in the way, but Georg asked me if it could be implemented in time for 3.2, which is why I'm bringing it here. Getting a pronouncement would be good anyway. Regards Antoine.
Antoine Pitrou wrote:
Why ? You can use multiple inheritance to plug the existing class structure into a new one.
There's nothing wrong with code such as this: try: ... except socket.error: print 'Network problem' except IOError: print 'File I/O problem' except OSError: print 'File system problem' You probably have a different use of try-except in mind. Those exception handling blocks are used at various levels in an application and the code run between try-except may well be dealing with sockets and file I/O, but require two different sets of problem resolution or reporting implementations. The PEP breaks such code. It also doesn't make a difference whether you additionally check errno or not in the except clauses, since Python will never branch into later except clauses if the class matches the first found exception class. In the above example, the except socket.error clause would receive all errors, but it's not prepared to deal with them, since the code was written with a different hierarchy in mind. And indeed, "careful" code using a more coarse class hierarchy as basis is more likely to break because of such changes than code which works based on classes of exceptions. Simply due to the fact that the errno value loses the additional context information of how it came to be under the PEP proposal. Since this is a known fact, PEP 348 used the 2.x-3.x transition to implement those changes. The 3.x-4.x transition would be the next chance to apply such structural changes.
Such changes need to be carefully documented in the NEWS file. They may well introduce DoS security problems in applications, e.g. by causing an application to crash with a traceback due to a try-except not triggering because of the change. So again, +1 on adding the new exceptions and integrating them with the existing IOError subclasses. -1 on removing those subclasses. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, 11 Nov 2010 17:01:00 +0100 Georg Brandl <g.brandl@gmx.net> wrote:
I can understand the desire to intercept network exceptions to print out messages. However, socket.error can also indicate a programming error: for example, trying to read from an unconnected socket. So, having a catch-all for socket.error can hide bugs in the code. The real way to discriminate errors is by using errno, not the module raising the error. That's why the PEP proposes errno-specific classes such as ConnectionError. Besides, useful feedback for the user requires printing the error message (which is filled out by the OS and doesn't depend on Python's exception hierarchy), and preferably which network service failed responding; just printing "network error" is completely unhelpful. Regards Antoine.
Georg Brandl wrote:
I'm not sure where Antoine got this impression. Stdlib modules traditionally always implemented a base error class for exceptions specific to that module and sometimes also added subclasses for additional breakdown. The OSError originated in the os module's os.error and was only raised for os module exceptions. With the introduction of OSError as a builtin, other modules started using this exception as well, which is a bit unfortunate. I still don't find its use arbitrary, though. IOError, OTOH, was used for exceptions raised by C lib I/O related functions and operations. This was never tied to a single module, but almost always used for typical I/O operations on files, sockets, devices, etc. As such, both depend on the runtime environment, hence the EnvironmentError base class. I also find assumptions such as "However, the user is interested in the nature of the error, not in which part of the interpreter it comes from (since the latter is obvious from reading the traceback message or application source code)." misleading. Of course, the user is interested in which part of the interpreter an exception originated and yes, she does want to take action on the exception based on the context of origin. That was the whole point why we added per module error base classes in the first place. Antoine can't really be serious when expecting the user to run the exception by the inspect module to find that context, e.g. to determine whether it was caused by a socket, an open file, or an attempt to ask the file system for a directory listing. All of these can e.g. raise an error with errno EACCESS. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, Nov 12, 2010 at 1:03 AM, M.-A. Lemburg <mal@egenix.com> wrote:
No, that's the kind of code we're saying is already subtly broken, and we're just changing the way it is broken to be far more obvious (instead of *sometimes* printing the wrong message based on exactly what went wrong, it will instead consistently print the message associated with whichever of IOError and OSError is listed first). The standard library may have been consistent about IOError vs OSError at one point in time, but that time is in the past. Any code which doesn't catch (or ignore) them both and treat them as equivalent to each other is either already broken (if the try block contains code which may raise either exception) or won't be affected by the proposal in the PEP (if the try block is guaranteed to raise only the exception type already caught by the try block). Trying to restore a consistent distinction between them is even more of compatibility problem than simply giving up and merging them. I thought the PEP already made this point, but it may not be as clear if you don't already agree with the conclusion. There is a slightly better case for keeping socket.error around since we've been fairly consistent in having the networking code raise that rather than a bare IOError. However, even there the value of the distinction is blurring, since the other layers of the IO stack will still raise IOError. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I don't follow you. Example: If I run code that e.g. uses os.fork() and get an IOError, I don't want to catch that, since this is likely a programming error of some sort which I don't want to mask. However, if I get an OSError (the error raised by os.fork()), I can report this as a resource problem to higher level code which can then take appropriate action. Your arguing that because the stdlib has blurred the distinction between IOError and OSError (as a result of making OSError builtin rather than letting it live in the os module), it is OK to now blur the distinction between error handling code that was meant to run for issues that are related to the actually used functions and possibly causing the error reporting to malfunction. Antoine is arguing that simply by relying on the errno you can write "careful" code. That's not correct. If I write code that only checks the errno without knowing in which context this was set, I cannot always make assumptions on what that errno really means, e.g. EACCES, EBADF or EPERM are used by lots of C lib and OS functions, so there's no implicit hint to the subsystem where the error originated. If we go down that route, we might as well merge TypeError into ValueError as well, since the distinction between those has been blurred long before OSError was even introduced. But we don't do that. And the reason is that we know which functions or methods return which types of errors and use that knowledge to direct the flow of error handling into the right lanes. Besides, there are lots of functions in the os module that don't do any I/O in the standard sense - why should those raise an IOError ? Most of the functions deal with OS services, so the name OSError is a lot more appropriate. Note that the same kind of blurring is likely going to happen for the newly suggested additional exceptions. Sooner or later people are going to use ConnectionError for all kinds of connections, not only socket-related ones. Same for TimeoutError and PermissionError. So in the end, you have to go and check the errno code again, just to see whether the exception you're looking at really originated from a socket or file operation which set errno, provided .errno isn't set to the E-values mentioned in the PEP per default... or we just merge everything back again into IOError and leave it to the programmers to find some other way to signal where the exception originated. Not very helpful. Perhaps it's indeed better to clearly draw the line between IOError and OSError again, like you say, instead of trying to "fix" things by further blurring information. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 11 2010)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Fri, 12 Nov 2010 00:31:18 +0100 "M.-A. Lemburg" <mal@egenix.com> wrote:
If you're writing code such as (which I'm sure is frowned upon in the stdlib): try: ... 99 lines of code ... os.fork() ... 99 other lines of code ... except OSError: # oh, os.fork() must have failed you're already living dangerously, and even without this PEP. OSError is not "the error raised by os.fork()", it is the error raised by *many* stdlib functions including os.fork(). It could be raised by many of the other 198 lines of code in the try ... except block above. So, the goal of the exception category is to tell you what the problem is, not where it happened. How you know where it happened is by having fine-grained try ... except blocks. (and if you're a human, you can also read the traceback)
Checking exceptions without knowing in which context they are raised *is* doomed to failure. Your OSError could come from a lot of places already, so you can't do anything meaningful with it. On the other hand, if using a fine-grained try ... except block you know that e.g. EACCES (or PermissionError) was issued by os.rename(), you know precisely what it means.
Well, no, on the contrary. The distinction between TypeError and ValueError (and KeyError and SyntaxError and MemoryError... etc.) is *precisely* the kind of distinction which is advocated by this PEP, that is: a distinction between different kinds of problems. Not a distinction based on the location where a problem happens. (otherwise we would have DictError, StrError, BytesError, IntError... instead) And, indeed, the PEP helps improve the consistency of the exception hierarchy in that respect. (if you prefer the other kind of consistency, you can propose a PEP to introduce DictError, ForkError, RenameError, etc.)
Well, you were part of the earlier discussion where this was discussed. I'll let you browse through the archives.
I'm not sure what the problem with reusing exceptions is. We do it all the time. Regards Antoine.
participants (4)
-
Antoine Pitrou -
Georg Brandl -
M.-A. Lemburg -
Nick Coghlan