Add InvalidStateError to the standard exception hierarchy

I frequently find that I want to raise an exception when the target of a call is not in an appropriate state to perform the requested operation. Rather than choosing between `Exception` or defining a custom exception, it would be nice if there were a built-in `InvalidStateError` exception that my code could raise. In cases where I want to define a custom exception anyway, I think it would be nice if it could have a generic `InvalidStateError` exception class for it to inherit from. Of course, I would be open to other ideas for what the name of this exception should be. Other possibilities off the top of my head are `BadStateError` or `StateError`.

What's wrong with defining a custom exception? It's literally one line: `class InvalidStateError(Exception): pass`. Two lines if you want to put the `pass` on its own line. The built in exceptions are ones that are raised by the core interpreter. Even the stdlib doesn't get builtin exceptions, look at sqlite3.Error, for example. Defining a custom exception in the module alongside the function that raises it is both normal practice, and far more discoverable. Paul On Thu, 1 Sept 2022 at 22:42, Steve Jorgensen <stevecjor@gmail.com> wrote:

Steve Jorgensen writes:
Paul Moore wrote:
The built in exceptions are ones that are raised by the core interpreter.
I get 154 hits for PyExc_ValueError in cpython/Python. That's exactly the logic that Paul mentions. :-) I suspect that the reason that there is no PyExc_InvalidStateError is that the interpreter rarely can detect it. For example, in the following very contrived example def getcontents(file): # argument should be "filename" # Oops, forgot to open the file named by `file`. return file.read() you'll get an AttributeError. If you typecheck `file` at the top of the function, ValueError is appropriate. It's not hard to contrive a case where you can detect it: >>> with open(RandomExistingFile) as f: ... f.close() ... f.read() ... Traceback (most recent call last): File "<stdin>", line 3, in <module> ValueError: I/O operation on closed file. and Python chooses to shoehorn the round InvalidStateError peg into the square ValueError hole. This isn't entirely implausible in Python's implementation of object-oriented programming: the object `f` is an invalid value for the `self` argument of the read function. So, assuming you use this "it's in the interpreter" logic in advocating this new Error, I would say collect examples of other errors raised where you would prefer InvalidStateError. If they're consistently ValueError, I would say you have a case. Probably you need to derive InvalidStateError from ValueError for backward compatibility.

Java calls this IllegalStateException so I would suggest IllegalStateError. Looking at other exceptions that Java has, it would also be nice to have UnsupportedOperationError (I have used NotImplementedError for this but that suggests it might someday be implemented). On the other hand, as you noted, it's fairly trivial to implement new exceptions where needed. --- Bruce On Thu, Sep 1, 2022 at 2:41 PM Steve Jorgensen <stevecjor@gmail.com> wrote:

Le 01/09/2022 à 23:40, Steve Jorgensen a écrit :
https://docs.python.org/3/library/exceptions.html#ValueError states that ValueError is “Raised when an operation or function receives an argument that has the right type but an inappropriate value, and the situation is not described by a more precise exception such as |IndexError| <https://docs.python.org/3/library/exceptions.html#IndexError>.” How would a "state error" differ from this more precisely? What value would this new exception type add? Both ValueError and this proposed StateError are very generic.

On Thu, Sep 1, 2022 at 2:57 PM Jean Abou Samra <jean@abou-samra.fr> wrote:
Some examples: * a stream-like object that has been closed and you attempt to read from or write data to it. * a random number generator that has not been initialized with a seed (in the case where you have a constructor which doesn't also initialize it). * a hash function which you try to compute the digest without having added any data to it. In all these cases, the current call fails because a *previous* call was not done. The parameters to this invocation are not by themselves incorrect. --- Bruce

On Thu, Sep 01, 2022 at 03:11:29PM -0700, Bruce Leban wrote:
* a stream-like object that has been closed and you attempt to read from or write data to it.
That would be a ValueError:
Its arguable that this could (should?) have been some sort of IOError instead, but that ship has sailed.
* a random number generator that has not been initialized with a seed (in the case where you have a constructor which doesn't also initialize it).
That would be a bug in the constructor.
* a hash function which you try to compute the digest without having added any data to it.
That shouldn't be an error at all:
-- Steve

Jean Abou Samra wrote:
`ValueError` is about for when the value of an argument passed to the function is unacceptable. The exception that I propose would be for when there is nothing wrong with any argument value, but the object is not in the correct state for that method to be called. I should have provided an example. One example is when trying to call methods to interact with a remote system either before a connection has been made or after the connection has been terminated.

On Thu, Sep 01, 2022 at 09:40:05PM -0000, Steve Jorgensen wrote:
If the target of the call isn't in an appropriate state, isn't that a bug in the constructor that it allows you to construct objects that are in an invalid state? You should fix the object so that it is never in an invalid state rather than blaming the caller. I believe that the interpreter may sometimes raise RuntimeError for cases where objects are in a broken internal state, but I've never seen one in real life. Oh, no, I was thinking of SystemError. Nevertheless, for my own classes, I've sometimes used RuntimeError for broken internal state. I have always considered that a bug in my code.
What functionality do you expect this InvalidStateError superclass to provide that isn't already provided by Exception? -- Steve

You can't really do that with files that have been closed. Unless you disallow manual closing of files altogether. That being said, I'd suggest that people raise custom exception, so your callers can catch exactly what they want to handle. An generic exception like ValueError or the proposed InvalidStateError could be thrown by almost anything you call in your block, instead of just what you actually intend to catch.

Matthias Görgens wrote:
It depends on context whether it makes sense to define a custom exception, and I agree that I frequently should define a custom exception. In that case though, it would still be nice to have an appropriate generic exception for that to inherit from, just as I would inherit from `ValueError` for a special case of a value error.

On Fri, 2 Sept 2022 at 10:51, Steve Jorgensen <stevecjor@gmail.com> wrote:
Can you describe a situation where you'd want to catch InvalidStateError from two unrelated libraries at the same time? That's the main value of an exception hierarchy - you don't have to worry about exactly which part of a block of code raised LookupError, you just catch it and cope (eg if you delve deep into a complex structure that has both lists and dicts, you can "except LookupError: not_found()" without caring which one failed). TBH I'm actually a bit confused as to how you'd cope with InvalidStateError anyway. In what situations would you be able to recover from such an error (other than generic "log the error and continue", which works fine with "except Exception")? ChrisA

On Thu, Sep 1, 2022, 9:12 PM Chris Angelico
I'm +0 on this. Maybe +0.5. Making a custom error isn't hard, but inheriting from something more in the ballpark is nicer. Doing `except Exception ` feels very likely to be too broad. If my code hits ZeroDivisionError, I probably want to do something different than I do for an ended subprocess or a closed file. Maybe they all are just "log and continue," sure. But for many resources, it's possible to require them, or at least something functionally equivalent/usable. Maybe *that* subprocess ended, but I might be able to launch another with suitable behavior. Maybe *that* file handle is closed, but I can reopen the same file.

Matthias Görgens wrote:
I didn't say that I was talking about a file. In fact, today, I'm talking about an object that manages a subprocess. If a caller tries to call a method of the manager to interact with the subprocess when the subprocess has not yet been started or after it has been terminated, then I want to raise an appropriate exception. I am raising a custom exception, and it annoys me that it has to simply inherit from Exception when I think that an invalid state condition is a common enough kind of issue that it should have a standard exception class in the hierarchy.

On Fri, Sep 02, 2022 at 12:53:47AM -0000, Steve Jorgensen wrote:
Your API shouldn't allow the object to be created and returned until it has been started. Long ago I read a great blog post describing this idiom as an anti-pattern: obj = SomeObject(args) obj.make_it_work() obj.do_something() where constructing the object isn't sufficient to start using it, you have to call a second, magical method (sometimes called "init()"!) to make it actually usable. Unfortunately I lost the URL and have never been able to find it again, but it made a big impression on me. You will notice that we don't do that with Python builtins: # we don't do this! f = open('filename', 'w') f.open() # Actually open it for writing. f.write('stuff') # Or this: d = dict(key=value) d.init() # Can't use the dict until we make it work. d.update(something) Returning a dict in an initialiated but invalid state, which needs a second method call to make it valid, would be an obvious antipattern. Why would we do such a thing? Its a bad design! Just have the dict constuctor return the dict in the fully valid state. Which of course we do: for most (all?) objects in the stdlib, creating the object *fully* initialises it, so it is ready to use, immediately, without needing a special "make it work" method call. Of course there may be some exceptions to that rule, but they are uncommon.
Perhaps you should take your lead from file objects and inherit from ValueError. Trying to interact with a terminated process is analogous to writing to a closed file. What sort of features or APIs do you expect to inherit from this InvalidStateError class? -- Steve

On Fri, Sep 02, 2022 at 06:49:37AM +0800, Matthias Görgens wrote:
Files are not *constructed* in a closed state, or at least `open` doesn't return file objects that have to be closed. They are returned in an open state. Files can be closed afterwards. A closed file is not *invalid*, it is just closed. Typical file APIs go back to the earliest days of computing, long before object oriented programming was a thing, and we're stuck with them. If we were designing file I/O from scratch today, we might not have any such concept of open and closed files. But regardless, we don't need an InvalidStateError exception for closed files, because (1) they're not invalid, and (2) we already have an exception for that, ValueError. As I mentioned in a previous post, perhaps that should have been some sort of IOError, but we're stuck with ValueError. -- Steve

Steven D'Aprano writes:
A closed file is not *invalid*, it is just closed.
I think it's not very useful to focus on the individual words used here. I believe that Steve J's idea here is related to the fact that an object is a namespace and an environment. So the operation is not "read from specified file", it's "read as appropriate in the specified environment" (duck-typing), so the state of the environment of the call (which is the implicit argument "self") is invalid. I myself would argue it's not that the environment is in an invalid state for reading, but that reading is an invalid operation on a closed file. I note that XEmacs Lisp's docstring for the error function says: `invalid-operation' refers to all cases where code is trying to do something that's disallowed, or when an error occurred during an operation. (These two concepts are merged because there's no clear distinction between them.) I want to emphasize the remark in parentheses. It's frequently *very* difficult to make distinctions that will be clear to all users of the API, or even to all API users named "Steve". :-) In my personal opinion, the XEmacs Lisp hierarchy does a good job of specifying a hierarchy that lends itself to clear distinctions. YMMV on whether that hierarchy is a good one, but I think the point is important: there are no absolutes here, rather the quality of a distinction is relative to the rest of the hierarchy, it depends on how well it fits in. Bottom line: let's focus the argument on (1) where in the Python exception hierarchy the new builtin exception would go, and (2) per Paul Moore, if it could be used in the interpreter (I will add, or in a variety of stdlib modules), and where.

On Fri, 2 Sept 2022 at 23:08, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
And if not (2), then (3) whether there's actually value in being able to catch invalid state exceptions from different sources, because a core exception that isn't used by the stdlib could potentially have value for that, but otherwise it's better for each module to make its own. ChrisA

I am not sure this is necessary: RuntimeError already encapsulates this somewhat well. You'll find RuntimeError already widely used, especially in concurrency where it is easy to get into a situation where the state is invalid for a specific action.

What's wrong with defining a custom exception? It's literally one line: `class InvalidStateError(Exception): pass`. Two lines if you want to put the `pass` on its own line. The built in exceptions are ones that are raised by the core interpreter. Even the stdlib doesn't get builtin exceptions, look at sqlite3.Error, for example. Defining a custom exception in the module alongside the function that raises it is both normal practice, and far more discoverable. Paul On Thu, 1 Sept 2022 at 22:42, Steve Jorgensen <stevecjor@gmail.com> wrote:

Steve Jorgensen writes:
Paul Moore wrote:
The built in exceptions are ones that are raised by the core interpreter.
I get 154 hits for PyExc_ValueError in cpython/Python. That's exactly the logic that Paul mentions. :-) I suspect that the reason that there is no PyExc_InvalidStateError is that the interpreter rarely can detect it. For example, in the following very contrived example def getcontents(file): # argument should be "filename" # Oops, forgot to open the file named by `file`. return file.read() you'll get an AttributeError. If you typecheck `file` at the top of the function, ValueError is appropriate. It's not hard to contrive a case where you can detect it: >>> with open(RandomExistingFile) as f: ... f.close() ... f.read() ... Traceback (most recent call last): File "<stdin>", line 3, in <module> ValueError: I/O operation on closed file. and Python chooses to shoehorn the round InvalidStateError peg into the square ValueError hole. This isn't entirely implausible in Python's implementation of object-oriented programming: the object `f` is an invalid value for the `self` argument of the read function. So, assuming you use this "it's in the interpreter" logic in advocating this new Error, I would say collect examples of other errors raised where you would prefer InvalidStateError. If they're consistently ValueError, I would say you have a case. Probably you need to derive InvalidStateError from ValueError for backward compatibility.

Java calls this IllegalStateException so I would suggest IllegalStateError. Looking at other exceptions that Java has, it would also be nice to have UnsupportedOperationError (I have used NotImplementedError for this but that suggests it might someday be implemented). On the other hand, as you noted, it's fairly trivial to implement new exceptions where needed. --- Bruce On Thu, Sep 1, 2022 at 2:41 PM Steve Jorgensen <stevecjor@gmail.com> wrote:

Le 01/09/2022 à 23:40, Steve Jorgensen a écrit :
https://docs.python.org/3/library/exceptions.html#ValueError states that ValueError is “Raised when an operation or function receives an argument that has the right type but an inappropriate value, and the situation is not described by a more precise exception such as |IndexError| <https://docs.python.org/3/library/exceptions.html#IndexError>.” How would a "state error" differ from this more precisely? What value would this new exception type add? Both ValueError and this proposed StateError are very generic.

On Thu, Sep 1, 2022 at 2:57 PM Jean Abou Samra <jean@abou-samra.fr> wrote:
Some examples: * a stream-like object that has been closed and you attempt to read from or write data to it. * a random number generator that has not been initialized with a seed (in the case where you have a constructor which doesn't also initialize it). * a hash function which you try to compute the digest without having added any data to it. In all these cases, the current call fails because a *previous* call was not done. The parameters to this invocation are not by themselves incorrect. --- Bruce

On Thu, Sep 01, 2022 at 03:11:29PM -0700, Bruce Leban wrote:
* a stream-like object that has been closed and you attempt to read from or write data to it.
That would be a ValueError:
Its arguable that this could (should?) have been some sort of IOError instead, but that ship has sailed.
* a random number generator that has not been initialized with a seed (in the case where you have a constructor which doesn't also initialize it).
That would be a bug in the constructor.
* a hash function which you try to compute the digest without having added any data to it.
That shouldn't be an error at all:
-- Steve

Jean Abou Samra wrote:
`ValueError` is about for when the value of an argument passed to the function is unacceptable. The exception that I propose would be for when there is nothing wrong with any argument value, but the object is not in the correct state for that method to be called. I should have provided an example. One example is when trying to call methods to interact with a remote system either before a connection has been made or after the connection has been terminated.

On Thu, Sep 01, 2022 at 09:40:05PM -0000, Steve Jorgensen wrote:
If the target of the call isn't in an appropriate state, isn't that a bug in the constructor that it allows you to construct objects that are in an invalid state? You should fix the object so that it is never in an invalid state rather than blaming the caller. I believe that the interpreter may sometimes raise RuntimeError for cases where objects are in a broken internal state, but I've never seen one in real life. Oh, no, I was thinking of SystemError. Nevertheless, for my own classes, I've sometimes used RuntimeError for broken internal state. I have always considered that a bug in my code.
What functionality do you expect this InvalidStateError superclass to provide that isn't already provided by Exception? -- Steve

You can't really do that with files that have been closed. Unless you disallow manual closing of files altogether. That being said, I'd suggest that people raise custom exception, so your callers can catch exactly what they want to handle. An generic exception like ValueError or the proposed InvalidStateError could be thrown by almost anything you call in your block, instead of just what you actually intend to catch.

Matthias Görgens wrote:
It depends on context whether it makes sense to define a custom exception, and I agree that I frequently should define a custom exception. In that case though, it would still be nice to have an appropriate generic exception for that to inherit from, just as I would inherit from `ValueError` for a special case of a value error.

On Fri, 2 Sept 2022 at 10:51, Steve Jorgensen <stevecjor@gmail.com> wrote:
Can you describe a situation where you'd want to catch InvalidStateError from two unrelated libraries at the same time? That's the main value of an exception hierarchy - you don't have to worry about exactly which part of a block of code raised LookupError, you just catch it and cope (eg if you delve deep into a complex structure that has both lists and dicts, you can "except LookupError: not_found()" without caring which one failed). TBH I'm actually a bit confused as to how you'd cope with InvalidStateError anyway. In what situations would you be able to recover from such an error (other than generic "log the error and continue", which works fine with "except Exception")? ChrisA

On Thu, Sep 1, 2022, 9:12 PM Chris Angelico
I'm +0 on this. Maybe +0.5. Making a custom error isn't hard, but inheriting from something more in the ballpark is nicer. Doing `except Exception ` feels very likely to be too broad. If my code hits ZeroDivisionError, I probably want to do something different than I do for an ended subprocess or a closed file. Maybe they all are just "log and continue," sure. But for many resources, it's possible to require them, or at least something functionally equivalent/usable. Maybe *that* subprocess ended, but I might be able to launch another with suitable behavior. Maybe *that* file handle is closed, but I can reopen the same file.

Matthias Görgens wrote:
I didn't say that I was talking about a file. In fact, today, I'm talking about an object that manages a subprocess. If a caller tries to call a method of the manager to interact with the subprocess when the subprocess has not yet been started or after it has been terminated, then I want to raise an appropriate exception. I am raising a custom exception, and it annoys me that it has to simply inherit from Exception when I think that an invalid state condition is a common enough kind of issue that it should have a standard exception class in the hierarchy.

On Fri, Sep 02, 2022 at 12:53:47AM -0000, Steve Jorgensen wrote:
Your API shouldn't allow the object to be created and returned until it has been started. Long ago I read a great blog post describing this idiom as an anti-pattern: obj = SomeObject(args) obj.make_it_work() obj.do_something() where constructing the object isn't sufficient to start using it, you have to call a second, magical method (sometimes called "init()"!) to make it actually usable. Unfortunately I lost the URL and have never been able to find it again, but it made a big impression on me. You will notice that we don't do that with Python builtins: # we don't do this! f = open('filename', 'w') f.open() # Actually open it for writing. f.write('stuff') # Or this: d = dict(key=value) d.init() # Can't use the dict until we make it work. d.update(something) Returning a dict in an initialiated but invalid state, which needs a second method call to make it valid, would be an obvious antipattern. Why would we do such a thing? Its a bad design! Just have the dict constuctor return the dict in the fully valid state. Which of course we do: for most (all?) objects in the stdlib, creating the object *fully* initialises it, so it is ready to use, immediately, without needing a special "make it work" method call. Of course there may be some exceptions to that rule, but they are uncommon.
Perhaps you should take your lead from file objects and inherit from ValueError. Trying to interact with a terminated process is analogous to writing to a closed file. What sort of features or APIs do you expect to inherit from this InvalidStateError class? -- Steve

On Fri, Sep 02, 2022 at 06:49:37AM +0800, Matthias Görgens wrote:
Files are not *constructed* in a closed state, or at least `open` doesn't return file objects that have to be closed. They are returned in an open state. Files can be closed afterwards. A closed file is not *invalid*, it is just closed. Typical file APIs go back to the earliest days of computing, long before object oriented programming was a thing, and we're stuck with them. If we were designing file I/O from scratch today, we might not have any such concept of open and closed files. But regardless, we don't need an InvalidStateError exception for closed files, because (1) they're not invalid, and (2) we already have an exception for that, ValueError. As I mentioned in a previous post, perhaps that should have been some sort of IOError, but we're stuck with ValueError. -- Steve

Steven D'Aprano writes:
A closed file is not *invalid*, it is just closed.
I think it's not very useful to focus on the individual words used here. I believe that Steve J's idea here is related to the fact that an object is a namespace and an environment. So the operation is not "read from specified file", it's "read as appropriate in the specified environment" (duck-typing), so the state of the environment of the call (which is the implicit argument "self") is invalid. I myself would argue it's not that the environment is in an invalid state for reading, but that reading is an invalid operation on a closed file. I note that XEmacs Lisp's docstring for the error function says: `invalid-operation' refers to all cases where code is trying to do something that's disallowed, or when an error occurred during an operation. (These two concepts are merged because there's no clear distinction between them.) I want to emphasize the remark in parentheses. It's frequently *very* difficult to make distinctions that will be clear to all users of the API, or even to all API users named "Steve". :-) In my personal opinion, the XEmacs Lisp hierarchy does a good job of specifying a hierarchy that lends itself to clear distinctions. YMMV on whether that hierarchy is a good one, but I think the point is important: there are no absolutes here, rather the quality of a distinction is relative to the rest of the hierarchy, it depends on how well it fits in. Bottom line: let's focus the argument on (1) where in the Python exception hierarchy the new builtin exception would go, and (2) per Paul Moore, if it could be used in the interpreter (I will add, or in a variety of stdlib modules), and where.

On Fri, 2 Sept 2022 at 23:08, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
And if not (2), then (3) whether there's actually value in being able to catch invalid state exceptions from different sources, because a core exception that isn't used by the stdlib could potentially have value for that, but otherwise it's better for each module to make its own. ChrisA

I am not sure this is necessary: RuntimeError already encapsulates this somewhat well. You'll find RuntimeError already widely used, especially in concurrency where it is easy to get into a situation where the state is invalid for a specific action.
participants (10)
-
Bluenix
-
Bruce Leban
-
Chris Angelico
-
David Mertz, Ph.D.
-
Jean Abou Samra
-
Matthias Görgens
-
Paul Moore
-
Stephen J. Turnbull
-
Steve Jorgensen
-
Steven D'Aprano