foo.setParseAction(lambda a, b, c: raise FuckPython(":("))
foo.setParseAction(lambda a, b, c: raise FuckPython(":(")) is invalid syntax, which makes pyparsing useless.
On 10/27/2019 10:35 AM, Soni L. wrote:
that's a waste of a perfectly good name in the namespace.
Names are cheap. If "wasting" a name is a concern, then perhaps Python isn't the language for you. It also lacks local scopes inside code blocks, for example, which also "waste" names, as compared to say C. Eric
On Sun, Oct 27, 2019 at 06:56:16PM -0300, Soni L. wrote:
that's why I use nested functions.
I'm pretty sure that the memory and time overhead of using nested functions is far greater than the saving in names. I'm not even sure what it is that you think you are saving. Why do you care if a function used ten local names or 15 local names? -- Steven
throw is an expression, not a statement, in C++. I see no reason raise couldn't be an expression in Python. It doesn't even need a special rule in the grammar: from __future__ import raise_function foo.setParseAction(lambda a, b, c: raise(MumbleMumble())) Looking up and calling the raise function would add overhead to every explicitly raised exception, but exceptions are so expensive anyway that I think it wouldn't be noticeable.
On Oct 27, 2019, at 15:07, Ben Rudiak-Gould <benrudiak@gmail.com> wrote:
That’s a pretty big breaking change. Every line of code in every library and app that raises would have to change to add parens. That was acceptable between 2.x and 3.x for print, but I don’t think it would be acceptable between 3.8 and 3.9, or even between 3.8 and 3.11 with a future schedule unless there was a really good reason for it. And there really isn’t. Anyone who wants this can do this today: def throw(e): raise e Install it into builtins or import it into all of your modules, and you’re done. Meanwhile, all of the library code you use that has raise statements continues to work unchanged. And meanwhile, if you want to change the language to turn raise into an expression, why not just turn it into an expression with the same syntax, just like with yield somewhere around 2.4? Then all existing code that raises exceptions continues to work; only code that relies on the grammar or AST breaks (and that’s explicitly allowed to break between versions). I’m pretty sure you wouldn’t need any change to the bytecodes and ceval loop, and only a small change to the codegen part of the compiler, but I can’t be sure without actually trying it. It might be ambiguous to humans, if not to the parser, to do a bare raise in some contexts. If so, that can be solved the exact same way as yield. There’s a bit of bikeshedding to be done (the grammar for where it can appear doesn’t have to be identical to yield_expression, so you can decide, e.g., to ban x = raise e as silly and force people to write x = (raise e), or decide that’s consenting-adults silliness and no reason to ban it), but I don’t think there’s anything difficult beyond getting everyone to agree on your bikeshed color.
Are you sure about that? Exceptions aren’t nearly as slow as many people seem to think; otherwise, Python couldn’t get away with using them all over the place internally. Try this: def throw(e): raise e builtins.throw = throw del throw def f(): try: throw(ValueError('spam')) except Exception as e: return e %timeit f() And then replace the throw call with a raise statement and try again. I get 1350ns vs. 548ns, and that’s even before subtracting out the time to look up and call f and to return e. I don’t know how much of this is builtin-lookup vs. function calling vs. needing more of the code and data caches vs. whatever, and the function calling part would get faster if it were a C rather than Python function, but it’s still not going to be unnoticeable. And if you’re thinking maybe this isn’t a good test because maybe CPython is able to optimize something because the exception is handled in the same scope that raises it, you could check that with dis, or just try this: def f(): raise ValueError('spam') def g(): throw(ValueError('spam')) def h(fun): try: fun() except Exception(e): pass Now it’s 1950ns vs. 734ns. A raise expression, on the other hand, would add no performance cost. At any rate, if you really want this, just define that throw function yourself—or one that takes a type and *args and **kw and constructs the exception, or whatever API you want—and use it in place of raise in your code. It’s as trivial a function as could be, and if you don’t raise exceptions in a bottleneck, it probably doesn’t matter that you’ve made them more than twice as slow to raise, and you won’t be breaking any builtin or library code that expects it to be faster because that code is still using raise statements.
On Mon, Oct 28, 2019 at 10:20 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
And meanwhile, if you want to change the language to turn raise into an expression, why not just turn it into an expression with the same syntax, just like with yield somewhere around 2.4? Then all existing code that raises exceptions continues to work; only code that relies on the grammar or AST breaks (and that’s explicitly allowed to break between versions).
When yield was turned into an expression, it was to allow it to have a value: def coro(): value = yield "Start" print("I got value:", value) yield "End" x = coro() print(next(x)) print(x.send("Hello")) But a raise expression doesn't really make sense in that way - it can't possibly have any value. It would be possible to use a raise expression in a lambda function or an if/else expression, but other than that, there wouldn't be much to gain. ChrisA
On Oct 27, 2019, at 16:37, Chris Angelico <rosuav@gmail.com> wrote:
Well, also comprehensions. (And, I suppose, or/and expressions abused as if/else shortcuts, but that’s even less useful than lambdas and comprehensions.) But I’m not arguing that this is useful for the same reason yield_expression was useful. I don’t think this is useful enough to change the language at all. It’s perfectly fine that in the rare occasions when you have a good reason to raise in a lambda or a comprehension, you have to define a trivial one-line throw function. The point of the paragraph you’re replying to is that if I’m wrong about that, and we really do need to change the language to allow raise in expressions, it should be done like yield, not like print.
On Mon, Oct 28, 2019 at 11:17 AM Andrew Barnert <abarnert@yahoo.com> wrote:
Fair point. Given the 'if', I agree with the conclusion. I'm of the opinion that you aren't wrong about that, though, and the fact that throw() isn't in everyone's toolkits already suggests that this really isn't a major problem to be solved. ChrisA
On Oct 27, 2019, at 17:27, Chris Angelico <rosuav@gmail.com> wrote:
Searching on “python raise expression”, the first hit is a StackOverflow question where the asker is looking for exactly this functionality, and, while the accepted answer is “just use a statement” (with some nice explanation), another upvoted answers offers (and explains) a throw function named “raiser”. There are lots of other SO hits, and hits elsewhere, and about half of them are for this issue and include some variant of this solution. I didn’t see a single compelling examples where it really seemed justified, but still, people are looking for it, and finding it. (And they’re finding it under a variety of different names, which makes it harder to mass search code to see how often it’s being used…) Grepping my pile of one-off scripts, I haven’t used my throw function in a few years, but I have used it at least a couple dozen times in the past. Many of them are abusing throw(StopIteration) to fake a “takewhile clause” in comprehensions (which was never a good idea, and doesn’t even work anymore since 3.7, but apparently I used to do it…), and all of the rest would be not just more readable, but shorter too, if I broke up the giant expression the throw was in. But I think the OP might prefer my old giant-expression code, even though I think it’s terrible, because the cleaner and shorter version would usually “waste” at least one local variable storing an intermediate. Anyway, I don’t know how much all of this matters. Whether people are using it frequently or not, and whether the vast majority of uses are bad or not, it’s as easy to find it today as it would be if we added it to the docs somewhere or even included it somewhere in the stdlib, and trivial to use once you find it, so i don’t think we need to bend over backward to make it easier.
On Oct 28, 2019, at 14:23, Serhiy Storchaka <storchaka@gmail.com> wrote:
Well, what I really (thought I) needed was a more concise way to do the equivalent of takewhile and friends, and I still think the ideal solution there is a general way to write concise lambdas or the equivalent using a great new syntax that nobody has been able to come up with yet in any of the times the idea has been discussed. :) Faking it by raising StopIteration in the middle of the value expression or an if clause seemed like a clever workaround, albeit only for takewhile. But it actually has a lot of problems; various things that shouldn’t change your semantics do if you’re using this hack. (For example, chain up a few throw-breaking genexprs, then change one of the intermediate ones to a listcomp so you can see it in the debugger.) So, I’d already abandoned it before the PEP that broke it. Would a break expression be able to give me the same benefits without the problems? Maybe. But I honestly haven’t missed the throw(StopIteration) hack enough to experiment with that idea when people suggested it.
On Sun, Oct 27, 2019 at 4:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
I agree, it's a bad idea. It'd have to be Python 4, like print_function was Python 3.
def throw(e): raise e
The problem with this is that it adds an extra line to the traceback, so you have to choose between the convenience of using it and the convenience of more readable tracebacks. Writing it in C would fix that, but it's pretty complicated (I think I could copy the logic from do_raise in ceval.c, but I'm not completely sure, and it's almost 100 lines long), and of course not portable. I wouldn't mind having a sys.throw or something that's guaranteed to do it correctly. Actually, it'd be nice to have some general way to write trivial wrappers in Python without uglifying tracebacks, but that's a different discussion.
On Oct 31, 2019, at 07:29, Ben Rudiak-Gould <benrudiak@gmail.com> wrote:
Is a lot simpler than that. But you shouldn’t have to do it nonetheless. You can also hack up a solution in pure Python by manipulating the traceback manually. The quick&dirty way to do this using the frame objects is nonportable, but it is doable portably, it just might be a lot more of a pain. In fact, I think you can borrow and simplify code from importlib to do it. When the import machinery was rewritten from mostly C to pure Python around 3.4, they had to come up with a workaround to make ImportError tracebacks still look like they used to instead of including an extra 6 functions in the middle that nobody cares about except people working on the import system or an import hook. IIRC, what they do is call a bracketing function on entering the import machinery and then another one on calling back out, and then the top level functions have code that edits tracebacks that pass through by removing entries between those two brackets.
If you have an idea for a solution for that, it’s probably worth starting that discussion, because that would be worth having. It’s not a super-common need, but it does come up, and I think people usually solve it nonportably when it does. I know I’ve dealt with a couple third-party libraries (over the decades) that had to be patched to work in a non-CPython interpreter because of it. If everyone is doing it wrong because it’s too hard to do it right, it seems like it’s worth making it easier to do it right.
I think setting a patch mock's side_effect to raise an exception is a valid use case. In this example, we're simulating a blockchain error in an API that we're coding, and testing that instead of a 500 Internal Server Error, our API returns a 400 with a nice error message: with mock.patch('yourapi.w3.eth') as thing: eth.sendRawTransaction.side_effect = lambda: raise SomeException('some message') result = testclient.post('/yourapi') assert result.status_code == 400 assert result.message = 'some message'
Ok, just after I posted this I checked if side_effect would take an exception as value and handle raising it itself and it does, which sort of invalidates that specific use case, but at the same time we just deported that feature into python code of mock.
On Wed, Oct 30, 2019 at 11:29:55PM -0700, Ben Rudiak-Gould wrote:
I disagree. I think it's a pretty small breaking change, and precisely the sort of thing that __future__ was designed to handle. We could enable it on a module-by-module basis, and wouldn't need to rush to make it mandatory. The print thing in Python 3 was the most obvious and visible change, but the actual change from the perspective of the average coder was quite small. It was easily handled by a future import, and by 2to3 fixers. (Apart from one or two obscure corner cases involving softspace, if I remember correctly.) The raise statement is quite simple: raise [expression [from expression]] Syntactically, we could replace it with a function call with three cases: raise() raise(expression) raise(expression, from=expression) except we can't use "from" as the keyword name, we would have to use something else like "cause=expression" or similar. The only complexity I can think of is that tracebacks will have an extra entry, unless the traceback machinery knows enough to suppress it. I can't think of any situation where raise needs to be a statement.
Writing it in C would fix that,
That might work too :-) and if raise became a builtin function, that would have to happen regardless. -- Steven
On Oct 31, 2019, at 13:56, Steven D'Aprano <steve@pearwood.info> wrote:
I disagree. I think it's a pretty small breaking change,
From my test earlier on the thread, using a function to raise makes a function that does nothing but raise and handle an exception take more than twice as long. Which implies that the actual raise part (not counting the outer function call, handling the exception, and returning) is far more than twice as long. This wasn’t an issue with print. For one thing, print is almost always doing other slow stuff (calling str on everything, encoding the resulting string, and then writing it to a file, which typically involves a syscall, and on Windows waiting milliseconds for cmd.exe, and in 3.0 when print_function was added the horribly slow file object implementation), so the extra function overhead was a small fraction of a percent, while here it’s well over 100%. Also, there are many places in the Python data model where you have to raise an exception, such as StopIteration; there are no places where you need to print. And if you need output and print is too slow, you can always go below; there’s no way to raise an exception without raise. And what’s the advantage? If we need raise inside an expression without extra overhead, this doesn’t solve the problem; we’d need a raise expression (similar to the yield expression), not a raise function. Which has the advantage of not breaking any existing code (except for code that deals with AST nodes or tokens). If we don’t care about the overhead, there is no problem to solve; you can already get this behavior with a one-liner. If that’s still too inconvenient or undiscoverable, we can add the one-liner to builtins without removing the statement at the same time. So, what’s the benefit of removing the statement?
On Fri, Nov 1, 2019 at 12:52 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
If "raise in an expression context" is an important-enough feature, making raise an expression would be the most logical way to do it (as you say, similar to a yield expression). Wouldn't need a future directive or anything. When print was turned into a function, part of the justification was that you could then shadow the builtin with your own. If anyone has a use-case for shadowing the raise() function, I've yet to hear it, although that's not to say there isn't one :) But I don't think the overhead of a function call is worth the cost. If raise were to become an expression like yield, it would still be able to use the syntax "raise Exception from None" rather than having to use a function call, which means the backward incompatibility would be minimal. But I'm still -0 on making that change. It feels wrong to have an expression that can never have any value. If you write "spam = raise Exception()", what is spam set to? Well, nothing, because we never do that assignment. There are already lots of things you can't do with lambda functions, and we don't need to make assignment an expression just for the sake of this: transaction.side_effect = lambda: counter["foo"] += 1 If this is really such a problem for people, I'd rather push the other direction, and allow the 'def' statement to be able to assign to anything: def transaction.side_effect(): counter["foo"] += 1 It's no less "weird" than making raise an expression, and potentially has other uses too. I'm sure there'll be a ton of bikeshedding (what should __name__ and __qualname__ be?), but that can go in its own thread, if people need a cleaner solution to this problem. (You can even put that on one line, if you want it to feel more like a lambda function!) ChrisA
On Thu, Oct 31, 2019 at 02:47:35PM +0100, Andrew Barnert wrote:
I don't recall seeing your test code, but I don't follow your reasoning here. Unless you are using a custom Python build where the raise statement has been replaced by a raise function, I don't see how you can conclude that the raise statement inside a function takes "far more than twice as long" as a bare raise statement. Making raise a function will add the overhead of a function call, which by my tests is smaller than the cost of a raise and catch: $ ./python -m timeit "try: raise ValueError > except ValueError: pass" 100000 loops, best of 5: 3.26 usec per loop $ ./python -m timeit -s "def throw(): raise ValueError" "try: throw() > except ValueError: pass" 50000 loops, best of 5: 5.44 usec per loop YMMV. So on my machine, raise-in-a-function is about 70% slower than raise, not 100% or more. That's a pure-Python function. It's possible that a builtin function in C will have less overhead. But to paraphrase the prevailing opinion expressed in the "word list" thread, you cannot possibly be seriously worried about a few micro- seconds difference. (Only slightly serious.) In this case, I'm not that worried about micro-benchmarks unless they can be shown to add up to a significant performance degradation in macro-benchmarks too. The cost of catching exceptions is, as I understand it, quite a bit more expensive than actually throwing them. We can't throw lots of exceptions without a corresponding catch, so I expect that any additional overhead from a function call will probably be lost in the noise of the catch. But if that's not the case, then "performance" may end up being a good reason to reject this proposal.
Also, there are many places in the Python data model where you have to raise an exception, such as StopIteration;
Does the Python data model not have function calls? *wink*
Yield expressions are a red herring here. Yield expressions are a weird syntactic form that allow you to feed data into a running generator. raise doesn't need to do that. The raise function will never return, so it doesn't need to return a value. It will be kind of like os._exit, and sys.exit.
"Only One Way To Do It" *wink* We could have added a print_ function and kept the print statement too. And for years, people wrote their own one-liner print_ functions. (I have lost count of the number of times I did.) If your argument was compelling, we never would have changed print and exec to functions. But we did. The "just use a one-liner" argument isn't very strong. It's a work-around for the missing functionality, not a positive argument in favour of the status quo. If, back in the early days of Python, Guido had made raise a function, we'd all accept it as normal and desirable. I doubt anyone would say "I really wish it was impossible to call raise from an expression without having to write an unnecessary boilerplate wrapper function". As I said above, I'll grant you the *possibility* that performance may tilt the balance, but honestly, I see no other reason why we should prefer a *less flexible, less useful* raise statement over a more flexible, more useful function. Especially if a built-in function written in C can avoid most of the pure-Python function call overhead, narrowing the performance gap. -- Steven
On Oct 31, 2019, at 16:24, Steven D'Aprano <steve@pearwood.info> wrote:
You replied to a message from Ben that was a reply to my message with the performance test in it, which Ben was referring to in the chunk you quoted.
def throw(e): raise e Test functions using throw(ValueError("something")) vs. raise ValueError("something").
So on my machine, raise-in-a-function is about 70% slower than raise, not 100% or more.
OK, and it’s possible that on a different machine those functions would be only 50% slower instead of 70% or more than 100%. So what? You can’t answer “it’s too slow” with “it’s only too slow on some machines”, much less “it’s only too slow on some machines; on others it’s arguably maybe on the border of acceptable”. It’s still significantly slower.
That's a pure-Python function. It's possible that a builtin function in C will have less overhead.
Sure, while I already mentioned in the previous message. Other things you didn’t address include the cost of looking up a builtin (which I don’t think is a big deal, but the OP was worried about it, and your test doesn’t include it)… but better if you go back and read the thread you’re replying to rather than me trying to summarize it badly.
But this is different. Parsing a global constant is something you do once, at module compile time, so who cares about a few us. Printing is already slow, and you already can and do work around that when performance matters, so who cares about a few us. Raising exceptions is something you’re forced to do all over the place. Maybe thousands of times per second. So you may need to care about a few us. For example, if I need to Iterate through a ton of tiny sequences, each one of them has to raise StopIteration. A ton of times in the middle of my code is not the same as one time at module compilation.
I think the onus is on people who want this feature to prove that it doesn’t affect the performance of real-line code, not on people who don’t think it’s necessary to prove that it does.
Yes, it does. So if you proposed something that made every function call 100% slower on my machine (even if it was only 70% slower on yours, even if it was only a matter of a few microseconds), then I would be raising the exact same objection. And if it also broke backward compatibility, then I’d be raising that objection as well.
No, they’re a model for how you can turn a statement into an expression between Python versions without breaking backward compatibility. That doesn’t quite prove that the same thing can be done here, but it does alleviate the concern (raised earlier in the thread) that someone thought it was unlikely that you could possibly do such a thing.
A function that never returns is no less weird than an expression that never has a value. (Especially given that the only way to use a function is with a call expression anyway.) They’re both equally weird. But exceptions are one of the handful of cases where you expect it to be weird—along with things like exit and abort and break and return. The only way to avoid that weirdness is to do what Python (and C++ and some other languages) do with about half of them (including raise), and make it a statement. I think the status quo is fine, but if there really is a need for raising in the middle of a lambda or a comprehension or whatever, it has to be an expression, and I think it makes more sense for that expression to still be custom syntax rather than a magic function.
We could have added a print_ function and kept the print statement too.
Yes, but that case was different, for the reasons already discussed (print doesn’t affect flow control and doesn’t need any magic to implement as a function—you can even write it in pure Python; performance wasn’t an issue; there’s sometimes a useful benefit to shadowing a print builtin; breaking backward compatibility was less serious of an issue in 3.0 than in 3.9; …).
If he’d made raise a syntactic expression rather than a statement, would anyone be saying “I really wish it was slower to raise an expression—and, even more, I wish it used a magic function that I can’t write myself, and that it was less recognizable as flow control (especially since half the IDEs and colorizers color it like a normal function)?” If raise had been a function, we probably wouldn’t complain (except maybe for some people asking why it’s gratuitously different from all the familiar languages). But maybe we’d have been much more apologetic or much more evasive in all those “why Python isn’t too slow” blog posts when it came to the exceptions section. Plus, every time some new mechanism was designed to use exceptions, like StopIteration, there would have been a more serious objection to overcome, so we might have ended up with a different and clunkier design.
More flexible isn’t always more useful. I’m not convinced that people really do frequently need to raise in the middle of an expression. And performance is far from the only consideration—there’s backward compatibility, and looking like flow control, and so on. None of which are an issue for a raise expression—which solves the same problem, if it really is a problem that needs to be solved.
On Sun, Oct 27, 2019, at 19:17, Andrew Barnert via Python-ideas wrote:
Could raise be made an expression without adding parens? Or, C#'s throw is allowed in certain specific contexts (lambda bodies and conditional expressions) without being a general expression. Being in the true portion of a conditional expression in python would require parentheses *around* the raise, though, e.g. (raise whatever) if condition else true.
On Nov 2, 2019, at 21:02, Random832 <random832@fastmail.com> wrote:
I think the best idea is to use yield as a model: the precedence is as “statement-like” as reasonably possible, so you almost never need parens around the expression being raised, but the raise expression itself is not an expression; it’s only usable (at least initially) in parens or as a statement on its own. Like this: raise_stmt ::= raise_expr enclosure ::= ... | raise_atom raise_atom ::= "(" + raise_expr + ")" raise_expr ::= "raise" [expression ["from" expression]] (I think making it an enclosure, a sister production to parenth_form, is the right answer, but I haven’t actually tried it or worked through any complex examples.)
Or, C#'s throw is allowed in certain specific contexts (lambda bodies and conditional expressions) without being a general expression.
Just like, e.g., yield_expr is allowed in assignment statements without being a general expression, and generator_expr is allowed in a call with one arg without being a general expression. Obviously neither of those makes sense for raise (not that it would be ambiguous there, just that it would nearly always be completely useless), but maybe there are places that are, maybe including the same ones as in C#. But I think it’s better to start with the most restrictive rule, and then add additional constructions or exceptions later. If it turns out that lambda could take expression | raise_expr without ever to being ambiguous to humans or the parser, and it would often make real code read nicer, it’s easy to add that later. If we start our allowing it and it turns out to be sometimes human-ambiguous, it probably either takes 3 versions to deprecate away or has to stay as a language wart forever.
Being in the true portion of a conditional expression in python would require parentheses *around* the raise, though, e.g. (raise whatever) if condition else true.
Sure. One of the advantages of making raise_stmt just raise_expr is that it gives all of these questions obvious answers. Zillions of lines of working code raise the result of a conditional expression without parens, so raise_stmt has to still take one, so raise_expr has to as well. And that means a raise_expr obviously can’t be used as the true clause of a conditional or it would be ambiguous. (Of course a raise_atom, in parens, is a general expression, and more specifically an or_test, so it can be used there.)
On Nov 3, 2019, at 15:38, Soni L. <fakedme+py@gmail.com> wrote:
What is that a counter-argument to? The fact that it would nearly always be completely useless to assign a raise? I don’t know what NIY means here, but why couldn’t you write this as `raise NIY`? Obviously nothing is getting assigned to `foo`. If this is part of some rare but occasionally useful idiom where you’re, say, forcing `foo` to be a local rather than a global by assigning to it, then I think having to write it as `foo = (raise NIY)` seems reasonable. At any rate, as I said, if it turns out that there are useful places to allow a bare raise expression, and those places aren’t human- or parser-ambiguous, it’s easy to add them later. If people are writing `foo = (raise NIY)` and finding that the parens are often more confusing or intrusive than they are helpful, they’ll complain, and the next version of Python will change assignment statements to handle raise_expr just like they handle yield_expr. (Or, if someone can make that case even without waiting for experience, it’ll come up while bikeshedding a raise_expr PEP and can be added even in the initial version.)
Andrew Barnert via Python-ideas wrote:
I don’t know what NIY means here,
Not Invented Yet. It's used by the time machine to prevent temporal paradoxes due to anachronistic use of technology, for example if you try to use van Finkelstein's algorithm for solving the halting problem before 2186... oops, I shouldn't have mentioned that, please pretend you didn't read it. -- Greg
On 2019-11-03 2:01 p.m., Andrew Barnert wrote:
That was meant to be NYI (Not Yet Implemented), sorry. (feel free to treat "NIY" as "Not Implemented Yet" :P) It seems useful to me to have the local declared but not defined, because you haven't got around to implementing what it calls/etc, but you still know what to do with it. I generally tend to just `foo = None`, but then you run the risk of that code path actually getting silently taken and messing up program state. Then you spend hours debugging what went wrong only to find out what happened. In Rust, for example, one can do `let foo = unimplemented!();`. Ofc, one could also do: def unimplemented(): raise NYI but having the "raise" in the assignment makes it easier for a human to process - you can tell from "foo = raise NYI" that it's gonna raise an error, whereas "foo = unimplemented()" could return a placeholder value of some sort instead.
On Nov 4, 2019, at 00:58, Soni L. <fakedme+py@gmail.com> wrote:
That was meant to be NYI (Not Yet Implemented), sorry. (feel free to treat "NIY" as "Not Implemented Yet" :P)
Ah. I use NotImplementedError for that, because there are unit testing frameworks that know how to count that differently from other unexpected exceptions—although it can be a problem when what you haven’t finished is an override for an abstract method (so it’s serving a double meaning), that doesn’t come up very often.
It seems useful to me to have the local declared but not defined, because you haven't got around to implementing what it calls/etc, but you still know what to do with it. I generally tend to just `foo = None`, but then you run the risk of that code path actually getting silently taken and messing up program state. Then you spend hours debugging what went wrong only to find out what happened.
I usually raise NotImplementedError at the very top of the function, because it’s most visible there, and serves as enough of a signal that I don’t need to also add a #TODO comment. And then, I do something like this further down: foo = figure_out_how_to_calculate_from(spam, eggs) Then I don’t need foo = None with a comment saying to figure out how to calculate it from spam and eggs, because the (compiling but not running) code already says that. And if I later think I’ve finished the function and remove the NotImplementedError incorrectly, I’ll get a NameError instead of possibly incorrect values from None. If I needed to test some but not all of the behavior of a partially-written function, I suppose raising in the middle might be helpful. But usually I refactor out the parts I know how to write early when they’re significant enough to be worth testing. But anyway, I appreciate that not everyone works the same way I do, so I can see why you might find foo = raise more useful than me. In particular, if you don’t refactor the function into smaller bits, the raise presumably shows exactly how far you’ve gotten and which part isn’t done yet.
In Rust, for example, one can do `let foo = unimplemented!();`.
I actually use this in Rust, but then Rust is pretty different. If you name something figure_out_how_to_calculate without defining it anywhere, you get a compile error rather than a runtime error. You can’t do operations that are guaranteed to be illegal even in code that’s guaranteed not to run. Figuring out the types (which aren’t being inferred for you in half-written code) is sometimes half the work of writing the code. Also, at least for me, bad habits imported from other languages mean I often don’t refactor functions as much (even when it’s easy to prove that the call and moves will get inlined and optimized out, because it’s not easy in C++ I don’t instinctively go there). So for me, that doesn’t transfer into wanting to do the same thing in Python. But again, I can see how for others it might. Anyway, I’m +/-0 on a raise_expr that’s maximally restricted in the way I described, -0 on the same thing but with a rule allowing it to be used on the right side of an assignment (just like yield_expr, so it should be no problem to implement), -0.5 on making it a general expression instead of a heavily restricted one, -1 on making it a magic function (or anything else that’s not backward compatible) instead of syntax, and -0.5 on leaving it as a statement but adding throw to the stdlib somewhere.
On Sun, Nov 3, 2019, at 11:01, Andrew Barnert via Python-ideas wrote:
As a side note, I have, occasionally, wanted to be able to resume a function after handling an exception (the use case was to turn a synchronous outer function calling an asynchronous callback into an asynchronous function), which - needless to say - is impossible in CPython. In a hypothetical implementation that would allow such a thing, having the raise return a value in such a scenario might not be unreasonable.
On Tue, Nov 5, 2019, at 16:22, Greg Ewing wrote:
Like I said, my scenario was wanting to bolt async (or generator yield, I suppose) onto functions that don't support it. In the implementation I got as far as designing before realizing there was no way to resume after an exception, the exception would have been thrown (and resumed) within a callback wrapper I controlled, but being able to "do something other than raising" doesn't help it with the actual goal, which is to *stop execution of the calling function until the result of the async callback is available*.
On Nov 5, 2019, at 21:48, Random832 <random832@fastmail.com> wrote:
As a side note, I have, occasionally, wanted to be able to resume a function after handling an exception (the use case was to turn a synchronous outer function calling an asynchronous callback into an asynchronous function), which - needless to say - is impossible in CPython. In a hypothetical implementation that would allow such a thing, having the raise return a value in such a scenario might not be unreasonable.
That’s not just a CPython limitation, it’s a limitation of the defined Python semantics. Python unwinds the stack before calling exception handlers. For function exits that’s not a big deal (you’re not allowed to rely on garbage being cleaned up deterministically, and you can’t detect that something is garbage until it’s cleaned up), but it would change the semantics—and break most non-trivial uses—of finally clauses and with statements. You could maybe redesign the context manager protocol so they understand and participate in resume logic somehow (separate __initial_exit__ and __final_exit__?). But what do you do with finally clauses? Another option is to do what Dylan does. I don’t actually remember the details, but it’s something like this: the way you resume an exception is to resume-raise a new exception back at the point it was raised. Then, the interpreter can stack up exception handlers and unwind chunks in a way that makes sense (it needs to rearrange that stack on the fly, but only in a simple way that Python already requires even for simple generators). Code that doesn’t ever resume works the same as always. Code that does has to be written differently, and has to be explicit about the order of unwinds and handlers, but can do so just by using the normal Python indentation (the try statement that handles the resume-raise is either inside or outside the with or try/finally or function). This also makes it very easy to turn a resume into a retry (by just putting a while outside the inner try). And anything retryable also has the advantage that pdb can optionally keep the whole stack around until you decide not to retry, which can be helpful for grubbing around in the debugger, but then clean it all up properly as soon as you do decide. IIRC, Common Lisp does something pretty similar to Dylan, except that instead of resumes being exceptions they’re a separate thing that just works nearly identical to exceptions. That might be more readable, or maybe even easier to implement, even if it adds more concepts to the language. I believe there’s also a rejected C++ proposal for resumable exceptions, and a g++ fork that implements it, from way back before C++11. Since C++ is heavily designed around RAII (everything is a context manager), they must have come up with a solution. On the other hand, there’s probably a reason it was rejected—although that reason might just be “We had an extensive discussion on signal-like vs. termination semantics when first doing ANSI C++, and this is close enough to signal-like that without any new use case we’re not going to reopen that discussion.” I don’t know of any other post-70s languages that do resumable exception handling that aren’t continuation-based, but they probably do exist. But for a new Python-like language with resumable exceptions, I think you’d want to do it with continuations, because that’s the easy way. Adding a callcc and first-class continuation objects would probably be easier if you first removed try and with entirely, then translate everything to cc semantics, then redesign them on top of cc. Everything should be pretty simple. And you can even experiment with writing variations of the logic in-language as functions before writing the Python syntactic sugar. At that point, you could even remove the first-class continuations if you want, and translate the language back to non-continuation-based semantics. It might be easier to optimize things that way, and it would probably make it a lot easier to write Python implementations similar to Jython and Iron that rely as much as possible on the semantics of an underlying high-level language. Or maybe don’t bother with any new syntax. Once you’ve got existing Python semantics plus callcc, just leave syntactic exceptions as-is with no resume, and make resumable (and retryable) exceptions a library thing that you can use in the cases where they‘re needed; those cases are probably rare enough that it doesn’t matter if they’re not pretty. Also, I think you’d find that almost everywhere you think you want a resumable exception, you can actually do it more simply by either using continuations directly, or using a different abstraction on top of them. That certainly seems true in your use case, where your goal (making a sync function async) has nothing to do with exceptions, and the only reason you thought of resumable exceptions was, effectively, as a way to clumsily simulate callcc.
On Sun, Oct 27, 2019 at 6:58 AM Soni L. <fakedme+py@gmail.com> wrote:
foo.setParseAction(lambda a, b, c: raise FuckPython(":(")) is invalid syntax, which makes pyparsing useless.
Insulting the project that you're trying to get help with is unnecessary to get your point across. Consider this a warning that such a poor attitude is not welcome here.
On 10/27/2019 10:35 AM, Soni L. wrote:
that's a waste of a perfectly good name in the namespace.
Names are cheap. If "wasting" a name is a concern, then perhaps Python isn't the language for you. It also lacks local scopes inside code blocks, for example, which also "waste" names, as compared to say C. Eric
On Sun, Oct 27, 2019 at 06:56:16PM -0300, Soni L. wrote:
that's why I use nested functions.
I'm pretty sure that the memory and time overhead of using nested functions is far greater than the saving in names. I'm not even sure what it is that you think you are saving. Why do you care if a function used ten local names or 15 local names? -- Steven
throw is an expression, not a statement, in C++. I see no reason raise couldn't be an expression in Python. It doesn't even need a special rule in the grammar: from __future__ import raise_function foo.setParseAction(lambda a, b, c: raise(MumbleMumble())) Looking up and calling the raise function would add overhead to every explicitly raised exception, but exceptions are so expensive anyway that I think it wouldn't be noticeable.
On Oct 27, 2019, at 15:07, Ben Rudiak-Gould <benrudiak@gmail.com> wrote:
That’s a pretty big breaking change. Every line of code in every library and app that raises would have to change to add parens. That was acceptable between 2.x and 3.x for print, but I don’t think it would be acceptable between 3.8 and 3.9, or even between 3.8 and 3.11 with a future schedule unless there was a really good reason for it. And there really isn’t. Anyone who wants this can do this today: def throw(e): raise e Install it into builtins or import it into all of your modules, and you’re done. Meanwhile, all of the library code you use that has raise statements continues to work unchanged. And meanwhile, if you want to change the language to turn raise into an expression, why not just turn it into an expression with the same syntax, just like with yield somewhere around 2.4? Then all existing code that raises exceptions continues to work; only code that relies on the grammar or AST breaks (and that’s explicitly allowed to break between versions). I’m pretty sure you wouldn’t need any change to the bytecodes and ceval loop, and only a small change to the codegen part of the compiler, but I can’t be sure without actually trying it. It might be ambiguous to humans, if not to the parser, to do a bare raise in some contexts. If so, that can be solved the exact same way as yield. There’s a bit of bikeshedding to be done (the grammar for where it can appear doesn’t have to be identical to yield_expression, so you can decide, e.g., to ban x = raise e as silly and force people to write x = (raise e), or decide that’s consenting-adults silliness and no reason to ban it), but I don’t think there’s anything difficult beyond getting everyone to agree on your bikeshed color.
Are you sure about that? Exceptions aren’t nearly as slow as many people seem to think; otherwise, Python couldn’t get away with using them all over the place internally. Try this: def throw(e): raise e builtins.throw = throw del throw def f(): try: throw(ValueError('spam')) except Exception as e: return e %timeit f() And then replace the throw call with a raise statement and try again. I get 1350ns vs. 548ns, and that’s even before subtracting out the time to look up and call f and to return e. I don’t know how much of this is builtin-lookup vs. function calling vs. needing more of the code and data caches vs. whatever, and the function calling part would get faster if it were a C rather than Python function, but it’s still not going to be unnoticeable. And if you’re thinking maybe this isn’t a good test because maybe CPython is able to optimize something because the exception is handled in the same scope that raises it, you could check that with dis, or just try this: def f(): raise ValueError('spam') def g(): throw(ValueError('spam')) def h(fun): try: fun() except Exception(e): pass Now it’s 1950ns vs. 734ns. A raise expression, on the other hand, would add no performance cost. At any rate, if you really want this, just define that throw function yourself—or one that takes a type and *args and **kw and constructs the exception, or whatever API you want—and use it in place of raise in your code. It’s as trivial a function as could be, and if you don’t raise exceptions in a bottleneck, it probably doesn’t matter that you’ve made them more than twice as slow to raise, and you won’t be breaking any builtin or library code that expects it to be faster because that code is still using raise statements.
On Mon, Oct 28, 2019 at 10:20 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
And meanwhile, if you want to change the language to turn raise into an expression, why not just turn it into an expression with the same syntax, just like with yield somewhere around 2.4? Then all existing code that raises exceptions continues to work; only code that relies on the grammar or AST breaks (and that’s explicitly allowed to break between versions).
When yield was turned into an expression, it was to allow it to have a value: def coro(): value = yield "Start" print("I got value:", value) yield "End" x = coro() print(next(x)) print(x.send("Hello")) But a raise expression doesn't really make sense in that way - it can't possibly have any value. It would be possible to use a raise expression in a lambda function or an if/else expression, but other than that, there wouldn't be much to gain. ChrisA
On Oct 27, 2019, at 16:37, Chris Angelico <rosuav@gmail.com> wrote:
Well, also comprehensions. (And, I suppose, or/and expressions abused as if/else shortcuts, but that’s even less useful than lambdas and comprehensions.) But I’m not arguing that this is useful for the same reason yield_expression was useful. I don’t think this is useful enough to change the language at all. It’s perfectly fine that in the rare occasions when you have a good reason to raise in a lambda or a comprehension, you have to define a trivial one-line throw function. The point of the paragraph you’re replying to is that if I’m wrong about that, and we really do need to change the language to allow raise in expressions, it should be done like yield, not like print.
On Mon, Oct 28, 2019 at 11:17 AM Andrew Barnert <abarnert@yahoo.com> wrote:
Fair point. Given the 'if', I agree with the conclusion. I'm of the opinion that you aren't wrong about that, though, and the fact that throw() isn't in everyone's toolkits already suggests that this really isn't a major problem to be solved. ChrisA
On Oct 27, 2019, at 17:27, Chris Angelico <rosuav@gmail.com> wrote:
Searching on “python raise expression”, the first hit is a StackOverflow question where the asker is looking for exactly this functionality, and, while the accepted answer is “just use a statement” (with some nice explanation), another upvoted answers offers (and explains) a throw function named “raiser”. There are lots of other SO hits, and hits elsewhere, and about half of them are for this issue and include some variant of this solution. I didn’t see a single compelling examples where it really seemed justified, but still, people are looking for it, and finding it. (And they’re finding it under a variety of different names, which makes it harder to mass search code to see how often it’s being used…) Grepping my pile of one-off scripts, I haven’t used my throw function in a few years, but I have used it at least a couple dozen times in the past. Many of them are abusing throw(StopIteration) to fake a “takewhile clause” in comprehensions (which was never a good idea, and doesn’t even work anymore since 3.7, but apparently I used to do it…), and all of the rest would be not just more readable, but shorter too, if I broke up the giant expression the throw was in. But I think the OP might prefer my old giant-expression code, even though I think it’s terrible, because the cleaner and shorter version would usually “waste” at least one local variable storing an intermediate. Anyway, I don’t know how much all of this matters. Whether people are using it frequently or not, and whether the vast majority of uses are bad or not, it’s as easy to find it today as it would be if we added it to the docs somewhere or even included it somewhere in the stdlib, and trivial to use once you find it, so i don’t think we need to bend over backward to make it easier.
On Oct 28, 2019, at 14:23, Serhiy Storchaka <storchaka@gmail.com> wrote:
Well, what I really (thought I) needed was a more concise way to do the equivalent of takewhile and friends, and I still think the ideal solution there is a general way to write concise lambdas or the equivalent using a great new syntax that nobody has been able to come up with yet in any of the times the idea has been discussed. :) Faking it by raising StopIteration in the middle of the value expression or an if clause seemed like a clever workaround, albeit only for takewhile. But it actually has a lot of problems; various things that shouldn’t change your semantics do if you’re using this hack. (For example, chain up a few throw-breaking genexprs, then change one of the intermediate ones to a listcomp so you can see it in the debugger.) So, I’d already abandoned it before the PEP that broke it. Would a break expression be able to give me the same benefits without the problems? Maybe. But I honestly haven’t missed the throw(StopIteration) hack enough to experiment with that idea when people suggested it.
On Sun, Oct 27, 2019 at 4:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
I agree, it's a bad idea. It'd have to be Python 4, like print_function was Python 3.
def throw(e): raise e
The problem with this is that it adds an extra line to the traceback, so you have to choose between the convenience of using it and the convenience of more readable tracebacks. Writing it in C would fix that, but it's pretty complicated (I think I could copy the logic from do_raise in ceval.c, but I'm not completely sure, and it's almost 100 lines long), and of course not portable. I wouldn't mind having a sys.throw or something that's guaranteed to do it correctly. Actually, it'd be nice to have some general way to write trivial wrappers in Python without uglifying tracebacks, but that's a different discussion.
On Oct 31, 2019, at 07:29, Ben Rudiak-Gould <benrudiak@gmail.com> wrote:
Is a lot simpler than that. But you shouldn’t have to do it nonetheless. You can also hack up a solution in pure Python by manipulating the traceback manually. The quick&dirty way to do this using the frame objects is nonportable, but it is doable portably, it just might be a lot more of a pain. In fact, I think you can borrow and simplify code from importlib to do it. When the import machinery was rewritten from mostly C to pure Python around 3.4, they had to come up with a workaround to make ImportError tracebacks still look like they used to instead of including an extra 6 functions in the middle that nobody cares about except people working on the import system or an import hook. IIRC, what they do is call a bracketing function on entering the import machinery and then another one on calling back out, and then the top level functions have code that edits tracebacks that pass through by removing entries between those two brackets.
If you have an idea for a solution for that, it’s probably worth starting that discussion, because that would be worth having. It’s not a super-common need, but it does come up, and I think people usually solve it nonportably when it does. I know I’ve dealt with a couple third-party libraries (over the decades) that had to be patched to work in a non-CPython interpreter because of it. If everyone is doing it wrong because it’s too hard to do it right, it seems like it’s worth making it easier to do it right.
I think setting a patch mock's side_effect to raise an exception is a valid use case. In this example, we're simulating a blockchain error in an API that we're coding, and testing that instead of a 500 Internal Server Error, our API returns a 400 with a nice error message: with mock.patch('yourapi.w3.eth') as thing: eth.sendRawTransaction.side_effect = lambda: raise SomeException('some message') result = testclient.post('/yourapi') assert result.status_code == 400 assert result.message = 'some message'
Ok, just after I posted this I checked if side_effect would take an exception as value and handle raising it itself and it does, which sort of invalidates that specific use case, but at the same time we just deported that feature into python code of mock.
On Wed, Oct 30, 2019 at 11:29:55PM -0700, Ben Rudiak-Gould wrote:
I disagree. I think it's a pretty small breaking change, and precisely the sort of thing that __future__ was designed to handle. We could enable it on a module-by-module basis, and wouldn't need to rush to make it mandatory. The print thing in Python 3 was the most obvious and visible change, but the actual change from the perspective of the average coder was quite small. It was easily handled by a future import, and by 2to3 fixers. (Apart from one or two obscure corner cases involving softspace, if I remember correctly.) The raise statement is quite simple: raise [expression [from expression]] Syntactically, we could replace it with a function call with three cases: raise() raise(expression) raise(expression, from=expression) except we can't use "from" as the keyword name, we would have to use something else like "cause=expression" or similar. The only complexity I can think of is that tracebacks will have an extra entry, unless the traceback machinery knows enough to suppress it. I can't think of any situation where raise needs to be a statement.
Writing it in C would fix that,
That might work too :-) and if raise became a builtin function, that would have to happen regardless. -- Steven
On Oct 31, 2019, at 13:56, Steven D'Aprano <steve@pearwood.info> wrote:
I disagree. I think it's a pretty small breaking change,
From my test earlier on the thread, using a function to raise makes a function that does nothing but raise and handle an exception take more than twice as long. Which implies that the actual raise part (not counting the outer function call, handling the exception, and returning) is far more than twice as long. This wasn’t an issue with print. For one thing, print is almost always doing other slow stuff (calling str on everything, encoding the resulting string, and then writing it to a file, which typically involves a syscall, and on Windows waiting milliseconds for cmd.exe, and in 3.0 when print_function was added the horribly slow file object implementation), so the extra function overhead was a small fraction of a percent, while here it’s well over 100%. Also, there are many places in the Python data model where you have to raise an exception, such as StopIteration; there are no places where you need to print. And if you need output and print is too slow, you can always go below; there’s no way to raise an exception without raise. And what’s the advantage? If we need raise inside an expression without extra overhead, this doesn’t solve the problem; we’d need a raise expression (similar to the yield expression), not a raise function. Which has the advantage of not breaking any existing code (except for code that deals with AST nodes or tokens). If we don’t care about the overhead, there is no problem to solve; you can already get this behavior with a one-liner. If that’s still too inconvenient or undiscoverable, we can add the one-liner to builtins without removing the statement at the same time. So, what’s the benefit of removing the statement?
On Fri, Nov 1, 2019 at 12:52 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
If "raise in an expression context" is an important-enough feature, making raise an expression would be the most logical way to do it (as you say, similar to a yield expression). Wouldn't need a future directive or anything. When print was turned into a function, part of the justification was that you could then shadow the builtin with your own. If anyone has a use-case for shadowing the raise() function, I've yet to hear it, although that's not to say there isn't one :) But I don't think the overhead of a function call is worth the cost. If raise were to become an expression like yield, it would still be able to use the syntax "raise Exception from None" rather than having to use a function call, which means the backward incompatibility would be minimal. But I'm still -0 on making that change. It feels wrong to have an expression that can never have any value. If you write "spam = raise Exception()", what is spam set to? Well, nothing, because we never do that assignment. There are already lots of things you can't do with lambda functions, and we don't need to make assignment an expression just for the sake of this: transaction.side_effect = lambda: counter["foo"] += 1 If this is really such a problem for people, I'd rather push the other direction, and allow the 'def' statement to be able to assign to anything: def transaction.side_effect(): counter["foo"] += 1 It's no less "weird" than making raise an expression, and potentially has other uses too. I'm sure there'll be a ton of bikeshedding (what should __name__ and __qualname__ be?), but that can go in its own thread, if people need a cleaner solution to this problem. (You can even put that on one line, if you want it to feel more like a lambda function!) ChrisA
On Thu, Oct 31, 2019 at 02:47:35PM +0100, Andrew Barnert wrote:
I don't recall seeing your test code, but I don't follow your reasoning here. Unless you are using a custom Python build where the raise statement has been replaced by a raise function, I don't see how you can conclude that the raise statement inside a function takes "far more than twice as long" as a bare raise statement. Making raise a function will add the overhead of a function call, which by my tests is smaller than the cost of a raise and catch: $ ./python -m timeit "try: raise ValueError > except ValueError: pass" 100000 loops, best of 5: 3.26 usec per loop $ ./python -m timeit -s "def throw(): raise ValueError" "try: throw() > except ValueError: pass" 50000 loops, best of 5: 5.44 usec per loop YMMV. So on my machine, raise-in-a-function is about 70% slower than raise, not 100% or more. That's a pure-Python function. It's possible that a builtin function in C will have less overhead. But to paraphrase the prevailing opinion expressed in the "word list" thread, you cannot possibly be seriously worried about a few micro- seconds difference. (Only slightly serious.) In this case, I'm not that worried about micro-benchmarks unless they can be shown to add up to a significant performance degradation in macro-benchmarks too. The cost of catching exceptions is, as I understand it, quite a bit more expensive than actually throwing them. We can't throw lots of exceptions without a corresponding catch, so I expect that any additional overhead from a function call will probably be lost in the noise of the catch. But if that's not the case, then "performance" may end up being a good reason to reject this proposal.
Also, there are many places in the Python data model where you have to raise an exception, such as StopIteration;
Does the Python data model not have function calls? *wink*
Yield expressions are a red herring here. Yield expressions are a weird syntactic form that allow you to feed data into a running generator. raise doesn't need to do that. The raise function will never return, so it doesn't need to return a value. It will be kind of like os._exit, and sys.exit.
"Only One Way To Do It" *wink* We could have added a print_ function and kept the print statement too. And for years, people wrote their own one-liner print_ functions. (I have lost count of the number of times I did.) If your argument was compelling, we never would have changed print and exec to functions. But we did. The "just use a one-liner" argument isn't very strong. It's a work-around for the missing functionality, not a positive argument in favour of the status quo. If, back in the early days of Python, Guido had made raise a function, we'd all accept it as normal and desirable. I doubt anyone would say "I really wish it was impossible to call raise from an expression without having to write an unnecessary boilerplate wrapper function". As I said above, I'll grant you the *possibility* that performance may tilt the balance, but honestly, I see no other reason why we should prefer a *less flexible, less useful* raise statement over a more flexible, more useful function. Especially if a built-in function written in C can avoid most of the pure-Python function call overhead, narrowing the performance gap. -- Steven
On Oct 31, 2019, at 16:24, Steven D'Aprano <steve@pearwood.info> wrote:
You replied to a message from Ben that was a reply to my message with the performance test in it, which Ben was referring to in the chunk you quoted.
def throw(e): raise e Test functions using throw(ValueError("something")) vs. raise ValueError("something").
So on my machine, raise-in-a-function is about 70% slower than raise, not 100% or more.
OK, and it’s possible that on a different machine those functions would be only 50% slower instead of 70% or more than 100%. So what? You can’t answer “it’s too slow” with “it’s only too slow on some machines”, much less “it’s only too slow on some machines; on others it’s arguably maybe on the border of acceptable”. It’s still significantly slower.
That's a pure-Python function. It's possible that a builtin function in C will have less overhead.
Sure, while I already mentioned in the previous message. Other things you didn’t address include the cost of looking up a builtin (which I don’t think is a big deal, but the OP was worried about it, and your test doesn’t include it)… but better if you go back and read the thread you’re replying to rather than me trying to summarize it badly.
But this is different. Parsing a global constant is something you do once, at module compile time, so who cares about a few us. Printing is already slow, and you already can and do work around that when performance matters, so who cares about a few us. Raising exceptions is something you’re forced to do all over the place. Maybe thousands of times per second. So you may need to care about a few us. For example, if I need to Iterate through a ton of tiny sequences, each one of them has to raise StopIteration. A ton of times in the middle of my code is not the same as one time at module compilation.
I think the onus is on people who want this feature to prove that it doesn’t affect the performance of real-line code, not on people who don’t think it’s necessary to prove that it does.
Yes, it does. So if you proposed something that made every function call 100% slower on my machine (even if it was only 70% slower on yours, even if it was only a matter of a few microseconds), then I would be raising the exact same objection. And if it also broke backward compatibility, then I’d be raising that objection as well.
No, they’re a model for how you can turn a statement into an expression between Python versions without breaking backward compatibility. That doesn’t quite prove that the same thing can be done here, but it does alleviate the concern (raised earlier in the thread) that someone thought it was unlikely that you could possibly do such a thing.
A function that never returns is no less weird than an expression that never has a value. (Especially given that the only way to use a function is with a call expression anyway.) They’re both equally weird. But exceptions are one of the handful of cases where you expect it to be weird—along with things like exit and abort and break and return. The only way to avoid that weirdness is to do what Python (and C++ and some other languages) do with about half of them (including raise), and make it a statement. I think the status quo is fine, but if there really is a need for raising in the middle of a lambda or a comprehension or whatever, it has to be an expression, and I think it makes more sense for that expression to still be custom syntax rather than a magic function.
We could have added a print_ function and kept the print statement too.
Yes, but that case was different, for the reasons already discussed (print doesn’t affect flow control and doesn’t need any magic to implement as a function—you can even write it in pure Python; performance wasn’t an issue; there’s sometimes a useful benefit to shadowing a print builtin; breaking backward compatibility was less serious of an issue in 3.0 than in 3.9; …).
If he’d made raise a syntactic expression rather than a statement, would anyone be saying “I really wish it was slower to raise an expression—and, even more, I wish it used a magic function that I can’t write myself, and that it was less recognizable as flow control (especially since half the IDEs and colorizers color it like a normal function)?” If raise had been a function, we probably wouldn’t complain (except maybe for some people asking why it’s gratuitously different from all the familiar languages). But maybe we’d have been much more apologetic or much more evasive in all those “why Python isn’t too slow” blog posts when it came to the exceptions section. Plus, every time some new mechanism was designed to use exceptions, like StopIteration, there would have been a more serious objection to overcome, so we might have ended up with a different and clunkier design.
More flexible isn’t always more useful. I’m not convinced that people really do frequently need to raise in the middle of an expression. And performance is far from the only consideration—there’s backward compatibility, and looking like flow control, and so on. None of which are an issue for a raise expression—which solves the same problem, if it really is a problem that needs to be solved.
On Sun, Oct 27, 2019, at 19:17, Andrew Barnert via Python-ideas wrote:
Could raise be made an expression without adding parens? Or, C#'s throw is allowed in certain specific contexts (lambda bodies and conditional expressions) without being a general expression. Being in the true portion of a conditional expression in python would require parentheses *around* the raise, though, e.g. (raise whatever) if condition else true.
On Nov 2, 2019, at 21:02, Random832 <random832@fastmail.com> wrote:
I think the best idea is to use yield as a model: the precedence is as “statement-like” as reasonably possible, so you almost never need parens around the expression being raised, but the raise expression itself is not an expression; it’s only usable (at least initially) in parens or as a statement on its own. Like this: raise_stmt ::= raise_expr enclosure ::= ... | raise_atom raise_atom ::= "(" + raise_expr + ")" raise_expr ::= "raise" [expression ["from" expression]] (I think making it an enclosure, a sister production to parenth_form, is the right answer, but I haven’t actually tried it or worked through any complex examples.)
Or, C#'s throw is allowed in certain specific contexts (lambda bodies and conditional expressions) without being a general expression.
Just like, e.g., yield_expr is allowed in assignment statements without being a general expression, and generator_expr is allowed in a call with one arg without being a general expression. Obviously neither of those makes sense for raise (not that it would be ambiguous there, just that it would nearly always be completely useless), but maybe there are places that are, maybe including the same ones as in C#. But I think it’s better to start with the most restrictive rule, and then add additional constructions or exceptions later. If it turns out that lambda could take expression | raise_expr without ever to being ambiguous to humans or the parser, and it would often make real code read nicer, it’s easy to add that later. If we start our allowing it and it turns out to be sometimes human-ambiguous, it probably either takes 3 versions to deprecate away or has to stay as a language wart forever.
Being in the true portion of a conditional expression in python would require parentheses *around* the raise, though, e.g. (raise whatever) if condition else true.
Sure. One of the advantages of making raise_stmt just raise_expr is that it gives all of these questions obvious answers. Zillions of lines of working code raise the result of a conditional expression without parens, so raise_stmt has to still take one, so raise_expr has to as well. And that means a raise_expr obviously can’t be used as the true clause of a conditional or it would be ambiguous. (Of course a raise_atom, in parens, is a general expression, and more specifically an or_test, so it can be used there.)
On Nov 3, 2019, at 15:38, Soni L. <fakedme+py@gmail.com> wrote:
What is that a counter-argument to? The fact that it would nearly always be completely useless to assign a raise? I don’t know what NIY means here, but why couldn’t you write this as `raise NIY`? Obviously nothing is getting assigned to `foo`. If this is part of some rare but occasionally useful idiom where you’re, say, forcing `foo` to be a local rather than a global by assigning to it, then I think having to write it as `foo = (raise NIY)` seems reasonable. At any rate, as I said, if it turns out that there are useful places to allow a bare raise expression, and those places aren’t human- or parser-ambiguous, it’s easy to add them later. If people are writing `foo = (raise NIY)` and finding that the parens are often more confusing or intrusive than they are helpful, they’ll complain, and the next version of Python will change assignment statements to handle raise_expr just like they handle yield_expr. (Or, if someone can make that case even without waiting for experience, it’ll come up while bikeshedding a raise_expr PEP and can be added even in the initial version.)
Andrew Barnert via Python-ideas wrote:
I don’t know what NIY means here,
Not Invented Yet. It's used by the time machine to prevent temporal paradoxes due to anachronistic use of technology, for example if you try to use van Finkelstein's algorithm for solving the halting problem before 2186... oops, I shouldn't have mentioned that, please pretend you didn't read it. -- Greg
On 2019-11-03 2:01 p.m., Andrew Barnert wrote:
That was meant to be NYI (Not Yet Implemented), sorry. (feel free to treat "NIY" as "Not Implemented Yet" :P) It seems useful to me to have the local declared but not defined, because you haven't got around to implementing what it calls/etc, but you still know what to do with it. I generally tend to just `foo = None`, but then you run the risk of that code path actually getting silently taken and messing up program state. Then you spend hours debugging what went wrong only to find out what happened. In Rust, for example, one can do `let foo = unimplemented!();`. Ofc, one could also do: def unimplemented(): raise NYI but having the "raise" in the assignment makes it easier for a human to process - you can tell from "foo = raise NYI" that it's gonna raise an error, whereas "foo = unimplemented()" could return a placeholder value of some sort instead.
On Nov 4, 2019, at 00:58, Soni L. <fakedme+py@gmail.com> wrote:
That was meant to be NYI (Not Yet Implemented), sorry. (feel free to treat "NIY" as "Not Implemented Yet" :P)
Ah. I use NotImplementedError for that, because there are unit testing frameworks that know how to count that differently from other unexpected exceptions—although it can be a problem when what you haven’t finished is an override for an abstract method (so it’s serving a double meaning), that doesn’t come up very often.
It seems useful to me to have the local declared but not defined, because you haven't got around to implementing what it calls/etc, but you still know what to do with it. I generally tend to just `foo = None`, but then you run the risk of that code path actually getting silently taken and messing up program state. Then you spend hours debugging what went wrong only to find out what happened.
I usually raise NotImplementedError at the very top of the function, because it’s most visible there, and serves as enough of a signal that I don’t need to also add a #TODO comment. And then, I do something like this further down: foo = figure_out_how_to_calculate_from(spam, eggs) Then I don’t need foo = None with a comment saying to figure out how to calculate it from spam and eggs, because the (compiling but not running) code already says that. And if I later think I’ve finished the function and remove the NotImplementedError incorrectly, I’ll get a NameError instead of possibly incorrect values from None. If I needed to test some but not all of the behavior of a partially-written function, I suppose raising in the middle might be helpful. But usually I refactor out the parts I know how to write early when they’re significant enough to be worth testing. But anyway, I appreciate that not everyone works the same way I do, so I can see why you might find foo = raise more useful than me. In particular, if you don’t refactor the function into smaller bits, the raise presumably shows exactly how far you’ve gotten and which part isn’t done yet.
In Rust, for example, one can do `let foo = unimplemented!();`.
I actually use this in Rust, but then Rust is pretty different. If you name something figure_out_how_to_calculate without defining it anywhere, you get a compile error rather than a runtime error. You can’t do operations that are guaranteed to be illegal even in code that’s guaranteed not to run. Figuring out the types (which aren’t being inferred for you in half-written code) is sometimes half the work of writing the code. Also, at least for me, bad habits imported from other languages mean I often don’t refactor functions as much (even when it’s easy to prove that the call and moves will get inlined and optimized out, because it’s not easy in C++ I don’t instinctively go there). So for me, that doesn’t transfer into wanting to do the same thing in Python. But again, I can see how for others it might. Anyway, I’m +/-0 on a raise_expr that’s maximally restricted in the way I described, -0 on the same thing but with a rule allowing it to be used on the right side of an assignment (just like yield_expr, so it should be no problem to implement), -0.5 on making it a general expression instead of a heavily restricted one, -1 on making it a magic function (or anything else that’s not backward compatible) instead of syntax, and -0.5 on leaving it as a statement but adding throw to the stdlib somewhere.
On Sun, Nov 3, 2019, at 11:01, Andrew Barnert via Python-ideas wrote:
As a side note, I have, occasionally, wanted to be able to resume a function after handling an exception (the use case was to turn a synchronous outer function calling an asynchronous callback into an asynchronous function), which - needless to say - is impossible in CPython. In a hypothetical implementation that would allow such a thing, having the raise return a value in such a scenario might not be unreasonable.
On Tue, Nov 5, 2019, at 16:22, Greg Ewing wrote:
Like I said, my scenario was wanting to bolt async (or generator yield, I suppose) onto functions that don't support it. In the implementation I got as far as designing before realizing there was no way to resume after an exception, the exception would have been thrown (and resumed) within a callback wrapper I controlled, but being able to "do something other than raising" doesn't help it with the actual goal, which is to *stop execution of the calling function until the result of the async callback is available*.
On Nov 5, 2019, at 21:48, Random832 <random832@fastmail.com> wrote:
As a side note, I have, occasionally, wanted to be able to resume a function after handling an exception (the use case was to turn a synchronous outer function calling an asynchronous callback into an asynchronous function), which - needless to say - is impossible in CPython. In a hypothetical implementation that would allow such a thing, having the raise return a value in such a scenario might not be unreasonable.
That’s not just a CPython limitation, it’s a limitation of the defined Python semantics. Python unwinds the stack before calling exception handlers. For function exits that’s not a big deal (you’re not allowed to rely on garbage being cleaned up deterministically, and you can’t detect that something is garbage until it’s cleaned up), but it would change the semantics—and break most non-trivial uses—of finally clauses and with statements. You could maybe redesign the context manager protocol so they understand and participate in resume logic somehow (separate __initial_exit__ and __final_exit__?). But what do you do with finally clauses? Another option is to do what Dylan does. I don’t actually remember the details, but it’s something like this: the way you resume an exception is to resume-raise a new exception back at the point it was raised. Then, the interpreter can stack up exception handlers and unwind chunks in a way that makes sense (it needs to rearrange that stack on the fly, but only in a simple way that Python already requires even for simple generators). Code that doesn’t ever resume works the same as always. Code that does has to be written differently, and has to be explicit about the order of unwinds and handlers, but can do so just by using the normal Python indentation (the try statement that handles the resume-raise is either inside or outside the with or try/finally or function). This also makes it very easy to turn a resume into a retry (by just putting a while outside the inner try). And anything retryable also has the advantage that pdb can optionally keep the whole stack around until you decide not to retry, which can be helpful for grubbing around in the debugger, but then clean it all up properly as soon as you do decide. IIRC, Common Lisp does something pretty similar to Dylan, except that instead of resumes being exceptions they’re a separate thing that just works nearly identical to exceptions. That might be more readable, or maybe even easier to implement, even if it adds more concepts to the language. I believe there’s also a rejected C++ proposal for resumable exceptions, and a g++ fork that implements it, from way back before C++11. Since C++ is heavily designed around RAII (everything is a context manager), they must have come up with a solution. On the other hand, there’s probably a reason it was rejected—although that reason might just be “We had an extensive discussion on signal-like vs. termination semantics when first doing ANSI C++, and this is close enough to signal-like that without any new use case we’re not going to reopen that discussion.” I don’t know of any other post-70s languages that do resumable exception handling that aren’t continuation-based, but they probably do exist. But for a new Python-like language with resumable exceptions, I think you’d want to do it with continuations, because that’s the easy way. Adding a callcc and first-class continuation objects would probably be easier if you first removed try and with entirely, then translate everything to cc semantics, then redesign them on top of cc. Everything should be pretty simple. And you can even experiment with writing variations of the logic in-language as functions before writing the Python syntactic sugar. At that point, you could even remove the first-class continuations if you want, and translate the language back to non-continuation-based semantics. It might be easier to optimize things that way, and it would probably make it a lot easier to write Python implementations similar to Jython and Iron that rely as much as possible on the semantics of an underlying high-level language. Or maybe don’t bother with any new syntax. Once you’ve got existing Python semantics plus callcc, just leave syntactic exceptions as-is with no resume, and make resumable (and retryable) exceptions a library thing that you can use in the cases where they‘re needed; those cases are probably rare enough that it doesn’t matter if they’re not pretty. Also, I think you’d find that almost everywhere you think you want a resumable exception, you can actually do it more simply by either using continuations directly, or using a different abstraction on top of them. That certainly seems true in your use case, where your goal (making a sync function async) has nothing to do with exceptions, and the only reason you thought of resumable exceptions was, effectively, as a way to clumsily simulate callcc.
On Sun, Oct 27, 2019 at 6:58 AM Soni L. <fakedme+py@gmail.com> wrote:
foo.setParseAction(lambda a, b, c: raise FuckPython(":(")) is invalid syntax, which makes pyparsing useless.
Insulting the project that you're trying to get help with is unnecessary to get your point across. Consider this a warning that such a poor attitude is not welcome here.
participants (13)
-
Anders Hovmöller -
Andrew Barnert -
Ben Rudiak-Gould -
Brett Cannon -
Chris Angelico -
Eric V. Smith -
Greg Ewing -
J. Pic -
Jonathan Goble -
Random832 -
Serhiy Storchaka -
Soni L. -
Steven D'Aprano