Hi, I don't know if someone has already suggested this before, but here goes: With expressions allow using the enter/exit semantics of the with statement inside an expression context. Examples: contents = f.read() with open('file') as f #the most obvious one multiplecontents = [f.read() with open(name) as f for name in names] #reading multiple files I don't know if it's worth making the "as NAME" part of the with mandatory in an expression - is this a valid use case? data = database.selectrows() with threadlock Where this would benefit: I think the major use case is `f.read() with open('file') as f`. Previous documentation has suggested `open('file').read()` and rely on garbage collection; as the disadvantages of that became obvious, it transitioned to a method that couldn't be done in an expression: with open('file') as f: contents = f.read() Therefore `f.read() with open('file') as f`, I think, would be much welcomed as the best way to read a file in an expression. For those wondering about the scope semantics of the "as NAME", I think they would be identical to the scope semantics of the "for" expression - i.e. these are legal: contents = f.read() with open('file') as f grid = [[i] * 4 for i in range(4)] But these are not: contents = f.read() with open('file') as f f.seek(0) grid = [[i] * 4 for i in range(4)] grid[i][i] = 4 Is this a good idea? Are there some subtleties I've failed to explain? Please let me know. Sharing, Ken Hilton
On Thu, 2 Aug 2018 at 10:39, Ken Hilton <kenlhilton@gmail.com> wrote:
That use case is satisfied by pathlib: Path('file').read_text() see https://docs.python.org/3.7/library/pathlib.html#pathlib.Path.read_text Are there any other use cases? I don't see any real advantage here other than the non-advantage of being able to write one-liners. Paul
Is it true that Path('file').read_text() closes the file after the read? I think that is the sort of functionality that Ken is asking for. It's not clear to me by your linked documentation that it does. If it does, maybe that should be made more clear in that linked documentation? (Of course, maybe it's written there somewhere and I'm just blind...) Cheers, Thomas On 08/02/2018 11:53 AM, Paul Moore wrote:
On Thu, Aug 2, 2018 at 7:24 AM Thomas Nyberg via Python-ideas < python-ideas@python.org> wrote:
Agreed. Then again, the documentation includes a link to the source at the top and we find ( https://github.com/python/cpython/blob/3.7/Lib/pathlib.py#L1174) docstring: Open the file in text mode, read it, and close the file. Or ...
The documentation would be improved if it used the text from the docstring instead of its own one-line description. André Roberge
On Thu, 2 Aug 2018 at 11:25, Thomas Nyberg via Python-ideas <python-ideas@python.org> wrote:
I'm not sure I see why you think it wouldn't - opening and closing the file is a purely internal detail of the function. In any case, you don't get given a file object, so how could anything *other* than the read_text() close the file? So you're basically asking "does Path.read_text() have a bug that causes it to leak a filehandle?" to which my answer would be "I assume not, until someone demonstrates such a bug". But if someone wanted to raise a doc bug suggesting that we mention this, I'm not going to bother objecting... Paul
On 08/02/2018 12:43 PM, Paul Moore wrote:
To me the following look the same: Path('file').read_text() open('file').read() The first presumably creates a Path object while the second creates a file object. Why should I assume that the Path object closes the underlying file object after the method is called? I mean maybe my assumption is bad, but I doubt I'd be the only one making it given how open() works and that it looks similar superficially. Cheers, Thomas
On 08/02/2018 12:43 PM, Paul Moore wrote:
I opened a bug here: https://bugs.python.org/issue34319 We can see what others think. Cheers, Thomas
On 8/2/2018 7:53 AM, Thomas Nyberg via Python-ideas wrote:
I suggested on the issue that we add "The file is opened and then closed." before "The optional parameters have the same meaning as in open()." Another option would be to add "The file is closed after reading." after the latter sentence. -- Terry Jan Reedy
On Thu, Aug 2, 2018 at 5:24 AM Thomas Nyberg via Python-ideas <python-ideas@python.org> wrote:
Is it true that Path('file').read_text() closes the file after the read?
A quick look at the source confirms that the file is closed: https://github.com/python/cpython/blob/master/Lib/pathlib.py#L1174 The docstring is better than the official documentation, in my opinion. Cody
On Thu, Aug 02, 2018 at 11:35:11AM +0200, Ken Hilton wrote:
Perhaps so, but do we want to encourage that to the point of adding syntax to make it easier? f.read() is a (mild) code-smell. Unless your file is guaranteed to be smaller than the amount of free memory, it risks starting your OS thrashing. IMO that makes this an idiom only suitable for quick and dirty scripts where the the user knows the limitations of the script and can abide by them. (In these days of computers with multiple gigabytes of RAM, reading in an entire file is not as risky as it used to be. But on the other hand, in these days of terrabyte and even petabyte storage devices, there are more *really large* files too.) Don't get me wrong -- f.read() is not necessarily bad. I often write scripts that slurp in an entire file at once, but they're typically throw-away scripts, and I'm also the user of the script and I know not to call it on files above a certain size. (As Miss Piggy once said, "Never eat more in one sitting than you can lift.") But I'm not sure if this sort of thing is something we want to *encourage* rather than merely *allow*. Best practice for reading files is, after all, a with statement for a reason: we expect to read text files line by line, often wrapped in a try...except to handle exceptions. For your use-case, I suspect the best thing is a utility function: def read(name, size=-1, **kwargs): with open(name, **kwargs) as f: return f.read(size) Not every three-line function needs to be a built-in, let alone given syntax :-)
For those wondering about the scope semantics of the "as NAME", I think they would be identical to the scope semantics of the "for" expression
Its not really a "for expression" -- its a *comprehension*, which is much more than merely a for expression: # this isn't legal result = for x in seq One important difference is that unlike this proposed "with" expression, comprehensions have an obvious pair of delimiters which enclose the expression and give it a natural beginning and end. There's no need to memorise arcane operator precedences and parsing rules to work out where the "with...as" variable will be legal. Another important difference is that while there are good reasons for putting comprehension loop variables in their own sub-local scope, I don't see any such benefit to doing the same for this proposed with expression. I don't think we should encourage the proliferation of more and more layers of extra scopes. We already have six: sublocal (comprehensions) local nonlocal (enclosing functions) class global (module) builtins Let's be cautious about adding more varieties of sublocal scope. -- Steve
This brings the discussion of variable assignement in Expression. Functional programming community seems to be more interested in python. lines = (f.readlines() with open('hello') as f) digit = (int('hello') except ValueError: 5) value = (x+y**2 where x,y = (2,4)) values = [x+y**2 for x in range(5) for y in range(7)] values = [x+y**2 for x,y in product (range(5), range(7))] y = 5 if condition else 2 y = (lambda x: x+2)(x=5) vs with open('hello') as f: lines = f.readlines() del f # f is leaked ! x,y = 2,4 value = x+y**2 del x, y # x,y are leaked ! try: digit = (int('hello') except ValueError: digit = 5 if condition: y = 5 else: y = 2 def f(x): return x+2 y = f(x=2) del f # we want an anonymous function ! Those "oneliners" is not only the will to be quicker in interactive mode, it's the way functional programming Thinks. If we add one, it's Logical to add the others to be consistent. Of course, one can always write functions like read_text but the ide of those construction is like the lambda, we want anonymous. Le jeu. 2 août 2018 à 13:56, Steven D'Aprano <steve@pearwood.info> a écrit :
On Thu, Aug 02, 2018 at 03:13:25PM +0200, Robert Vanden Eynde wrote:
This brings the discussion of variable assignement in Expression. Functional programming community seems to be more interested in python.
I'm not sure what you mean there. Your English grammar is just slightly off, enough to make your meaning unclear, sorry.
lines = (f.readlines() with open('hello') as f)
readlines has the same problems as read, as I described earlier, and the same trivial three-line solution.
digit = (int('hello') except ValueError: 5)
try...except exceptions have been proposed before and rejected.
value = (x+y**2 where x,y = (2,4))
A "where" *statement* is interesting, but this is not a good example of it. The above is better written in the normal syntax: value = 2 + 4**2 no need to introduce temporary variables that exist only to obfuscate the code.
These already exist, because they are useful.
y = (lambda x: x+2)(x=5)
This is not a good example of the use of a lambda. Better: y = 5 + 2 Why bother writing a function with such a simple body if you are going to immediately call it on the spot? Unless the body is more complex, or you are going to call it elsewhere, or call it repeatedly, the lambda adds nothing. Nobody denies that *some* statements are well-suited and useful as expressions. The question is whether "with" is one of those.
99% of the time, I would think that "del f" was a waste of time. If that code is in function, then f will be closed when the "with" block is exited, and garbage collected when the function returns. If you are worried about the memory efficiency of one tiny closed file object, then Python is the wrong language for you. If that code is in the top-level of a script, who cares about f? You surely don't delete all your variables when you are done with them: name = input("what is your name?") print("Hello,", name) del name play_game() The only time I would explicitly delete f like you do above was if I was writing a library which specifically and explicitly supported the "from module import *" syntax, AND there were too many exportable names to list them all in __all__ by hand. Only in that very rare case would I care about tidying up the global namespace by using del.
If you are writing code like this, you are just obscuring the code. Much better to just use the values where you need them, not to invent unnecessary temporary variables that you don't need! value = 2 + 4**2 [...]
If we add one, it's Logical to add the others to be consistent.
My car has round wheels. Since we use one shape (circle) for wheels, it must be "logical" to make wheels in all other shapes, to be consistent: - triangular wheels - square wheels etc. Consistency just for the sake of consistency is *not* a virtue. Unless those expression forms justify *themselves* on their own merits, it is just a waste of time and effort to let them sneak into the language "for consistency".
Of course, one can always write functions like read_text but the ide of those construction is like the lambda, we want anonymous.
We can say: text = read('filename') text = f.read() with open('filename') as f and both are equally unanonymous (both use a named variable), or we can say: process(spam, eggs, read('filename'), foo, bar) process(spam, eggs, f.read with open('filename') as f, foo, bar) and both are equally anonymous. If Python had a built-in function "read", surely you wouldn't refuse to use it because it was a named function? We don't complain that map() and filter() are named functions and demand "anonymous" ways to do the same thing. A read function should be no different. The only point of difference is that it is not built-in, you have to write it yourself. But not every trivial three-line function must be built-in. -- Steve
Thanks for answering each line. If someone wants "too long didn't read", just check my code at the paragraph "readlines is a toy example, but maybe the code would be more creative". Le ven. 3 août 2018 à 03:07, Steven D'Aprano <steve@pearwood.info> a écrit :
sorry.
When I say "functional programming", I speak about the paradigm used in language like Haskell. In language like those, all constructs are "expression-based". I consider the code "result = 5 if condition else 2" more "functional style" than "if condition: result = 5; else: result = 2". Functional style focus on the result, uses expressions. Imperative focus on the process, "we must do a condition, then we set the variable, else, we set a variable to something else".
I'm wondering why, that must have been the same reasons of not accepting "with". if condition: something = x else: something = y Can be refactored something = x if condition else y Or something = (x if condition else y) But, try: something = x except: something = y Can't? The use cases seems similar. One can use the first form, using more of a imperative programming, or the second line, which is more "functional programming", more expressions oriented.
That's the discussion we had on the list called "variable assignement in expressions". What you did here is inlining the variables, technically it's not possible if you're calling a function and using the variable more than once. So we're really comparing it to : x,y = (2,4) value = x+y**2 Or x = 2 y = 4 value = x+y**2 Where the idea is to separate the steps of a computation, introducing temporary variables with a meaningful name is useful (And as mentioned, if a function is called and the variable reused, it's called once, but that's not the main point). In Haskell there is "value = (x = 2 in y = 4 in x+y**2)" or similar. position = initial + speed * time position_inch = distance / 2.54 Vs position_inch = (initial + speed * time) / 2.54 The programmer chooses what's more clear given the context and the audience. Or maybe he wants to emphasize that the code creates a position_inch variable usable in the code after ? Or both, he wants to explain how he computes position_inch using a temporary variable but doesn't want the rest of the code depend o "position" ?). Yes the del is generally useless, more about leaking below.
I see those as a refactoring of imperative programming style as well (values = []; for ...: values.append(...))
Same thing as in the where syntax. However, some constructs are easier to refactor as def meaningfulname(x): return x+2 y = meaningfulname(5)
Indeed, in complex expressions. Or if I want to separate the steps of a computation. position = initial + speed * time position_inch = distance / 2.54
Nobody denies that *some* statements are well-suited and useful as expressions. The question is whether "with" is one of those.
I'm just pointing out those constructs are very similar, it kind of makes sense to compare them. Of course I don't know about real world examples that would simplify a code. But often as I'm a "expression first" guy I write: result = ... # code using result and that doesn't care about the temporary variable it uses. Then I figure how to compute result, without polluting the namespace. Again adding temporary variables before "result = ..." Is totally fine and that's the way to go in imperative programming.
Yes of course, but what if "f" was defined before? We lose its value, even if "f" was created only as a temporary variable to have the variables lines. Maybe it came from: lines = ... # code using lines Yes naming it "f" where one has a 'f' in the same block is confusing, yes it could be named "_". But maybe we are in a script and we have a lots of variables? That kind of questions arise, when we wanted a temporary variable. readlines is a toy example, but maybe the code would be more creative, something like : if condition: filename = "session-{}-{}.txt".format(x, y*10) else: filename = "data_{}_{}.txt".format(w, z) with open('hello') as f: lines = [l.strip('\n') for l in f if len(l) > 0 if not l.stri().startswith('#')] parsed_data = [ ... # list expression mapping lines ] The extra indent seems a bit weird, opening a file is just a secondary effect of wanting to compute "lines = ..." filename = ("session-{}-{}.txt".format(x, y*10) if condition else "data_{}_{}.txt".format(w,z)) lines = ... # Implement that later parsed_data = [ ... # list expression mapping lines ] To: filename = ("session-{}-{}.txt".format(x, y*10) if ... else "data_{}_{}.txt".format(x, y)) lines = ([l.strip('\n') for l in f if len(l) > 0 if not l.stri().startswith('#')] with open (filename) as f) parsed_data = [ ... # list expression mapping lines ] The creator chose to keep exactly the three variables because they have a meaning, f is just an implementation detail. The creator wanted to emphasize that the code is simply: filename = ... lines = ... parsed_data = ... But the "..." being not so useful to have their own function. #Very verbose def compute_filename(x,y,w,z): ... def compute_lines(..., filename): ... def compute_parsed_data(..., lines, filename): ... filename = compute_filename(...) lines = compute_lines(...) parsed_data = compute_parsed_data(...) The functions there should be anonymous, but lambda invoked directly are not very readable.
If you are worried about the memory efficiency of one tiny closed file object, then Python is the wrong language for you.
Yes I don't talk about the "free" of C or "delete" of C++.
print("Hello,", name)
Yes that's a good example of me not wanting to "pollute the namespace, not wanting to create simple functions, not wanting to indent sometimes, but wanting to have tempory variables". If someone reads the module to see its content, they look for all the lines at zero indent begining with 'something = ...' or 'def ...'
Temporary variable has always been a choice to the programmer, they explain more the intermediary steps, but focus less on the final result.
I agree, that's why I said earlier "I didn't think so much about the use cases". But trust me, when I write code, I'm really often in "something = ..." Case. And sometimes the ... must depending on a try except, sometimes on a with, sometimes on other variables that are temporary.
Of course naming stuff is important (that's one of the reasons to build temporary variables). One difficultly of finding use cases, it that it's about changing the paradigm, probably all cases have a really readable implementation in current python / imperative style. But when I see: try: a_variable = int(input("...")) except ValueError: try: a_variable = fetch_db() except DbError: a_variable = default I really think "why can't I put it one one line like I do with if". a_variable = (int(input("...")) except ValueError: fetch_db() except DbError: default) For "with", I'm just wondering "why do I have to indent, it will lose the focus of the reader on the result itself".
On Fri, Aug 3, 2018 at 3:49 AM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
sure -- but Python is explicitly NOT a functional language in that sense. It does support some functional paradigms, but features are not added to conform to the functional paradigm per-se, but because they are considered useful features. So one needs to defend a new proposal with arguments about how it makes python programming better (by SOME definition of better) on its own merits. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Aug 03, 2018 at 12:49:24PM +0200, Robert Vanden Eynde wrote:
I know what functional programming is. What I don't understand is what you mean when you say that the F.P. community "seems to be more interested in python". Surely they are more interested in functional languages than a multi-paradigm language like Python which does not privilege functional idioms over imperative idioms. [...]
Read the PEP. Or the (long!) discussions on Python-Ideas and Python-Dev. https://www.python.org/dev/peps/pep-0463/
Which is why I said it was not a good example. If you're going to propose syntax, you ought to give good examples, not bad examples. In any case, this is not a proposal for a "where" expression. You aren't going to convince people to add a "with" expression by showing them expression forms of "if", "for" or hypothetical "where". Each feature must justify itself, not sneak in behind another feature. [...]
Then don't use f. Use F, fi, fp, fileobj, f_, f1, ϕ, фп, or myfileobject. Surely you haven't used them all. There is no shortage of useful identifier names. Or factor your code into named functions with isolated namespaces, so the f in one function doesn't clobber the f in another function. Structured programming won the debate against unstructured programming a long time ago. https://en.wikipedia.org/wiki/Structured_programming#History [...]
Whereas when I see somebody using a double if...else expression in a single line, I wish they would spread it out over multiple lines so it is readable and understandable, instead of trying to squeeze the maximum amount of code per line they can.
For "with", I'm just wondering "why do I have to indent, it will lose the focus of the reader on the result itself".
If this is a problem, then you can easily focus the reader on the result by factoring the code into a named function. text = read('filename') requires no indentation, no complicated new syntax, and it is easily extensible to more complex examples: text = (prefix + (read('filename') or 'default') + moretext).upper() This argument isn't about whether it might occasionally be useful to use a with expression, but whether it is useful *enough* to justify adding new syntax to the language. -- Steve
Just a personal feeling, it's not really thought out.
I'm reading it. The prelude at the beginning then says there are no convincing use case if I get it right?
Which is why I said it was not a good example. If you're going to propose syntax, you ought to give good examples, not bad examples.
When showing toy examples I thought some people would think "indeed, that happens to me often".
Indeed, I'm talking about it because I think it all relates to "expressionalize simple statements", that's also why I speak about FP, because that's an "expression-first" paradigm.
Of course I would do that, i completely agree that refactoring is useful, but as shown in the end of my post: filename = compute_filename(...) lines = compute_lines(...) parsed_data = compute_parsed_data(...) The functions body being a bit too small to be refactored and doesn't really have another meaning of "code to compute filename", I feel like the "filename =" already catches the idea, I feel like repeating myself (DRY). And the function body in those cases is not very reusable so it doesn't make sense to give it a meaningful name. I would do a named function otherwise indeed.
Multiline is clearly better yes. When I say 'one line' I'm just saying "using the expression-statement but probably gonna span it on multiple lines because the Expressions are not short.
For 'with', as you all say, it really boils down to finding use cases other than "file manipulation". It's true I can't think of any (expect the fact it does create a function that may not have a meaningful name).
I like this idea in theory, but I'm not sold yet. I think there's a lot of draw to the concept of "expressionizing" statements because many statements require an unnatural ordering in-which the most important code, the logic, comes after some necessary but ultimately noisy (from the readers perspective) preamble. So I expect people to keep asking for expressionized statements and slowly, but, surely, they'll make their way into the language. They just need to be very carefully thought out. Expressionization may break the "one and only on obvious way" guideline, but it can offer concise, readable code in a lot of instances where a statement-based version would be clumsy and noisy, and there's already some precedent for it: function declaration => lambda for-loops => generator expressions and comprehensions if-else => ternary statements With the exception of lambda, expressionized statements usually allow one to put the "meat before the vegitables" so to speak. That is; the highest value part of the expression comes first and all the book-keeping follows. To illustrate this, I'll write a part of an expression and gradually reveal the complete expression, you can see how progressively easier it is to predict the next reveal: def initials(people): return {"".join(name[0] ... # The identifier "name" isn't in scope so it must be assigned in the for clause of a comprehension. def initials(people): return {"".join(name[0] for name in ... # This is a nested comprehension so it's not much of a surprise that the iterator might be related to another # yet-to-be assigned identifier. def initials(people): return {"".join(name[0] for name in person.names ... # Blindly accessing the first element of an empty sequence could cause problems def initials(people): return {"".join(name[0] for name in person.names if name) ... # The inner generator is closed but we still need a binding for "person" def initials(people): return {"".join(name[0] for name in person.names if name) for person in ... # There's not much left to iterate over and decent variable names point to one obvious choice def initials(people): return {"".join(name[0] for name in person.names if name) for person in people} The same could be said for lambdas if they were defined logic-first because they're usually used in a context where the call signature is obvious: hand = sorted(cards, key=(card.suit if card is not wild else max_value <== card))[-5:] Of course, no such thing exists so the '<==' syntax is made up (in-fact a possibly better alternative is, "with"), but it doesn't really matter because a reverse lambda isn't going to happen. You can see, however; that the function's signature is pretty obvious from context, so it's more for the computer's sake than the reader's sake and would be better placed out of the way. I like the current proposal because it follows that design idea, however; I haven't taken the time to think about all the edge cases yet. For instance, what would the following do? initial = person.name[0] with suppress(AttributeError) # Hangover from PEP 505 discussion... Now that I think of it, this seems to inherently make assignment part of the expression: data = file.read() with open(...) as file is supposed to be equivalent to: with open(...) as file: data = file.read() Right? So maybe it only makes sense to use expression assignment (PEP 572): data = (d := file.read() with open(...) as file) To which I say, "Eww..." Also: initial = (i := person.name[0] with suppress(AttributeError)) Is still ambiguous (and still eww). One tactic that other expressionizations have taken is to limit the scope. For instance, the ternary operation only covers expressionization of "if-else" not "just if" or "if-elif-..." or "if-elif-...-else", and generator expressions don't allow the 'else' clause <http://book.pythontips.com/en/latest/for_-_else.html> of normal for-loops. So maybe you can obviate some of the edge cases by requiring an as clause or something. I don't know how that would help with the suppress(AttributeError) case thought... On Fri, Aug 3, 2018 at 12:56 PM, Todd <toddrjen@gmail.com> wrote:
Totally agree. That's the problem when being multi paradigm, but we already have that "problem" and that's alright.
As the code I showed, being just that: filename = ... lines = ... parsed_data = ... With implementation details? The highest value is there, the alg is clear., one fetches the filename, one preprocess the lines, then parse the data.
About if elif elif else, ternary if does have that: y = (x+1 if x < 0 else x-1 if x > 0 else 0) Limiting the scope is interesting, for "with" the only limitation in that the body must have exactly one assignment, like in the ternary if case? a = ([l.strip() for l in f.readlines()] with open ('name') as f) By cutting the edge cases, "with something():" is not possible, only "with ... as" being possible? But the "scope limitation" for try except in expression concept would be not to have "else" or "finally"? The else and finally clauses do not make sense in Expression style assignment anyway. a = (int('stuff') except ValueError: 5)
On Sat, Aug 04, 2018 at 12:28:40AM +0200, Robert Vanden Eynde wrote:
As programmers, surely we have to care about implementation details of the code we are maintaining or writing. If we don't care about implementation, who does? Of course there is a tension between having to deal with implementation details too earlier, or hiding them too deeply. We have to balance too little refactoring from too much refactoring, and people can legitimately disagree as to when a function, method or class carries its own weight. The same applies to syntactic features. That's why *concrete use-cases* are important, not pretend, made-up toy examples. The Python community as a whole is not very receptive to arguments from functional programming purity. I know Python veterans who still don't like list comprehensions (although they're a minority). As a whole, the community does not believe that using multiple statements is always a problem to be fixed. The question is not whether it is *possible* to have a with expression, or whether we might find some toy examples that kinda look good, but how much it improves *real code*. And arguments that we should have a "with" expression because we already have "if" expressions and comprehensions will just fall flat. Arguments by consistency ("we have A, so we ought to have E too, because they're both vowels") are not very productive. -- Steve
with supress(AttributeError): tmp = person.name[0] initial = tmp # Error on assignment wouldn't get suppressed. Not relevant for this case but still. I don't understand where this is ambigous?
data = file.read with open(...) as file works perfectly fine so why would you want to additonaly assign it to another variable "d"? Overall I like the idea of the with-expression as it allows you to make some common use cases like the open/read example more readable. It's clear at first sight that what is actually done is reading the file. While it's true that it's usually easy to write a simple function to do the job that still isn't as simple to understand and in most cases when reading the code you then have to look at the function to see if anything else is done there. Also what if you then do readlines somewhere? Then you need another function.
[Benedikt Werner]
Ah, yes. That makes much more sense now. I would say the error on assignment IS relevant for that case and it's not clear to me how you would present that error to the user without causing a lot of confusion. If it were implemented as: tmp = None with suppress(AttributeError): tmp = person.name[0] initial = tmp Then it would solve many of the common use cases for the None-aware operators proposed in PEP 505, especially if we made it easy to filter out None-specific errors: class NoneError(BaseException): pass class NoneAttributeError(AttributeError, NoneError): pass ... code to make erroneous attribute access on NoneType objects throw NoneAttributeErrors ... ... shouldn't break much code other than maybe some doc-string tests ... initial = person.name[0] with suppress(NoneError) [Robert Vanden Eynde]
Yes. Sorry for using confusing language. I was trying to say that I like your proposed syntax (for both with and except) because it follows the same principal. At the same time, I was trying to demonstrate part of the value of expression-ized statements to everyone else. Since Python is strictly evaluated statement by statement, there is no notion of looking ahead and re-ordering code. You can't use a variable in one statement then define that variable later on (like you do in comprehensions). Expressions, on the other hand; are parsed in a more complex way, so you can do things like put the relevant logic up front and move all the book-keeping to the end. [Robert Vanden Eynde]
True, and your except syntax could chain in a similar manner. In-fact it might be possible to make a separate "finally" operator that executes the left side then executes the right side no matter what: y = ((x[0] except InxexError: x.default) except AttributeError: None) finally print("hello!") [Robert Vanden Eynde]
I think it's better to forget what I said about limiting the scope of the operator. It was a half-baked thought. As for assignment expressions, they're already on their way thanks to PEP 572. I like the "where" version you proposed because it places the logic first, but unfortunately PEP 572 (4th alternate spelling <https://www.python.org/dev/peps/pep-0572/#alternative-spellings>) states that they had considered that version and rejected it because "where" is a pretty common variable name. From the PEP: SQLAlchemy and numpy have where methods, as does tkinter.dnd.Icon in the
standard library
So adding a new "where" keyword would break backwards compatibility in a major way. On Fri, Aug 3, 2018 at 5:40 PM, Benedikt Werner <1benediktwerner@gmail.com> wrote:
One last thing: Since expressions tend to have pretty limited space, it might be worth it to consider more concise options like maybe instead of: <expression1> except (Exception[-list])[ as e]): <expression2> it could be: <expression1> (Exception[-list])[ as e])! <expression2> So: y = x[0] IndexError! default instead of y = x[0] except IndexError: default or: y = spam.ham[0] (AttributeError, IndexError) as e! foo(e) instead of y = spam.ham[0] except (AttributeError, IndexError) as e: foo(e) Or something to that effect. (thought I think it might be best to err on the side of 'except' because it's more clear what it does and it only adds 7 characters). On Fri, Aug 3, 2018 at 6:57 PM, Abe Dillon <abedillon@gmail.com> wrote:
On Fri, Aug 3, 2018 at 5:26 PM Abe Dillon <abedillon@gmail.com> wrote:
You're reinventing PEP 463: https://www.python.org/dev/peps/pep-0463/
On Fri, Aug 03, 2018 at 06:57:40PM -0500, Abe Dillon wrote:
No it would not. The None-aware operators are not about suppressing AttributeError, please stop suggesting that it is. If you want to propose a general exception suppressing mechanism (aside from the existing try...except statement) then propose it as an independent PEP. Even if we had a general purpose exception-suppressing expression, it wouldn't meet the functional requirements for PEP 505. It would do too much, like offering somebody a bulldozer when all we want is a dustpan and broom.
They're not None specific. Any object can raise them. -- Steve
On Sat, Aug 04, 2018 at 12:40:13AM +0200, Benedikt Werner wrote:
Overall I like the idea of the with-expression as it allows you to make some common use cases like the open/read example more readable.
For some definition of "readable".
It's clear at first sight that what is actually done is reading the file.
Only because the file.read() comes early in the expression. But that applies equally to text = read('filename')
While it's true that it's usually easy to write a simple function to do the job that still isn't as simple to understand
(1) read(filename) Explanation to a beginner: "It reads text from the named file." (2) f.read() with open(filename) as f Explanation to a beginner: "It opens a file -- I'll explain what it means to open a file later -- in a context manager -- that's advanced Python programming, don't worry about context managers for now -- and binds the context manager to the name f, then calls the read method on the object f -- I'll explain object oriented programming later -- which reads text from the named file and closes the file... oh I forgot to mention that the context manager is also a file object." Do you still think that explaining a with expression is simpler than explaining a self-descriptively named function? The beauty of the named function is that it hides a lot of irrelevant detail and focuses *only* on the important feature, which is reading from the file. A with-statement is great for when you care about the implementation details. Somebody has to care about the process of opening a file, reading from it and closing it. But when you *don't* care about those implementation details, a simple interface like a read() function is superior to a with-statement, *or* a with-expression, which shoves those details in your face.
I doubt that many people really spend a lot of time digging into functions to see whether they do more than what they say. Unless and until I ran into unexpected problems, I'd be no more inclined to look into a function called "read" than I would be to do the same to len or math.sin. I'm sure it does what it says it does. In Python 2, one of the problems with the input() function was that it was used by people who didn't read the docs and where taken completely by surprise by the fact that it called eval on the input. That shows that people don't make a habit of digging into functions "just in case".
Also what if you then do readlines somewhere? Then you need another function.
Indeed. But not everything has to be a built-in feature, and refactoring common code into a named function will often be a good idea *even if* ``with`` is an exception: def read(filename, size=-1, **kwargs): return f.read(size) with open(filename, **kwargs) as f -- Steve
Totally agree. In the case where you care about the implementation details. You can explain to a beginner: something = (f.read() with open('...') as f) Is another way to write: with open('filename') as f: something = f.read() Exactly like the ternary if. And I agree that because with is often relevant for "implementation details" or beginning use as a "enter.. exit' statement (like a cpp destructor), it matches more imperative programming. with stack.push_matrix(Scale(4)): triangle = draw_triangle_using(stack) Is used for it's "enter/close" functionality, but can still be used like: triangle = (draw_triangle_using(stack) with stack.push_matrix(Scale(4)) as NotUsed) But the "as close" is useless here.
Fair point, the "def" statements generally are passed when read so they don't really count as overhead.
On Thu, 2 Aug 2018 at 10:39, Ken Hilton <kenlhilton@gmail.com> wrote:
That use case is satisfied by pathlib: Path('file').read_text() see https://docs.python.org/3.7/library/pathlib.html#pathlib.Path.read_text Are there any other use cases? I don't see any real advantage here other than the non-advantage of being able to write one-liners. Paul
Is it true that Path('file').read_text() closes the file after the read? I think that is the sort of functionality that Ken is asking for. It's not clear to me by your linked documentation that it does. If it does, maybe that should be made more clear in that linked documentation? (Of course, maybe it's written there somewhere and I'm just blind...) Cheers, Thomas On 08/02/2018 11:53 AM, Paul Moore wrote:
On Thu, Aug 2, 2018 at 7:24 AM Thomas Nyberg via Python-ideas < python-ideas@python.org> wrote:
Agreed. Then again, the documentation includes a link to the source at the top and we find ( https://github.com/python/cpython/blob/3.7/Lib/pathlib.py#L1174) docstring: Open the file in text mode, read it, and close the file. Or ...
The documentation would be improved if it used the text from the docstring instead of its own one-line description. André Roberge
On Thu, 2 Aug 2018 at 11:25, Thomas Nyberg via Python-ideas <python-ideas@python.org> wrote:
I'm not sure I see why you think it wouldn't - opening and closing the file is a purely internal detail of the function. In any case, you don't get given a file object, so how could anything *other* than the read_text() close the file? So you're basically asking "does Path.read_text() have a bug that causes it to leak a filehandle?" to which my answer would be "I assume not, until someone demonstrates such a bug". But if someone wanted to raise a doc bug suggesting that we mention this, I'm not going to bother objecting... Paul
On 08/02/2018 12:43 PM, Paul Moore wrote:
To me the following look the same: Path('file').read_text() open('file').read() The first presumably creates a Path object while the second creates a file object. Why should I assume that the Path object closes the underlying file object after the method is called? I mean maybe my assumption is bad, but I doubt I'd be the only one making it given how open() works and that it looks similar superficially. Cheers, Thomas
On 08/02/2018 12:43 PM, Paul Moore wrote:
I opened a bug here: https://bugs.python.org/issue34319 We can see what others think. Cheers, Thomas
On 8/2/2018 7:53 AM, Thomas Nyberg via Python-ideas wrote:
I suggested on the issue that we add "The file is opened and then closed." before "The optional parameters have the same meaning as in open()." Another option would be to add "The file is closed after reading." after the latter sentence. -- Terry Jan Reedy
On Thu, Aug 2, 2018 at 5:24 AM Thomas Nyberg via Python-ideas <python-ideas@python.org> wrote:
Is it true that Path('file').read_text() closes the file after the read?
A quick look at the source confirms that the file is closed: https://github.com/python/cpython/blob/master/Lib/pathlib.py#L1174 The docstring is better than the official documentation, in my opinion. Cody
On Thu, Aug 02, 2018 at 11:35:11AM +0200, Ken Hilton wrote:
Perhaps so, but do we want to encourage that to the point of adding syntax to make it easier? f.read() is a (mild) code-smell. Unless your file is guaranteed to be smaller than the amount of free memory, it risks starting your OS thrashing. IMO that makes this an idiom only suitable for quick and dirty scripts where the the user knows the limitations of the script and can abide by them. (In these days of computers with multiple gigabytes of RAM, reading in an entire file is not as risky as it used to be. But on the other hand, in these days of terrabyte and even petabyte storage devices, there are more *really large* files too.) Don't get me wrong -- f.read() is not necessarily bad. I often write scripts that slurp in an entire file at once, but they're typically throw-away scripts, and I'm also the user of the script and I know not to call it on files above a certain size. (As Miss Piggy once said, "Never eat more in one sitting than you can lift.") But I'm not sure if this sort of thing is something we want to *encourage* rather than merely *allow*. Best practice for reading files is, after all, a with statement for a reason: we expect to read text files line by line, often wrapped in a try...except to handle exceptions. For your use-case, I suspect the best thing is a utility function: def read(name, size=-1, **kwargs): with open(name, **kwargs) as f: return f.read(size) Not every three-line function needs to be a built-in, let alone given syntax :-)
For those wondering about the scope semantics of the "as NAME", I think they would be identical to the scope semantics of the "for" expression
Its not really a "for expression" -- its a *comprehension*, which is much more than merely a for expression: # this isn't legal result = for x in seq One important difference is that unlike this proposed "with" expression, comprehensions have an obvious pair of delimiters which enclose the expression and give it a natural beginning and end. There's no need to memorise arcane operator precedences and parsing rules to work out where the "with...as" variable will be legal. Another important difference is that while there are good reasons for putting comprehension loop variables in their own sub-local scope, I don't see any such benefit to doing the same for this proposed with expression. I don't think we should encourage the proliferation of more and more layers of extra scopes. We already have six: sublocal (comprehensions) local nonlocal (enclosing functions) class global (module) builtins Let's be cautious about adding more varieties of sublocal scope. -- Steve
This brings the discussion of variable assignement in Expression. Functional programming community seems to be more interested in python. lines = (f.readlines() with open('hello') as f) digit = (int('hello') except ValueError: 5) value = (x+y**2 where x,y = (2,4)) values = [x+y**2 for x in range(5) for y in range(7)] values = [x+y**2 for x,y in product (range(5), range(7))] y = 5 if condition else 2 y = (lambda x: x+2)(x=5) vs with open('hello') as f: lines = f.readlines() del f # f is leaked ! x,y = 2,4 value = x+y**2 del x, y # x,y are leaked ! try: digit = (int('hello') except ValueError: digit = 5 if condition: y = 5 else: y = 2 def f(x): return x+2 y = f(x=2) del f # we want an anonymous function ! Those "oneliners" is not only the will to be quicker in interactive mode, it's the way functional programming Thinks. If we add one, it's Logical to add the others to be consistent. Of course, one can always write functions like read_text but the ide of those construction is like the lambda, we want anonymous. Le jeu. 2 août 2018 à 13:56, Steven D'Aprano <steve@pearwood.info> a écrit :
On Thu, Aug 02, 2018 at 03:13:25PM +0200, Robert Vanden Eynde wrote:
This brings the discussion of variable assignement in Expression. Functional programming community seems to be more interested in python.
I'm not sure what you mean there. Your English grammar is just slightly off, enough to make your meaning unclear, sorry.
lines = (f.readlines() with open('hello') as f)
readlines has the same problems as read, as I described earlier, and the same trivial three-line solution.
digit = (int('hello') except ValueError: 5)
try...except exceptions have been proposed before and rejected.
value = (x+y**2 where x,y = (2,4))
A "where" *statement* is interesting, but this is not a good example of it. The above is better written in the normal syntax: value = 2 + 4**2 no need to introduce temporary variables that exist only to obfuscate the code.
These already exist, because they are useful.
y = (lambda x: x+2)(x=5)
This is not a good example of the use of a lambda. Better: y = 5 + 2 Why bother writing a function with such a simple body if you are going to immediately call it on the spot? Unless the body is more complex, or you are going to call it elsewhere, or call it repeatedly, the lambda adds nothing. Nobody denies that *some* statements are well-suited and useful as expressions. The question is whether "with" is one of those.
99% of the time, I would think that "del f" was a waste of time. If that code is in function, then f will be closed when the "with" block is exited, and garbage collected when the function returns. If you are worried about the memory efficiency of one tiny closed file object, then Python is the wrong language for you. If that code is in the top-level of a script, who cares about f? You surely don't delete all your variables when you are done with them: name = input("what is your name?") print("Hello,", name) del name play_game() The only time I would explicitly delete f like you do above was if I was writing a library which specifically and explicitly supported the "from module import *" syntax, AND there were too many exportable names to list them all in __all__ by hand. Only in that very rare case would I care about tidying up the global namespace by using del.
If you are writing code like this, you are just obscuring the code. Much better to just use the values where you need them, not to invent unnecessary temporary variables that you don't need! value = 2 + 4**2 [...]
If we add one, it's Logical to add the others to be consistent.
My car has round wheels. Since we use one shape (circle) for wheels, it must be "logical" to make wheels in all other shapes, to be consistent: - triangular wheels - square wheels etc. Consistency just for the sake of consistency is *not* a virtue. Unless those expression forms justify *themselves* on their own merits, it is just a waste of time and effort to let them sneak into the language "for consistency".
Of course, one can always write functions like read_text but the ide of those construction is like the lambda, we want anonymous.
We can say: text = read('filename') text = f.read() with open('filename') as f and both are equally unanonymous (both use a named variable), or we can say: process(spam, eggs, read('filename'), foo, bar) process(spam, eggs, f.read with open('filename') as f, foo, bar) and both are equally anonymous. If Python had a built-in function "read", surely you wouldn't refuse to use it because it was a named function? We don't complain that map() and filter() are named functions and demand "anonymous" ways to do the same thing. A read function should be no different. The only point of difference is that it is not built-in, you have to write it yourself. But not every trivial three-line function must be built-in. -- Steve
Thanks for answering each line. If someone wants "too long didn't read", just check my code at the paragraph "readlines is a toy example, but maybe the code would be more creative". Le ven. 3 août 2018 à 03:07, Steven D'Aprano <steve@pearwood.info> a écrit :
sorry.
When I say "functional programming", I speak about the paradigm used in language like Haskell. In language like those, all constructs are "expression-based". I consider the code "result = 5 if condition else 2" more "functional style" than "if condition: result = 5; else: result = 2". Functional style focus on the result, uses expressions. Imperative focus on the process, "we must do a condition, then we set the variable, else, we set a variable to something else".
I'm wondering why, that must have been the same reasons of not accepting "with". if condition: something = x else: something = y Can be refactored something = x if condition else y Or something = (x if condition else y) But, try: something = x except: something = y Can't? The use cases seems similar. One can use the first form, using more of a imperative programming, or the second line, which is more "functional programming", more expressions oriented.
That's the discussion we had on the list called "variable assignement in expressions". What you did here is inlining the variables, technically it's not possible if you're calling a function and using the variable more than once. So we're really comparing it to : x,y = (2,4) value = x+y**2 Or x = 2 y = 4 value = x+y**2 Where the idea is to separate the steps of a computation, introducing temporary variables with a meaningful name is useful (And as mentioned, if a function is called and the variable reused, it's called once, but that's not the main point). In Haskell there is "value = (x = 2 in y = 4 in x+y**2)" or similar. position = initial + speed * time position_inch = distance / 2.54 Vs position_inch = (initial + speed * time) / 2.54 The programmer chooses what's more clear given the context and the audience. Or maybe he wants to emphasize that the code creates a position_inch variable usable in the code after ? Or both, he wants to explain how he computes position_inch using a temporary variable but doesn't want the rest of the code depend o "position" ?). Yes the del is generally useless, more about leaking below.
I see those as a refactoring of imperative programming style as well (values = []; for ...: values.append(...))
Same thing as in the where syntax. However, some constructs are easier to refactor as def meaningfulname(x): return x+2 y = meaningfulname(5)
Indeed, in complex expressions. Or if I want to separate the steps of a computation. position = initial + speed * time position_inch = distance / 2.54
Nobody denies that *some* statements are well-suited and useful as expressions. The question is whether "with" is one of those.
I'm just pointing out those constructs are very similar, it kind of makes sense to compare them. Of course I don't know about real world examples that would simplify a code. But often as I'm a "expression first" guy I write: result = ... # code using result and that doesn't care about the temporary variable it uses. Then I figure how to compute result, without polluting the namespace. Again adding temporary variables before "result = ..." Is totally fine and that's the way to go in imperative programming.
Yes of course, but what if "f" was defined before? We lose its value, even if "f" was created only as a temporary variable to have the variables lines. Maybe it came from: lines = ... # code using lines Yes naming it "f" where one has a 'f' in the same block is confusing, yes it could be named "_". But maybe we are in a script and we have a lots of variables? That kind of questions arise, when we wanted a temporary variable. readlines is a toy example, but maybe the code would be more creative, something like : if condition: filename = "session-{}-{}.txt".format(x, y*10) else: filename = "data_{}_{}.txt".format(w, z) with open('hello') as f: lines = [l.strip('\n') for l in f if len(l) > 0 if not l.stri().startswith('#')] parsed_data = [ ... # list expression mapping lines ] The extra indent seems a bit weird, opening a file is just a secondary effect of wanting to compute "lines = ..." filename = ("session-{}-{}.txt".format(x, y*10) if condition else "data_{}_{}.txt".format(w,z)) lines = ... # Implement that later parsed_data = [ ... # list expression mapping lines ] To: filename = ("session-{}-{}.txt".format(x, y*10) if ... else "data_{}_{}.txt".format(x, y)) lines = ([l.strip('\n') for l in f if len(l) > 0 if not l.stri().startswith('#')] with open (filename) as f) parsed_data = [ ... # list expression mapping lines ] The creator chose to keep exactly the three variables because they have a meaning, f is just an implementation detail. The creator wanted to emphasize that the code is simply: filename = ... lines = ... parsed_data = ... But the "..." being not so useful to have their own function. #Very verbose def compute_filename(x,y,w,z): ... def compute_lines(..., filename): ... def compute_parsed_data(..., lines, filename): ... filename = compute_filename(...) lines = compute_lines(...) parsed_data = compute_parsed_data(...) The functions there should be anonymous, but lambda invoked directly are not very readable.
If you are worried about the memory efficiency of one tiny closed file object, then Python is the wrong language for you.
Yes I don't talk about the "free" of C or "delete" of C++.
print("Hello,", name)
Yes that's a good example of me not wanting to "pollute the namespace, not wanting to create simple functions, not wanting to indent sometimes, but wanting to have tempory variables". If someone reads the module to see its content, they look for all the lines at zero indent begining with 'something = ...' or 'def ...'
Temporary variable has always been a choice to the programmer, they explain more the intermediary steps, but focus less on the final result.
I agree, that's why I said earlier "I didn't think so much about the use cases". But trust me, when I write code, I'm really often in "something = ..." Case. And sometimes the ... must depending on a try except, sometimes on a with, sometimes on other variables that are temporary.
Of course naming stuff is important (that's one of the reasons to build temporary variables). One difficultly of finding use cases, it that it's about changing the paradigm, probably all cases have a really readable implementation in current python / imperative style. But when I see: try: a_variable = int(input("...")) except ValueError: try: a_variable = fetch_db() except DbError: a_variable = default I really think "why can't I put it one one line like I do with if". a_variable = (int(input("...")) except ValueError: fetch_db() except DbError: default) For "with", I'm just wondering "why do I have to indent, it will lose the focus of the reader on the result itself".
On Fri, Aug 3, 2018 at 3:49 AM, Robert Vanden Eynde <robertve92@gmail.com> wrote:
sure -- but Python is explicitly NOT a functional language in that sense. It does support some functional paradigms, but features are not added to conform to the functional paradigm per-se, but because they are considered useful features. So one needs to defend a new proposal with arguments about how it makes python programming better (by SOME definition of better) on its own merits. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Aug 03, 2018 at 12:49:24PM +0200, Robert Vanden Eynde wrote:
I know what functional programming is. What I don't understand is what you mean when you say that the F.P. community "seems to be more interested in python". Surely they are more interested in functional languages than a multi-paradigm language like Python which does not privilege functional idioms over imperative idioms. [...]
Read the PEP. Or the (long!) discussions on Python-Ideas and Python-Dev. https://www.python.org/dev/peps/pep-0463/
Which is why I said it was not a good example. If you're going to propose syntax, you ought to give good examples, not bad examples. In any case, this is not a proposal for a "where" expression. You aren't going to convince people to add a "with" expression by showing them expression forms of "if", "for" or hypothetical "where". Each feature must justify itself, not sneak in behind another feature. [...]
Then don't use f. Use F, fi, fp, fileobj, f_, f1, ϕ, фп, or myfileobject. Surely you haven't used them all. There is no shortage of useful identifier names. Or factor your code into named functions with isolated namespaces, so the f in one function doesn't clobber the f in another function. Structured programming won the debate against unstructured programming a long time ago. https://en.wikipedia.org/wiki/Structured_programming#History [...]
Whereas when I see somebody using a double if...else expression in a single line, I wish they would spread it out over multiple lines so it is readable and understandable, instead of trying to squeeze the maximum amount of code per line they can.
For "with", I'm just wondering "why do I have to indent, it will lose the focus of the reader on the result itself".
If this is a problem, then you can easily focus the reader on the result by factoring the code into a named function. text = read('filename') requires no indentation, no complicated new syntax, and it is easily extensible to more complex examples: text = (prefix + (read('filename') or 'default') + moretext).upper() This argument isn't about whether it might occasionally be useful to use a with expression, but whether it is useful *enough* to justify adding new syntax to the language. -- Steve
Just a personal feeling, it's not really thought out.
I'm reading it. The prelude at the beginning then says there are no convincing use case if I get it right?
Which is why I said it was not a good example. If you're going to propose syntax, you ought to give good examples, not bad examples.
When showing toy examples I thought some people would think "indeed, that happens to me often".
Indeed, I'm talking about it because I think it all relates to "expressionalize simple statements", that's also why I speak about FP, because that's an "expression-first" paradigm.
Of course I would do that, i completely agree that refactoring is useful, but as shown in the end of my post: filename = compute_filename(...) lines = compute_lines(...) parsed_data = compute_parsed_data(...) The functions body being a bit too small to be refactored and doesn't really have another meaning of "code to compute filename", I feel like the "filename =" already catches the idea, I feel like repeating myself (DRY). And the function body in those cases is not very reusable so it doesn't make sense to give it a meaningful name. I would do a named function otherwise indeed.
Multiline is clearly better yes. When I say 'one line' I'm just saying "using the expression-statement but probably gonna span it on multiple lines because the Expressions are not short.
For 'with', as you all say, it really boils down to finding use cases other than "file manipulation". It's true I can't think of any (expect the fact it does create a function that may not have a meaningful name).
I like this idea in theory, but I'm not sold yet. I think there's a lot of draw to the concept of "expressionizing" statements because many statements require an unnatural ordering in-which the most important code, the logic, comes after some necessary but ultimately noisy (from the readers perspective) preamble. So I expect people to keep asking for expressionized statements and slowly, but, surely, they'll make their way into the language. They just need to be very carefully thought out. Expressionization may break the "one and only on obvious way" guideline, but it can offer concise, readable code in a lot of instances where a statement-based version would be clumsy and noisy, and there's already some precedent for it: function declaration => lambda for-loops => generator expressions and comprehensions if-else => ternary statements With the exception of lambda, expressionized statements usually allow one to put the "meat before the vegitables" so to speak. That is; the highest value part of the expression comes first and all the book-keeping follows. To illustrate this, I'll write a part of an expression and gradually reveal the complete expression, you can see how progressively easier it is to predict the next reveal: def initials(people): return {"".join(name[0] ... # The identifier "name" isn't in scope so it must be assigned in the for clause of a comprehension. def initials(people): return {"".join(name[0] for name in ... # This is a nested comprehension so it's not much of a surprise that the iterator might be related to another # yet-to-be assigned identifier. def initials(people): return {"".join(name[0] for name in person.names ... # Blindly accessing the first element of an empty sequence could cause problems def initials(people): return {"".join(name[0] for name in person.names if name) ... # The inner generator is closed but we still need a binding for "person" def initials(people): return {"".join(name[0] for name in person.names if name) for person in ... # There's not much left to iterate over and decent variable names point to one obvious choice def initials(people): return {"".join(name[0] for name in person.names if name) for person in people} The same could be said for lambdas if they were defined logic-first because they're usually used in a context where the call signature is obvious: hand = sorted(cards, key=(card.suit if card is not wild else max_value <== card))[-5:] Of course, no such thing exists so the '<==' syntax is made up (in-fact a possibly better alternative is, "with"), but it doesn't really matter because a reverse lambda isn't going to happen. You can see, however; that the function's signature is pretty obvious from context, so it's more for the computer's sake than the reader's sake and would be better placed out of the way. I like the current proposal because it follows that design idea, however; I haven't taken the time to think about all the edge cases yet. For instance, what would the following do? initial = person.name[0] with suppress(AttributeError) # Hangover from PEP 505 discussion... Now that I think of it, this seems to inherently make assignment part of the expression: data = file.read() with open(...) as file is supposed to be equivalent to: with open(...) as file: data = file.read() Right? So maybe it only makes sense to use expression assignment (PEP 572): data = (d := file.read() with open(...) as file) To which I say, "Eww..." Also: initial = (i := person.name[0] with suppress(AttributeError)) Is still ambiguous (and still eww). One tactic that other expressionizations have taken is to limit the scope. For instance, the ternary operation only covers expressionization of "if-else" not "just if" or "if-elif-..." or "if-elif-...-else", and generator expressions don't allow the 'else' clause <http://book.pythontips.com/en/latest/for_-_else.html> of normal for-loops. So maybe you can obviate some of the edge cases by requiring an as clause or something. I don't know how that would help with the suppress(AttributeError) case thought... On Fri, Aug 3, 2018 at 12:56 PM, Todd <toddrjen@gmail.com> wrote:
Totally agree. That's the problem when being multi paradigm, but we already have that "problem" and that's alright.
As the code I showed, being just that: filename = ... lines = ... parsed_data = ... With implementation details? The highest value is there, the alg is clear., one fetches the filename, one preprocess the lines, then parse the data.
About if elif elif else, ternary if does have that: y = (x+1 if x < 0 else x-1 if x > 0 else 0) Limiting the scope is interesting, for "with" the only limitation in that the body must have exactly one assignment, like in the ternary if case? a = ([l.strip() for l in f.readlines()] with open ('name') as f) By cutting the edge cases, "with something():" is not possible, only "with ... as" being possible? But the "scope limitation" for try except in expression concept would be not to have "else" or "finally"? The else and finally clauses do not make sense in Expression style assignment anyway. a = (int('stuff') except ValueError: 5)
On Sat, Aug 04, 2018 at 12:28:40AM +0200, Robert Vanden Eynde wrote:
As programmers, surely we have to care about implementation details of the code we are maintaining or writing. If we don't care about implementation, who does? Of course there is a tension between having to deal with implementation details too earlier, or hiding them too deeply. We have to balance too little refactoring from too much refactoring, and people can legitimately disagree as to when a function, method or class carries its own weight. The same applies to syntactic features. That's why *concrete use-cases* are important, not pretend, made-up toy examples. The Python community as a whole is not very receptive to arguments from functional programming purity. I know Python veterans who still don't like list comprehensions (although they're a minority). As a whole, the community does not believe that using multiple statements is always a problem to be fixed. The question is not whether it is *possible* to have a with expression, or whether we might find some toy examples that kinda look good, but how much it improves *real code*. And arguments that we should have a "with" expression because we already have "if" expressions and comprehensions will just fall flat. Arguments by consistency ("we have A, so we ought to have E too, because they're both vowels") are not very productive. -- Steve
with supress(AttributeError): tmp = person.name[0] initial = tmp # Error on assignment wouldn't get suppressed. Not relevant for this case but still. I don't understand where this is ambigous?
data = file.read with open(...) as file works perfectly fine so why would you want to additonaly assign it to another variable "d"? Overall I like the idea of the with-expression as it allows you to make some common use cases like the open/read example more readable. It's clear at first sight that what is actually done is reading the file. While it's true that it's usually easy to write a simple function to do the job that still isn't as simple to understand and in most cases when reading the code you then have to look at the function to see if anything else is done there. Also what if you then do readlines somewhere? Then you need another function.
[Benedikt Werner]
Ah, yes. That makes much more sense now. I would say the error on assignment IS relevant for that case and it's not clear to me how you would present that error to the user without causing a lot of confusion. If it were implemented as: tmp = None with suppress(AttributeError): tmp = person.name[0] initial = tmp Then it would solve many of the common use cases for the None-aware operators proposed in PEP 505, especially if we made it easy to filter out None-specific errors: class NoneError(BaseException): pass class NoneAttributeError(AttributeError, NoneError): pass ... code to make erroneous attribute access on NoneType objects throw NoneAttributeErrors ... ... shouldn't break much code other than maybe some doc-string tests ... initial = person.name[0] with suppress(NoneError) [Robert Vanden Eynde]
Yes. Sorry for using confusing language. I was trying to say that I like your proposed syntax (for both with and except) because it follows the same principal. At the same time, I was trying to demonstrate part of the value of expression-ized statements to everyone else. Since Python is strictly evaluated statement by statement, there is no notion of looking ahead and re-ordering code. You can't use a variable in one statement then define that variable later on (like you do in comprehensions). Expressions, on the other hand; are parsed in a more complex way, so you can do things like put the relevant logic up front and move all the book-keeping to the end. [Robert Vanden Eynde]
True, and your except syntax could chain in a similar manner. In-fact it might be possible to make a separate "finally" operator that executes the left side then executes the right side no matter what: y = ((x[0] except InxexError: x.default) except AttributeError: None) finally print("hello!") [Robert Vanden Eynde]
I think it's better to forget what I said about limiting the scope of the operator. It was a half-baked thought. As for assignment expressions, they're already on their way thanks to PEP 572. I like the "where" version you proposed because it places the logic first, but unfortunately PEP 572 (4th alternate spelling <https://www.python.org/dev/peps/pep-0572/#alternative-spellings>) states that they had considered that version and rejected it because "where" is a pretty common variable name. From the PEP: SQLAlchemy and numpy have where methods, as does tkinter.dnd.Icon in the
standard library
So adding a new "where" keyword would break backwards compatibility in a major way. On Fri, Aug 3, 2018 at 5:40 PM, Benedikt Werner <1benediktwerner@gmail.com> wrote:
One last thing: Since expressions tend to have pretty limited space, it might be worth it to consider more concise options like maybe instead of: <expression1> except (Exception[-list])[ as e]): <expression2> it could be: <expression1> (Exception[-list])[ as e])! <expression2> So: y = x[0] IndexError! default instead of y = x[0] except IndexError: default or: y = spam.ham[0] (AttributeError, IndexError) as e! foo(e) instead of y = spam.ham[0] except (AttributeError, IndexError) as e: foo(e) Or something to that effect. (thought I think it might be best to err on the side of 'except' because it's more clear what it does and it only adds 7 characters). On Fri, Aug 3, 2018 at 6:57 PM, Abe Dillon <abedillon@gmail.com> wrote:
On Fri, Aug 3, 2018 at 5:26 PM Abe Dillon <abedillon@gmail.com> wrote:
You're reinventing PEP 463: https://www.python.org/dev/peps/pep-0463/
On Fri, Aug 03, 2018 at 06:57:40PM -0500, Abe Dillon wrote:
No it would not. The None-aware operators are not about suppressing AttributeError, please stop suggesting that it is. If you want to propose a general exception suppressing mechanism (aside from the existing try...except statement) then propose it as an independent PEP. Even if we had a general purpose exception-suppressing expression, it wouldn't meet the functional requirements for PEP 505. It would do too much, like offering somebody a bulldozer when all we want is a dustpan and broom.
They're not None specific. Any object can raise them. -- Steve
On Sat, Aug 04, 2018 at 12:40:13AM +0200, Benedikt Werner wrote:
Overall I like the idea of the with-expression as it allows you to make some common use cases like the open/read example more readable.
For some definition of "readable".
It's clear at first sight that what is actually done is reading the file.
Only because the file.read() comes early in the expression. But that applies equally to text = read('filename')
While it's true that it's usually easy to write a simple function to do the job that still isn't as simple to understand
(1) read(filename) Explanation to a beginner: "It reads text from the named file." (2) f.read() with open(filename) as f Explanation to a beginner: "It opens a file -- I'll explain what it means to open a file later -- in a context manager -- that's advanced Python programming, don't worry about context managers for now -- and binds the context manager to the name f, then calls the read method on the object f -- I'll explain object oriented programming later -- which reads text from the named file and closes the file... oh I forgot to mention that the context manager is also a file object." Do you still think that explaining a with expression is simpler than explaining a self-descriptively named function? The beauty of the named function is that it hides a lot of irrelevant detail and focuses *only* on the important feature, which is reading from the file. A with-statement is great for when you care about the implementation details. Somebody has to care about the process of opening a file, reading from it and closing it. But when you *don't* care about those implementation details, a simple interface like a read() function is superior to a with-statement, *or* a with-expression, which shoves those details in your face.
I doubt that many people really spend a lot of time digging into functions to see whether they do more than what they say. Unless and until I ran into unexpected problems, I'd be no more inclined to look into a function called "read" than I would be to do the same to len or math.sin. I'm sure it does what it says it does. In Python 2, one of the problems with the input() function was that it was used by people who didn't read the docs and where taken completely by surprise by the fact that it called eval on the input. That shows that people don't make a habit of digging into functions "just in case".
Also what if you then do readlines somewhere? Then you need another function.
Indeed. But not everything has to be a built-in feature, and refactoring common code into a named function will often be a good idea *even if* ``with`` is an exception: def read(filename, size=-1, **kwargs): return f.read(size) with open(filename, **kwargs) as f -- Steve
Totally agree. In the case where you care about the implementation details. You can explain to a beginner: something = (f.read() with open('...') as f) Is another way to write: with open('filename') as f: something = f.read() Exactly like the ternary if. And I agree that because with is often relevant for "implementation details" or beginning use as a "enter.. exit' statement (like a cpp destructor), it matches more imperative programming. with stack.push_matrix(Scale(4)): triangle = draw_triangle_using(stack) Is used for it's "enter/close" functionality, but can still be used like: triangle = (draw_triangle_using(stack) with stack.push_matrix(Scale(4)) as NotUsed) But the "as close" is useless here.
Fair point, the "def" statements generally are passed when read so they don't really count as overhead.
participants (13)
-
Abe Dillon -
Andre Roberge -
Benedikt Werner -
Chris Barker -
Cody Piersall -
Eric Fahlgren -
Ken Hilton -
Paul Moore -
Robert Vanden Eynde -
Steven D'Aprano -
Terry Reedy -
Thomas Nyberg -
Todd