Statements vs Expressions... why?

Greetings, Something that has started to annoy me in the last couple of years is the fact that most Python control statements cannot be used as expressions. I feel this is a pretty deep limitation and personally I don't feel it's well-justified. As I understand it, the reason for the distinction mostly has to do with the premise "flat is better than nested", which I can understand, but I don't think carries enough weight anymore. Specifically, I'd like to see things like "if" statements, "for" loops, etc, become expressions. This would not preclude them from being used as if they were statements (an expression can stand alone on a line of Python code), but would add a lot of expressiveness to the language as well as make Python more viable for creating DSLs. Additionally, removing statements from Python would also allow the language to be simplified. No need for a ternary "if" operator with different semantics than a normal "if" statement, "for" loops would be brought closer to generators in functionality, and the perceived limitations of lambda would disappear, amongst other things. We'd gain a lot of features found in languages like Lisp and Ruby as a side-effect (i.e. anonymous code blocks). Overall it seems this design decision is specifically geared toward forcing programmers into an imperative style in order to enforce program readability. In Python 1.5, this made a bit of sense, but as Python has "matured" (or in my view, gotten over-complicated) this makes much less sense. Many parts of Python's extensive syntax are explicit workarounds to this design decision. So on the one hand we have the perceived danger that programmers will write nested code and on the other we have an ever-expanding syntax. I'd take the former any day. I've not delved into the internals of the Python interpreter to check, but I suspect that converting most statements to expressions would not be very difficult (changing the grammar definition and generated bytecode a small amount in most cases). Any thoughts on this? I'm sure it's been brought up before, but I haven't found any definitive discussions on why this rather arbitrary design decision continues to hold in the face of a general migration away from imperative languages (especially when it seems it could be changed without much backwards-compatibility issues). Regards, Cliff

Cliff Wells wrote:
Two thoughts: Please elaborate how you like to change the syntax of Python. I like to see some concrete examples how your syntax would look like. I also like to know how your are planing to implement features like lazy evaluation. The if else ternary operator statement is evaluated lazy. The same construct as expression wouldn't be lazy any more. Secondly any syntax change won't happen until we start planing Python 4000 ;) Christian

On Wed, 2008-09-10 at 21:46 +0200, Christian Heimes wrote:
No changes. Simply lifting of a particular restriction.
Why not? a = ( if a > 1 then: long_calculation() else: other_long_calculation() ) Clearly only one of these blocks would be evaluated at runtime.
Secondly any syntax change won't happen until we start planing Python 4000 ;)
Yes, that's my expectation, although hopefully PyPy will make some of these things possible to experiment with well before then =) Cliff

On Sep 10, 2008, at 6:22 PM, Cliff Wells wrote:
Two restrictions, both that statements can be used in place of expressions and that statements now return values. But please explain how to do it in a way that is clear. It's a plus if it is backwards compatible :)
And now ifs return a value
You can do that now in PyPy or any other python version... or maybe use Logix to show us a proof of concept version: http://www.livelogix.net/logix/ Now if I remeber correctly this is already the case of logix, look at: http://www.livelogix.net/logix/tutorial/3-Introduction-For-Python-Folks.html... Why would you need lambda then? couldn't you just write x = def (x,y): return x+y ?

On Wed, 2008-09-10 at 18:52 -0300, Leonardo Santagada wrote:
Well, one or two, depending on how you word it ;-) My position is to simply do away with statements and provide equivalent expressions. As far as backwards-compatibility, this is legal Python: 1 2 3 It doesn't do anything, but it is valid syntax. This demonstrates that expressions can be valid when used independent of a statement, therefore, the following is also legal (assuming "if" were now implemented as an expression): if (a==1): pass
Which you are not required to use, just as today: def foo(x): return x**2 foo(2)
Yes, Logix is quite interesting. Unfortunately the author has discontinued work on it (and was, in fact, unreachable when I last tried to contact him). He also mentioned retargeting to another VM (rather than the Python VM) which made it less appealing to me. It is interesting that you mention Logix, since it demonstrates that it's quite possible to be backward-compatible with this particular change.
Why would you need lambda then? couldn't you just write x = def (x,y): return x+y ?
Yes, although I suspect that this particular change to "def" would be more substantial and "lambda" is currently equivalent. Although this might satisfy GvR's inclination to dump lambda at some future point ;-) Cliff

Arnaud Delobelle schrieb:
OT: I attended a lecture called "abstract machines" and we had to write a little VM (parser, bytecode interpreter) on paper during the test. It was a expression based language and there was a "if ... then ... else ..." AND a "if ... then ..." construct. I asked the Prof. exactly the same question you did. He said: "Oh... well... strike that out." He really didn't notice that this is a problem when he wrote the test. *g* :P (Sorry, for the noise.) -panzi

On 13 Sep 2008, at 23:17, Cliff Wells wrote:
Assuming the return value of "None", I go back to an example I gave earlier: factors = for x in range(2, n): if n % x == 0: x This doesn't work as intended (filtering out the non-factors). How to make it work? The only way I can think of is to make (if 0: 1) return a special "non-value" which loops will then filter out. But then we all know what happens to non-values. So how would you solve this problem?
Cliff
-- Arnaud

On Sun, 2008-09-14 at 07:23 +0100, Arnaud Delobelle wrote:
By writing it properly ;-) factors = for x in range ( 2, n ): if n % x == 0: yield x As I mentioned previously, in order to merge the concept of generator with a for-expression would require bringing in the yield keyword, just as it does now for generator functions. The example you gave would evaluate to None (or perhaps an empty list or generator - that's a detail that would take more consideration before defining it). Regards, Cliff

On 14 Sep 2008, at 07:36, Cliff Wells wrote:
OK, but this seems to me incompatible with current Python: def chain(I, J): for i in I: yield i for j in J: yield j Currently
'-'.join(chain('spam', 'eggs')) 's-p-a-m-e-g-g-s'
With your proposal, the first *expression* (for i in I: yield i) will evaluate to something like iter(I) and then be discarded. Then the second *expression* (for j in J: yield j) will evaluate to something like iter(J) which will be discarded. So chain('spam', 'eggs') will return None.
Regards, Cliff
-- Arnaud

On Sun, 2008-09-14 at 08:36 +0100, Arnaud Delobelle wrote:
It seems you have me on this one. There's clearly other ways to do the same thing, but since backwards-compatibility is a prerequisite I'll have to concede the point. A potential solution would to be to use a different keyword than "yield" to separate current syntax from my proposed syntax (that is, distinguish expression-scoped yield from function-scoped yield), and just side-step the issue, but that seems unappealing to me. Regards, Cliff

On Sun, 2008-09-14 at 01:25 -0700, Cliff Wells wrote:
Actually, a few more minutes of pondering got me a solution I don't find abhorrent. Let yield mean what it currently does. Instead, let "continue" accept an optional parameter (like "return"). Then your above example continues to work and my proposed change would look like: for i in j: continue i I haven't considered this too deeply, but it is at least consistent with "return" syntax and doesn't add a new keyword. Regards, Cliff

On Sun, 2008-09-14 at 01:36 -0700, Cliff Wells wrote:
I'm probably replying way too fast (in fact, I know I am), but I have two thoughts on this: 1) it seems to alter the semantics of "continue" too much when considered against current syntax, but... 2) with the new syntax, it seems not too bad because j = range(3) for i in j: i # evaluates to [] for i in j: continue # evaluates to [] for i in j: continue i # evaluates to [0,1,2] Overall I'm a bit torn on the idea. Thoughts? Regards, Cliff

On Sun, 2008-09-14 at 01:44 -0700, Cliff Wells wrote:
Bah, I knew I was replying too fast. I'm thinking that "continue" would be redefined to mean "yield value and continue" which means that for i in j: continue # evaluates to [ None, None, None ] not [] would seem the most consistent, but I fear it might be less practical (as it would create problems trying to use for/continue inside other expressions, although the effect when for/continue is used as a statement remains fine). Cliff

On Sun, 2008-09-14 at 01:54 -0700, Cliff Wells wrote:
It would *have* to evaluate to an empty list otherwise this code: for i in range(10000000): continue would create a huge list as a side-effect. So the question is, does this seem too inconsistent? Clearly returning [None, None, None] fits nicely with how yield currently works but it's not going to work in this case. Cliff

On 14 Sep 2008, at 09:44, Cliff Wells wrote:
Let's not call it continue, but YIELD for now: for i in J: YIELD i Now this won't work for nested loops. E.g. in current python def flatten(I): for J in I: for j in J: yield j >>> '-'.join(flatten(['spam', 'eggs'])) 's-p-a-m-e-g-g-s' Now say you want to write that inline with a for-expression: '-'.join( for J in I: for j in J: YIELD j ) That won't work because the j's will be accumulated in the inner loop and the outer loop won't accumulate anything, therefore returning an empty iterable. The flatten() example above works because the the scope of the yield statement is clearly defined by the enclosing def statement. To make it work, not only you need a special yield expression but you also need a special for expression: '-'.join( FOR J in I: for j in J: YIELD j ) Here it is clear what happens: the YIELD accumulates values in the FOR loop. Not very coder friendly though :) Now compare with the current syntax: '-'.join(j for J in I for j in J)
Regards, Cliff
-- Arnaud

On Sun, 2008-09-14 at 10:08 +0100, Arnaud Delobelle wrote:
How about this way instead (since for-loop is now an expression): '-'.join( for j in ( for J in I: YIELD J ): YIELD j )
Now compare with the current syntax:
'-'.join(j for J in I for j in J)
Certainly more clear and concise, but since (luckily for me this time) we're maintaining backwards-compatibility, that form would still be available. Cliff

On Sun, Sep 14, 2008 at 11:28 AM, Arnaud Delobelle <arnodel@googlemail.com> wrote:
Agreed. For all of the semantic and syntactic gymnastics and discussion about how statements -> expressions would make Python a better language, all I can conclude from the above is "I'm glad Python doesn't do that." - Josiah

On Sun, 2008-09-14 at 12:36 -0700, Josiah Carlson wrote:
Well, realize that being able to do something doesn't make it the right thing to do in a particular situation. I don't think Arnaud's goal here is to show that functional programming is bad, rather he's forcing me to work out whether or not it could be done without breaking existing syntax (and a fine job he's doing too). The examples we've been working through are testing specific cases. It doesn't mean it would be the recommended idiom for these cases. When I was testing the macro feature for Breve (a bad name for the feature, I now realize), I wrote intentionally horrific code, simply to test that the solution was general enough: http://breve.twisty-industries.com/snippets/macro-madness Would I ever do that in real life? Only for testing =) Cliff

On Sun, 2008-09-14 at 13:02 -0700, Cliff Wells wrote:
Using Breve as an example again: by your reasoning, the mere existence of Breve should be reason enough not to support OO in Python (or at the very least, magic methods). Breve more or less a functional DSL that barely resembles Python but actually *is* Python (in fact, I've seen someone assert that Breve templates could not possibly be actual Python code). Even you were initially convinced that I was doing code-generation. Breve abuses classes and magic methods (albeit in an entirely legal way) to create the illusion of a declarative DSL. Clearly I think this ability is a good thing, but it could also be argued that what I've done is inscrutable and this type of code should not be allowed in Python. In fact, your own example of mimicking a dispatch "table" with a class is arguably object abuse. Personally I don't think it's a bad solution (especially in the absence of a better way), but you've basically mapped what is logically a functional problem onto an object. My point is that just because something *can* be abused, it isn't reason to throw the baby out with the proverbial bath water. The more general a tool is, the more able it is to be used incorrectly (as anyone who's pried a lid off a can of paint with a screwdriver can attest). Regards, Cliff

Cliff Wells wrote:
If it looks that little like Python, I'd say it really is a different language, and you have no right to expect to be able to use the Python compiler as-is to process it. Rather than twisting Python to make it possible to abuse it even further, you'd be better off writing a Breve compiler in Python that produces Python code (or maybe even translates it directly to Python bytecode). -- Greg

On Sep 14, 2008, at 10:35 PM, Greg Ewing wrote:
I was having fun reading the discussion, specially now that it got to the part were most people are starting to realise how ugly python would become, to put it lightly we got to the dificult part (the "how"). But all Cliff wanted is to have better support for DSL in python, that is his real use case. I say why don't we focus on it. The only clean way to support DSL in python in my view would be to support something like pyparsing or ANTLR and make it generate a parser and some simple form of compilation to python bytecodes. How can we support that? Is this even desired? -- Leonardo Santagada santagada at gmail.com

On Sun, 2008-09-14 at 22:58 -0300, Leonardo Santagada wrote:
Actually you can do quite a lot with plain objects and class magic. But if your DSL isn't purely procedural (Breve is declarative-functional) then you are left without much in the way of flow control as most flow control in Python is done via statements. Overall I've already made a decision about my programming future, but certainly better DSL support in Python would be a good thing. Regards, Cliff

Leonardo Santagada wrote:
The way to go is probably to have the parser build an AST. That way you get to reuse the compiler machinery for generating bytecode, without having to worry about the messy details of generating textual Python source. If the DSL->AST transformation is sufficiently context-free, it could probably be specified using some kind of declarative grammar. -- Greg

On Mon, 2008-09-15 at 14:23 +1200, Greg Ewing wrote:
This is the approach Logix took, which is probably why it no longer works under Python 2.5. This is exactly why I find this type of approach unappealing (vs writing an internal DSL). I'm not sure what approach EasyExtend takes, but it's another potential player in this field: http://www.fiber-space.de/EasyExtend/doc/EE.html Regards, Cliff

On Mon, 2008-09-15 at 13:35 +1200, Greg Ewing wrote:
Well *I* think it looks like Python, but then I understand (as can I think anyone familiar with Python's magic methods), but at least superficially it gives a different impression. In any case, Breve is mostly of interest in that it is the project that both made me appreciate the power of expressions and start bumping into Python's second-class support of them.
I think code generation is arguably worse than functional programming.
(or maybe even translates it directly to Python bytecode).
A seriously complicated and highly technical bit of code I'd likely have to fix every few years (and maintain multiple versions of to support multiple Python versions)? No thanks. In any case, I've already choked down my Python snobbery and ordered a Ruby book. Clearly where I want to go in programming isn't available in Python nor will it be any time soon. Regards, Cliff

On Sun, Sep 14, 2008 at 1:47 PM, Cliff Wells <cliff@develix.com> wrote:
Let me make it clear what I was expressing, because you have no idea. I have personal opinions on what I like and dislike. It became clear to me early on that we disagree on language aesthetics. All I was expressing was that I'm glad that Python didn't look like what you were writing. If you've got a problem with my expressing of that, that's fine, you already decided to move onto a different language. But don't claim to know what I was saying or implying about Python as a language and what it should or should not support. The fact is, Breve is implemented using OO in Python. Could it have been done using a functional approach? Sure, but then you would have had to use multi-calling semantics, closures (which is OO in disguise), or code copy/pasting. Breve is not proof that Python should or should not do something, and to claim otherwise (or believe someone is claiming otherwise) is silly.
I knew that Breve was Python, but I thought it was doing something smart by also doing transparent code generation (because there was discussion about the "speed" of Breve) a'la Cheetah/Spitfire, because I believed that you had worked more on the language than you actually had. I was wrong. But that doesn't make Breve a good example of a DSL, it makes it just another internal DSL using Python. As you say yourself, there are many.
Whom has it been argued by? I think the only argument is against adding syntax to make Python purely functional. Using Python syntax and semantics to write interesting features is part of what makes Python great. I've created SQL code generators using __getattr__ and the comparison operators plus __and__ and __or__ for where clauses. And you know what? They looked a million times better than embedding SQL in the code or using stored procedures. None of them would be possible without __magic__ methods.
You didn't even understand my example. I wasn't mapping it into an object, I was using the underlying class creation semantics (specifically metaclasses) to generate a dispatch dictionary. I'll be more clear this time. def dispatcher(name, bases, dict): import __builtin__ dispatch = {} for name, fcn in dict.iteritems(): if callable(name): if '_' in name: typnam, value in name.split('_')[:2] typ = getattr(__builtin__, typnam, None) if not typ: continue dispatch[typ(value)] = fcn else: dispatch[name] = fcn return dispatch If you set the __metaclass__ of a class to the above function, you don't get a class, you get a dictionary. A dispatch dictionary in particular. One that works with arbitrary types (the above only works for builtin types or those that have been inserted into the builtin module). This method shouldn't be new to anyone. I've been doing it since before metaclasses (using functions and locals() instead, because I liked the look better), and I've seen it used in various codebases. You can do similar things to build properties (I have), and any one of a number of other really useful things.
Ahh, but the example that I was complaining about *wasn't supposed to be an abuse*, it was supposed to be an example of "how can we make this really simple Python code work with an expression-only version of Python?" That the *simple* Python became so complicated and ugly in the translation to a non-existing expression-only version of Python is unfortunate. But again, I'm glad Python doesn't look like that. Again, don't try to tell me what my reasoning is, I was expressing an opinion on aesthetics. - Josiah

On Sun, 2008-09-14 at 20:26 -0700, Josiah Carlson wrote:
I clearly read more into it than you intended. My apologies. As for opinions on what Python should or shouldn't be, I'll stop having opinions on it when you do (or probably sooner, since it won't matter to me anyway).
I didn't claim Breve proved anything. I resort to it as an example because it's handy for doing so (and because there are so few such examples in Python).
If it did code generation, then it wouldn't be an internal DSL, it would be an external DSL that happened to target the Python language (Cheetah) or VM (Spitfire) as a runtime. That's why I was so careful to mention it several times.
I meant hypothetically ("could be argued"). In no way did I say that someone here on this list had taken that stance (it hadn't even come up).
Yep, I didn't get it. I'd assumed you were using __call__ (which would seem a much simpler way to achieve the same thing). Frankly this example only cements my belief that this is overly complicated given the problem to be solved.
Again I apologize. But given how many times you did the exact same thing to me here (and rather rudely at that) I think we can call it even. Anyway, I'm done. Regards, Cliff

[snip plenty of wordy arguments about Python, DSLs, misunderstandings, etc, with almost no code, and going nowhere in particular] I'm not a Python guru, I'm not even a professional programmer, worse than that I've never even been a CS student. But I think, humbly, that this thread is much too preoccupied with personal/philosophical opinions about the aesthetics of Python and not enough with *how* to make Python work assuming that statements become expressions. IMHO if we try to work out how, we will come to the conclusion that it would make it a horrible mess. But we can only establish this if we give it a try, and in the process we might learn something useful. And who knows, we might even discover that Python was really meant to be a functional language! After all, this list is called python-ideas. Sorry for the noise. -- Arnaud

On Sun, 2008-09-14 at 19:28 +0100, Arnaud Delobelle wrote:
Ha! I knew I should have gone to bed earlier =) I = [ 'spam', 'eggs' ] '-'.join ( for J in I: YIELD ( for j in J: YIELD j ) ) Now if only I'd followed Guido's suggestion I'd actually be able to test before I post ;-) Cliff

Cliff Wells wrote:
Noooooo..... this is getting worse and worse. You seem to be thinking of syntax issues as though they were purely technical puzzles. The're not -- they're at least as much human-factors issues.
But if you keep all the existing syntax as well, you haven't simplified anything. -- Greg

On Mon, 2008-09-15 at 13:00 +1200, Greg Ewing wrote:
Not that it's terribly relevant to what you say, but: '-'.join ( for J in I: YIELD ( for j in J: YIELD j ) ) is the corrected form (although YIELD continues to be a placeholder of course). In any case, my form does make it slightly more complicated for the simplest case, but makes it much less complicated for more complex cases (for the same reasons a plain for-statement can): x = [ J [ a ] for a in b if c ] vs x = for a in b: if c: continue J [ a ] The complexity has barely increased and yet the second form is already more readable (the formatting of the first reflects what I typically do as listcomps get more complex - overkill here but it demonstrates where I'm going). Given that the listcomp is the only direct form of looping available as an expression, I've written some really ugly looking ones. I've actually taken to commenting open/close brackets simply to help distinguish them. Not to mention, the listcomp's placement of the yielded result before the loop and condition only works well when it's a very simple expression. It doesn't scale well with complexity (not that I think it was meant to).
Yes, that's unfortunate. It might, however, obviate the need for newer ones. Cliff

Cliff Wells wrote:
No, I think you were right the first time. The above looks like it will generate a sequence of iterators, not a flat sequence of j values.
That's a matter of opinion. If you lay out the first one as x = [ J [ a ] for a in b if c ] there's not much difference between them.
You're right, it's not meant to. It's meant for the simple cases where all the syntactic overhead of a full for-statement and list appending code swamps the content. Given that, adding any extra syntactic baggage, even just a 'continue' keyword, reduces its effectiveness for its intended purpose. Also, I think it reads quite nicely with the expression at the beginning. It's modelled after the mathematical notation for describing sets: {x : some conditions on x} -- Greg

On Mon, 2008-09-15 at 14:11 +1200, Greg Ewing wrote:
join() takes care of flattening the final yielded iterator. The first one was actually wrong in that it didn't solve the presented problem.
I thought I'd been saying that all along ;-) I think listcomps are only a *clear* win when they are presented in their simplest form, otherwise it really is just preference. Get into nested listcomps and the readability (or more to the point the comprehensibility) pretty much vaporizes.
So then maybe what we could agree on is that there's a place for both?
Sure, I think listcomps have a place. I still maintain that they are logically redundant if you have if-expressions (not to mention less flexible), but if we decided they were justifiable syntactic sugar then that's fine too. Cliff

Cliff Wells wrote:
But the final iterator is yielding other iterators, not strings, unless I misunderstand the semantics you have in mind.
That depends on what you mean by "nested listcomps". I agree that nesting one entire listcomp inside another tends to look rather confusing: [f(a) for a in [g(b) for b in y]] But that's not the same thing as having nested *loops* within a single listcomp, which I don't think is particularly bad at all: [f(a, b) for a in x for b in y] or if you prefer, [f(a, b) for a in x for b in y]
Yes, but they're also logically redundant even if you don't have statement-expression equivalence, so that's not an argument for merging statements and expressions. -- Greg

On Tue, 2008-09-16 at 11:41 +1200, Greg Ewing wrote:
I think so. For the simplest case s = 'abc' x = for c in s: YIELD c # x = 'a', 'b', 'c' so for the previous example I = [ 'spam', 'eggs' ] for J in I: # J = 'spam' then 'eggs' YIELD ( # evaluate to an iterable for j in J: YIELD j # j is 's' then 'p' then ... ) so we get '-'.join( 's','p','a','m','e','g','g','s' ) maybe this is clearer as for J in I: tmp = for j in J: YIELD j YIELD tmp
Well part of the problem this entire thread has suffered is that there are several related issues being argued simultaneously. Perhaps if I were a better presenter this would have gone differently, but in any case what I was trying to get across is that Python has grown lots of extensions that could be considered redundant and will grow more unless what I see as the seed for these desired extensions is addressed (I believe it to be a desire for better FP support, which merged statements and expressions would largely address). I selected the ternary if-operator, generators and listcomps as examples but I may have overreached a bit. Regards, Cliff

Cliff Wells wrote:
s = 'abc' x = for c in s: YIELD c # x = 'a', 'b', 'c'
You mean that the value of (for c in s: YIELD c) is a tuple? Or that it's an iterator that produces that series of values?
I'm still not getting a clear idea of what semantics you intend for a for-loop-with-YIELD. Neither of the interpretations I suggested above (sequence or iterator) seems to produce this result.
No, that's not any clearer. Can you provide a translation into current, valid Python? -- Greg

On Tue, 2008-09-16 at 13:06 +1200, Greg Ewing wrote:
The latter, although I admit not seeing the importance of the distinction in this case.
My example was a translation of Arnaud's challenge: I = ['spam', 'eggs'] def flatten(I): for J in I: for j in J: yield j >>> '-'.join(flatten(['spam', 'eggs'])) 's-p-a-m-e-g-g-s' It's possible there's a nuance you're seeing that I'm not (and unfortunately we can't test). What are you predicting as the output? Cliff

On Tue, 2008-09-16 at 13:06 +1200, Greg Ewing wrote:
Actually, I think I see the issue. join() is getting a list of iterators (using list notation for simplicity): '-'.join ( [ [ 's','p','a','m'], ['e','g','g','s'] ] ) (I promise you I'm not being intentionally obtuse). Is this the conclusion you were coming to or something else? Cliff

Cliff Wells wrote:
Yes, that's right. Arnaud's original code is a single generator containing two nested loops, whereas your version consists of two nested generators. I think your first translation would have produced the right result, because it iterated over each of the inner generators and re-yielded their results in the outer generator. But that's a very expensive way of going about it. -- Greg

Cliff Wells wrote:
A potential solution would to be to use a different keyword than "yield" to separate current syntax from my proposed syntax
But inserting any kind of keyword into LCs is going to make them ugly and verbose. The point of having LCs in the first place is that they express certain kinds of things very concisely. Adding keywords and colons to them messes that up. Likewise with conditional expressions. It would be disappointing if, instead of x = a if b else c we had to write x = if a: b else: c Colons in the middle of expressions look ugly, IMO. -- Greg

Cliff Wells wrote:
Is this construct meant to be a list comprehension or a generator expression? If it's a GC, you just get factors bound to an iterator, not a list of results from that iterator. If it's an LC, then what do you do if you want a dict or tuple comprehension instead? Also, 'yield' already has a meaning that's different from what you seem to want here. Consider: def foo(n): for x in range ( 2, n ): if n % x == 0: yield x Here the whole function is a generator, and the yield causes an item to be returned to the generator's caller. Now, if there is no difference between a statement and an expression, the following should do exactly the same thing: def foo(n): factor = for x in range ( 2, n ): if n % x == 0: yield x -- Greg

On Mon, 2008-09-15 at 12:37 +1200, Greg Ewing wrote:
First thing, I've recanted reusing "yield" (see below), but your questions still hold in any case. It evaluates to an iterable, I think a generator being the logical choice (although I've been using "[]" to indicate an empty iterable in discussion for brevity).
If it's an LC, then what do you do if you want a dict or tuple comprehension instead?
Did we already grow those? I don't want to debate tuple comprehension (I admit I fail to see the point), but dict comprehensions would be great. To be clear, I'm aiming for as much backwards-compatibility as possible, so I'm in no way suggesting removing comprehensions (although I think they'd become somewhat redundant). In the hypothetical absence of a comprehension feature, I'd probably try to write one like these (I'm slipping "continue" in here as a replacement for yield, which I explain below): x = for i in j: if i: continue i else: continue i**2 # list comp x = dict ( for i in j: continue i, i**2 ) # not a dict comp ;-)
Also, 'yield' already has a meaning that's different from what you seem to want here. Consider:
Yes, that's why I had to retract using the yield keyword in this context. Instead I propose either a new keyword or allowing continue to accept an optional argument causing it to mean "yield expression and continue" rather than just "continue": factors = for x in range ( 2, n ): in n % x == 0: continue x continue without an argument would continue to mean what it does now. This raises some other concerns, but it's an option to be weighed against adding a keyword. In any case, using "yield" is straight out. Regards, Cliff

On Sat, Sep 13, 2008 at 7:01 PM, Arnaud Delobelle <arnodel@googlemail.com> wrote:
Just a note how this problem was solved in Nemerle language: you cannot use 'if' without 'else'. This way you always get a value. If you want an 'if' without 'else', there is a new keyword 'when' (well, there is also 'unless') for the imperative approach. Cheers, Peter

Piotr Wysocki wrote:
If you want an 'if' without 'else', there is a new keyword 'when' (well, there is also 'unless') for the imperative approach.
But in a language where every statement is an expression, this just moves the problem to that of what the value of a "when" statement should be. -- Greg

Cliff Wells wrote:
Can you show me what an "if" in a "lambda" used in a function call would look like? My major complaint with the statements-as-expressions is that multi-line statements really don't look good when used in expression contexts. Perhaps you have a good suggestion for the syntax, and if so, I'ld love to see it.
Well, I suspect that delving into the internals to check this theory would make people take your proposal a lot more seriously. Heck, I'ld go so far as to say that without doing that, it's all just talk, and talk is cheap. ;) Thanks, Blake.

[Sorry for the private reply earlier] On 10 Sep 2008, at 19:43, Cliff Wells wrote:
Can you post some sample code to illustrate how statements could be used as expressions? Do you propose that we write: y = if x == 0: 0 else: exp(x**-2) instead of: y = 0 if x == 0 else exp(x**-2) ? Or, how would you write factors = [x for x in range(2, n) if n % x == 0] ? Something like this maybe: factors = for x in range(2, n): if n % x == 0: x Or do you suggest something else?
So do you think readability is not as important now as it was?
I think to call this feature of Python an arbitrary design decision is a misjudgement. To me it is central to the identity of Python. -- Arnaud

On Wed, 2008-09-10 at 20:55 +0100, Arnaud Delobelle wrote:
Yes, and parentheses could be used to disambiguate, as anywhere else.
These are correct, albeit simple examples (you are using only assignment examples). To give something more interesting, I'd like to be able to do things like this: dispatch = { '1': lambda x: ( for i in range(x): if not x % 2: yield 0 else: yield 1 ), '2': lambda x: ( for i in range(x): yield i ) } for i in dispatch[val](1): print i Note that this isn't just about lambda, but rather general expressiveness. To clarify, I propose *not* making syntax changes that add to the language or alter the meaning of existing code. Rather my proposal is along the lines of "relax this particular syntax requirement" and let the chips fall where they may. In other words, Python 2.x code would still work, it simply would not take advantage of a particular coding style now available to it. The key difference between a statement and an expression is that 1) an expression has a return value, a statement does not 2) statements cannot be used inside of expressions In short, statements are more-or-less castrated expressions. They have no other special features.
Of course not. I'm saying that due to this limitation, readability is going down the drain even faster. It's getting to the point where it's not possible to keep the whole of Python in your head and I feel that a significant portion of this is due to this particular inflexibility. Further, I feel that this limitation forces programmers into using hacks and magic or overly spread-out code, which itself leads to readability concerns. Having used Python for around a decade, I'm quite aware of the fact that you can happily write tons and tons of nice code with Python in its current state. However, because of the direction of my work (developing a internal DSL in Python) I've suddenly become aware of this glass ceiling. I'd bumped into it before back when I was doing a lot of GUI development, but blamed it on lambda rather than realizing that it wasn't lambda so much as what I am bringing up now.
Sorry, I misspoke: it's not an arbitrary *decision*, but it is an arbitrary *distinction*. As I said, I'm aware of why this decision was originally made (prevention of nested code, a.k.a. fear of lisp), but the distinction itself is completely artificial, created to enforce a model of programming, rather than for technical or performance reasons. I agree that it is a distinguishing feature of Python, but I don't think it is central to Python's identity. That is, it wouldn't be "not Python" were it removed. Regards, Cliff

On Wed, Sep 10, 2008 at 3:18 PM, Cliff Wells <cliff@develix.com> wrote:
Python is not intended for DSLs. Really, don't do it. Python is for python code. If you want another language, write your own parser. I hear lisp is simple to parse, and has no annoying statements to hold you back! Seriously though, there is an advantage to basing so much on statements rather than expressions. We're specialized for one statement per line, which is the most common case, and it allowed us to have extraneous semicolons, braces, or whatever. Readability benefits, style consistency benefits. Now there are some use cases that suffer here, including the one you just gave: defining a dispatch dict with the functions inline. The best I could do is define the dict first, then stick a decorator on each function to register them. That's still ugly though. A creative solution is needed, but none come to mind. An example where this has happened before is the with-statement, which is spectacularly successful IMO. Now, you may notice it could have been done in a library rather than core syntax if generic anonymous blocks were allowed — so what? The library is still part of the language! It's still something that has to be learned. And the syntax would be substantially uglier using a generic mechanism, rather than the specialized with-statement syntax. -- Adam Olsen, aka Rhamphoryncus

On Wed, 2008-09-10 at 15:57 -0600, Adam Olsen wrote:
The DSL I work on *is* Python code. And really, this is the first time I've heard anyone assert anything like this. Python is a general-purpose language. It's not VBA ;-) DSL's are an extremely useful concept. To summarily dispatch the whole of them with such an assertion is pretty much bolstering my argument: you've just asserted that Python is inherently limited in scope.
Ah, except Python is the language I like in every way, *except* for this one particular wart. Really, had I not entered new programming domains and tried to take Python with me, I'd probably never have had a complaint. Also, external parsers defeat the entire reasoning behind internal DSL's (taking advantage of an established VM/compiler, requiring users to learn a new syntax in addition to their primary programming language).
Clearly it's the most common case in existing Python code since nothing else is allowed. But frankly, even Javascript doesn't follow this idiom anymore. Expression-oriented languages have seen a rebirth for good reasons (although I admit I'm none-to-fond of many of them, for various reasons).
and it allowed us to have extraneous semicolons, braces, or whatever.
Not following this. You mean to *not* have extraneous syntax?
Readability benefits, style consistency benefits.
I strongly disagree. The artificial distinction between statements and expressions is the definition of *inconsistent*. Why do we have two versions of the "if" conditional? Why do we have "for" loops *and* list comprehensions? They express the same ideas, but the limitations in one required growing the language another direction. In short, we could have had a single, more powerful construct in place of two lesser constructs and simultaneously had less syntax to memorize and more consistency across the board.
That's because there is none. And this is my fundamental problem: it's not so much that it's hard to do in Python, it's that you *cannot* do it in Python. No amount of creativity, time, or experience will help, and this is disappointing. I won't pretend that any example we might toss up in here won't appear contrived, but there are definite cases where readability can be substantially enhanced with such structures.
The "so what" is that it could *only* be implemented by the core devs. It was not possible for an average (or even above-average) Python programmer to write such a library, whereas it *could* have been had the language not prohibited it. Regards, Cliff

On Wed, Sep 10, 2008 at 4:39 PM, Cliff Wells <cliff@develix.com> wrote:
I agree, DSL's are useful. They're just not something python currently supports well. They're the use-case you need to justify against the substantial changes you propose. Maybe it's worth it, or maybe it's better to add an indent-sensitive string literal that would then allow *arbitrary* DSL syntax?
I appreciate what you're saying here, and feel much the same way about my own pet-features, but this is a really poor argument. Everybody has just one wafer-thin feature they'd like to add.
Again, "XYZ language has it" is an ineffective argument.
Consider a loop in C vs in python for (i = 0; i < 50; i++) { a(); b(); } for i in range(50): a() b() We avoid the semicolons and braces because the newline and indentation indicate the same thing. That's what a statement it. Perhaps you can retain this with your proposal, but only because a statement is still the default, with expression as a rarely used option.
I was referring to consistency of the programs, not the language itself. No silly arguments about brace position because they *are no braces*. There's actually a problem with trying to merge a for-loop and a list comprehension. A generator expression is the canonical generic form, but [genexp] would create a list containing a single genexp object. Likewise, a for-loop would become lazy, so without some extra syntax to evaluate it (and trigger the side effects!), your programs would cease to do anything. So you see, despite significant and obvious similarity between the features, there's some important differences as well. These are so obvious from the contexts that you don't even think of them, so clearly the mental load of having 3 (soon to be 5) different forms of looping is not all that great. Mental load is what really counts, not some abstract concept of complexity.
I didn't mean to do it in Python. I meant to modify the language.
As important as it is to extend the language via a library, somewhere you need to draw the line and start modifying the syntax or other fundamental builtins. The is a universal tenant, applying even to lisp (which has very little syntax, rather than extensible syntax). This is why we have a+b, rather than add(a, b). More syntax when it's worth it. So it's not that we don't want to allow extensibility - quite the opposite. It's that we want the common statement to be simpler, and the extra syntax hasn't been justified for your use cases. -- Adam Olsen, aka Rhamphoryncus

On Wed, 2008-09-10 at 17:16 -0600, Adam Olsen wrote:
Well, this is where we agree. Where we seem to disagree is whether or not something needs to be done about it =) At some level, the whole concept of OO programming *is* DSL support. I consider "DSL" to mean "mapping of abstract (language) constructs to real-world (domain-specific) constructs". This type of DSL construction is quite-well supported by Python (and, in fact, is the way Breve is constructed under the hood). However, this support is hampered (to some degree) by having a somewhat inflexible syntax.
They're the use-case you need to justify against the substantial changes you propose.
I guess I don't see it as substantial to people who don't wish to use it (although it's quite substantial to people who do). Overall, I think this is why I feel the change doesn't require a huge amount of justification: you aren't *forced* to use it, but if you need it, it's huge. It doesn't impose any significant stylistic change on people who prefer the imperative style, but it opens vast doors for people wishing to approach problems from a functional path.
Hm, I'd have to see an example (even contrived) of what you mean here.
Again, this is only wafer-thin if you don't want to use it (and that is part of its appeal - it's actually rather non-intrusive to classical statement-oriented programming). What I see happening in Python is exactly what you appear to be arguing against. Little specialized features are added one after the other to satisfy particular needs, when what is actually needed is one sweeping change that would make those features redundant.
Not really what I was trying to say. What I was saying is that expression-oriented languages are rising (practically from the grave C put them in) because they are inherently useful and typically more powerful than their imperative counterparts. Where they've tended to suffer is the place where Python shines: readability. What I'm claiming is that Python's readability is not due to its imperative nature, rather due to it's whitespace-oriented syntax and lack of line-noise, so Python could literally become the best of both worlds were it to shed its imperative roots.
That's what I thought you meant. You dropped the "not" in your original statement.
Exactly. My point remains that the imperative *style* might still be preferred (and encouraged) because it *is* inherently simpler to understand, but that it shouldn't be enforced because it's also inherently limiting (not surprisingly). I think the community as a whole has been successful in forwarding "Pythonic" idioms even when the language allows "unPythonic" coding without resorting to b&d tactics within the language itself. I think the same external discipline can be applied to this concept without limiting Python's applicability in other domains.
But you are assuming that eliminating statements somehow requires extra syntax. I think Logix, which was mentioned previously, adequately demonstrates that this is not the case. The only "extra" syntax that would need to be used would be parentheses for logical grouping of expressions. This syntax already exists in Python and is already used for the same.
I don't think so. Think about how Python currently defines a generator: the presence of the "yield" keyword within a function. I think this same logic could be applied to a for loop (but I'm willing to be corrected if you see a problem). A for loop without "yield" is more or less what it is today (except it might evaluate to []). A for loop with "yield" is a generator (and by extension, useful as an expression).
This isn't abstract. It's a matter of countable constructs, rules, and exceptions to rules. Consider: Rules in statement-oriented Python: 1) distinguishes between statements and expressions 2) expressions return a value, statements do not 3) expressions can be used inside other expressions and statements, statements cannot 4) there is an if statement 5) there is an if expression (ternary if operator) 6) there are for loop statements 7) there are list comprehensions 8) there are generators ("yield" defines when a function is a generator) Equivalent rules in expression-oriented Python: 1) if expression returns a value (that can be ignored or used in an expression) 2) for expression returns an empty list or a list if it's a generator (has "yield" keyword) that can be ignored or used in an expression. Which has more "mental load"?
I don't consider modifying the language an acceptable solution for most programmers. It's a maintenance nightmare.
Yes and no. I believe it should be possible to prototype almost any construct via a library. Whether the language should then embrace the concept embodied in that prototype to provide better integration, performance, or simply syntactic sugar, can then be argued much more fruitfully. If a language prevents you from creating such prototypes then I think
But the language didn't prevent you from creating the hypothetical add() function. It merely provided syntactic sugar for making it more readable. I feel we're getting slightly side-tracked here =)
Once again, I am forwarding *zero* extra syntax. In fact I am suggesting *less* syntax overall and an accompanying reduction in rules regarding the remaining syntax. I am suggesting removing (or deprecating) syntactic additions such as the ternary operator in favor of extending the power (or more to the point, removing arbitrary limitations) of the existing core language. Regards, Cliff

On Wed, Sep 10, 2008 at 6:14 PM, Cliff Wells <cliff@develix.com> wrote:
Once language syntax is added to, changed, etc., it's very difficult to remove those additions, changes, etc., even when the feature is rarely used, ugly, and generally a bad idea (see back-quotes `x` for repr(x) ). This may not seem like a big deal to you, because you want this feature, but for the rest of us who have little (arguably no) use for the feature, adding semantics to syntax, or adding syntax = additional mental overhead; never mind the additional sections in the tutorial where we have to explain why this *particular* special case of a DSL was special enough to break the rules of explicit is better than implicit (why should a multi-line if statement implicitly return something in one place but not in another?) I know, "it's just one little change". I've made the argument myself. But that doesn't mean that my idea was a good idea (it wasn't), nor does it mean that your current idea is (I think everyone in this thread except you would agree that it's a bad idea). Before continuing on in defending your proposal here, I suggest you try comp.lang.python . If you can't get a dozen people to agree that it's a good idea, or if (like here) the only replies are negative (or even if the majority of them are), then I would suggest you drop it. - Josiah

On Wed, 2008-09-10 at 18:30 -0700, Josiah Carlson wrote:
Again I assert the opposite: that Python is currently forced to explain the arbitrary distinction that currently exists and that a significant amount of extra syntax and language features exist to work around that distinction. People may not clamor for the change but there's quite a few newcomers to Python who must have the distinction explained to them.
I respect your work and knowledge of Python, but that's insufficient to get me to respect a straw-man argument. I've not suggested that Python add anything for any particular situation. I'm suggesting a change that would make it more logically consistent, expressive, and help reduce the language and syntax clutter it's been accruing at a steady pace. The fact that it also helps in a rather broad range of problems is ancillary and my particular use case was presented as an example, not as *the* problem to be solved. Also, I don't know where you got the idea that I suggest any expression should return a value sometimes but not others. I'm suggesting they *always* return values, but that you don't always need to worry about them unless you plan to use the value in another expression. Kind of how functions already work in Python.
Sure. Because everyone who might have agreed with me has already migrated to Ruby or started a new language (Logix, Boo, et al). I just happen to be lazy enough to entertain the thin hope that Python could be fixed so I don't have to learn a new language or programming environment ;-)
Actually I avoided c.l.py because I saw this list suggested as the appropriate place for new or controversial ideas (and the list name seemed to suggest the same). The fact that lambda was so bitterly defended against GvR's strong desire to remove it should be hint enough that a significant number of Pythonistas are interested in functional and expression-oriented programming. In any case, I actually got the response I expected and ultimately I expect the discussion here was probably far more enlightened than I would expect on c.l.py. It seems most Pythoneers work in particular domains where this isn't an issue, are satisfied with the workarounds, or simply are unable to see there's a vast world of elegant solutions beyond what imperative languages can describe. Unfortunately this only confirms my (rather reluctant) expectation that I'll be forced to move to a more expressive language to satisfy my programming itches. In any case, thanks for the feedback. Regards, Cliff

On Thu, Sep 11, 2008 at 12:34 AM, Cliff Wells <cliff@develix.com> wrote: On Wed, 2008-09-10 at 18:30 -0700, Josiah Carlson wrote:
For better or for worse, most people come to Python with imperative rather than functional language background, so the distinction is either not even realized or it usually seems "natural" when realized. Personally, I find some features to be more natural as statements (e.g. it is typically obvious whether to use a "for" loop or a list/gen. comprehension) but others unnecessarily limiting (e.g. having to write "lambda x: x.__setitem__(0,1)" instead of "lambda x: x[0]=1"). George

On Wed, Sep 10, 2008 at 9:34 PM, Cliff Wells <cliff@develix.com> wrote:
For people who have gone through the Python tutorial or any one of the many good 'learning Python' texts (online, paper, free, and paid-for) the distinction has already been explained. For those who refuse (for some reason) to go through one of those resources, that's not an issue with Python, that's an issue with the user. As a data point; everyone that I've taught Python over the years (undergrads and interns who only ever used Java before to 30-year programming veterans of apl/cobol/fortran), not a single one of them has ever had a problem with the difference between statements and expressions. It is possible that I've only ever worked with gifted students and coworkers, though this is unlikely.
What I've found (historically) is that any time I find myself trying to do something that is not available within the syntax of Python, it's usually because it's a bad idea. In your case, you want (for example) the following to be valid Python... lfcn = lambda x: ( if x: True else: False) Similarly... lfcn = lambda x: ( for i in xrange(x): yield fcn(i, x, ofcn(i)) ) Why? Initially it is for a templating language that you have designed that makes these kinds of things difficult to do during an automatic translation from working Python source code into compiled source code. Presumably something of the form... html [ body [ a (href='http://python.org') ["Python!"] ] ] To be converted into something like... def generate_html(out): out.write('''<html> <body> <a href="http://python.org">Python!</a> </body> </html>''') Trust me, I understand your desire. I have used Cheetah and Spitfire (http://code.google.com/p/spitfire/), and the speedups gained by translating templates into Python source are not to be laughed at. And before I used those, I saw nevow.stan (from which your syntax is effectively identical to), and wrote my own variant (http://code.activestate.com/recipes/440563/ - I use multi-calling semantics rather than __getitem__ stuff). The only reason I didn't write a compiler (and run into the same issues you have) was because I didn't care about dynamic content generation, I was using it for pre-generation of static web pages. But I digress. At some point, you acknowledge that your templating language is constrained by your programming language of choice (as is the case for systems that use the base programming language for syntax, because of the convenience of parsing), or really, your templating language is constrained because you refuse to write your own parser. Don't get me wrong, the convenience of writing a templating language in Python syntax is amazing...but you've noticed that your compilation steps are hampered by Python's immediate execution of code. In fact, your own renderers are a work-around for something that is /relatively/ natural in Cheetah/Spitfire: $for $link in $links: <a href="$link.url">$link.label</a> Which is generally compiled into the following (with a bit of extra garbage, but you get the idea)... for link in links: out.write('''<a href="%s">%s</a>'''%(link.url, link.label)) Now, I loathe writing Cheetah/Spitfire because I loathe writing html, which is why I wrote my own templating system, but there are work-arounds. For example, if you want to stick with Python syntax... For("$link", "$links", "<a href="$link.url">$link.label</a") Now you just need to write your For factory and flattener, which produces the right code (easy), and with a little work, you can even make for loops composable, add ifs, etc. I know, it's not as elegant (for you) in this situation as statements becoming expressions, but it will work today. (note that pie-in-the-sky best-case scenario is 18+ months for Python 3.1, but more likely 15 years for Python 4 ;) )
You are right, I misread an earlier email in this thread.
That's not necessary. You've already solved your problem with renderers, and I've pointed out another method for getting some behavior in-line (If you are careful, you could probably build all of Python out of constructs similar to the For above). Of course there's always the option of just using some version of the output of the compiler/ast (I have done as much to pull out class definitions, function definitions, etc., in the editor that I have written). You could even write a tool for doing the conversion ;) .
Indeed! I have a penchant for the use of list comprehensions, generator expressions, map, etc. - when they are reasonably applicable. The question is always: what is reasonably applicable.
Pythoneers do see that there are /alternate/ solutions to what is currently available within Python, hence why people tend to like to write parsers in Python ;) . One thing you may want to look into is that you aren't using the full expressive power of Python in your templating language, and because of this, it is actually quite easy to write a parser for your subset, adding all of the necessary behavior and translation. I used DParser in Python for a few years and loved it, though it seems like the home page for it is no longer available. On the one hand, yes, you are currently constrained by the limits of Python's syntax. And yes, you could move to Ruby, Boo, etc. Or you could try alternate constructs (For, If, Def, etc.), write your own parser (which isn't hard), or shrug and admit to yourself that the limitations really aren't all that bad. Inconvenient, sure. The end of breve in Python? Not necessarily. - Josiah

On Wed, 2008-09-10 at 22:47 -0700, Josiah Carlson wrote:
You won't catch me disagreeing with this, but I was trying (clumsily) to point out that the distinction violates (albeit subtly) the principal of least surprise and so requires explanation (even if the explanation is very simple).
Without any supporting data, my hunch is that most people who migrate to Python come from other imperative languages (myself included) and so don't find the distinction foreign.
Actually, it's much simpler than that. The HTML tags are objects which simply get serialized into their text representation. I specifically do not generate Python source. In fact, you can simply do something like this in the interactive interpreter:
print html [ body [ a (href='http://python.org') ["Python!"] ] ]
<html><body><a href="http://python.org">Python!</a></body></html>
This is more or less how Nevow Stan does it as well.
I've actually employed several methods for working around this issue in Breve: For looping: - list comprehensions in place of for loops (arguably the natural choice in Python, but list comps tend toward unreadability quickly). - tag multiplication e.g. li(class_='$class')['$value'] * [{'class': 'even', 'value': 1 }, {'class': 'odd', 'value': 2}] results in <li class="even">1</li><li class="odd">2</li> For conditionals: - ternary if-operator for Python >=2.5 - test(val) and [ block ] in place of if-statement (for <2.5 support) e.g. test (session.logged_in) and div['hello', session.user] For both: - renderers - push the logic back into Python proper (as you suggested) At one point, I even wrote a set of custom tag classes that allowed things like this: switch ( x ) [ case ( 1 ) [ 'x is 1' ], case ( 2 ) [ 'x is 2' ], case ( 3 ) [ 'x is 3' ], default [ 'x is not in list' ] ] While these worked superficially, they were ill-fated due to lack of short-circuiting/lazy evaluation (which rendered them both inefficient and incorrect). At the end of the day, while these sorts of methods work, they feel like what they are: workarounds (although 2.5 helps a bit with the ternary if-operator). This might be nit-picking on my part, but I don't think I'm especially unusual in my desire to do the "Right Thing". Ultimately though, whether/how I solved these issues can be considered two ways: from a practical point-of-view (what you are espousing) or from a purist's point-of-view (which I know is not a popular view amongst Pythonistas). I mentally weighed the pros and cons of having statements and ended up finding the pros lacking (ultimately coming down to enforcing an imperative style).
I've considered this, but frankly find AST manipulation a bit daunting (and even more of a workaround than the above pure-Python solutions). I've also considered trying out EasyExtend to accomplish my goals but it still doesn't feel quite right.
As you might expect, I subscribe to the "consenting adult" perspective. I think "reasonably applicable" is a determination best left to the programmer managing a particular piece of code rather than the programming language. Readability matters most to those who have to read it (which is, more often than not, the person who wrote it). What is readable to one isn't always equally readable to someone else. For example, list comprehensions used to appear completely inside-out to me whereas map/filter/reduce was as simple as could be, and yet I've hear GvR claim on several occasions that the exact opposite is true for him. Most likely we'd differ just as much on any sufficiently non-trivial bit of code. But at the end of the day, GvR probably won't need to worry about how readable any bit of my code is, it only matters to me. And even if someone else were to need to read my code, it's a toss-up which way he'd find more readable. Ultimately I don't think there's a clear win here either way, but I feel that Python needlessly enforces a single vision to the exclusion of other equally valid visions.
Well, as far as Breve goes, I think there are adequate workarounds. What has disturbed me was finding I that I actually needed workarounds (something I haven't needed a lot of in my years of Python programming). This got me to thinking about the dichotomy of statements and expressions in Python and I was irrevocably drawn to the conclusion that this distinction is not as inconsequential as I had once believed.
Admittedly, this is largely a matter of taste. But of course, many seemingly critical decisions about programming are. Unfortunately, once I'd had my eyes opened to this limitation, it's now impossible for me not to notice. For example, when I see how much more natural Twisted's deferreds appear in MochiKit than in Python (where they originated), I can't help but believe that Twisted's failure to reach a larger audience is at least partially because that approach isn't as easily expressed in Python as it would be in a more expression-oriented language. In this case, it's not even that it can't be done (it clearly can), it's that the approach feels forced rather than natural. Overall, I'm left with the nagging suspicion that because of my language choice, I'm missing an important paradigm that would elevate my programming to the next level. In any case, I expect I'll stew in my current situation for a while longer at least. The unfortunate fact remains that most of the languages that speak to me (e.g. Boo and Io) don't have the broad range of libraries and frameworks available that Python does and ultimately this tends to outweigh my esoteric desires. Regards, Cliff

On Thu, Sep 11, 2008 at 2:05 AM, Cliff Wells <cliff@develix.com> wrote:
Your hunch is likely correct. I personally came from Scheme and C/C++. Many of Python's functional constructs fit the part of me that got scheme, and everything else fit the part of me that got C/C++.
Not translating to Python source code is a lot less fun ;), and also makes your templates far less dynamic in real-world scenarios. [snip]
Then what the hell are you waiting around here for? If you are feeling constrained by Python (as a general-purpose language), try some others! You may want to add Haskell, Caml, or even Lua to your list of languages to check out. Maybe your Python will get better, or maybe you'll move on from Python. Who knows?
One thing that you need to remember about Python is that it's a general language. It's syntax and semantics are such that it works in a fairly large variety of situations. Because it's a general language, it sometimes doesn't do the things that would be convenient in a domain specific language. In particular, it doesn't have a lot of those things that would make functional programming more convenient (a shorter way of spelling 'lambda', everything is an expression, etc.), but no one language can be perfect for everyone. I wish you luck in your adventures with alternate languages. - Josiah

On Thu, 2008-09-11 at 10:41 -0700, Josiah Carlson wrote:
Hey, I said I was lazy! ;-) Anyway, my biggest hurdle to most of the languages I've considered boils down to: 1) Lua - not expression based 2) Haskell - too ugly 3) Io - no libraries 4) Lisp - I find it borderline unreadable, plus I'd have to talk to other Lispers 5) Boo - I'd have to learn .NET and deal with the fact that 95% of the community is on Windows 6) Erlang - too ugly, too fast 7) Ruby - Too ugly, too slow, plus I'd probably have to make a cutesy website with cartoon animals and robots to show my fandom Two other options: 1) Logix - I could attempt to revive it. This is actually fairly appealing if a bit daunting to dive into. 2) Fork Python - I've considered making a proof-of-concept with a Python fork that does what I want, but it would be disappointing to discard it when it was inevitably rejected, and I have no intention of becoming a language maintainer. I think I'm going to play with Io for a bit in spite its shortcomings as it is explicitly a multiparadigm language and seems to look the cleanest.
But it's so damn close =)
I wish you luck in your adventures with alternate languages.
Thanks for all your time and thoughts. Regards, Cliff

On Wed, Sep 10, 2008 at 7:14 PM, Cliff Wells <cliff@develix.com> wrote:
Aye, but.. it's hard to explain. There's different levels upon which you define a language. At the lowest extreme a crude language with no functions will show patterns in its structure, due to how you use it. At the highest extreme you change how the underlying parser works, or maybe even use a graphical language instead. In between is a balance of how predictable the language is (defining new functions rather than a new way to *call* functions), with how powerful it is. You can't simply have both. If the syntax is too open you may as well stick a new parser in every file, and hope they learn to read it.
Today, you'd have to use something like this: foo(""" bar """) It's ugly. It uses the equivalent of braces to tell the parser something that's obvious to us from the indentation. We want it to look more like a normal block: foo: bar However, this is no longer a function call. You can't put a value inside parenthesis without moving the parenthesis to the last line, which is what we don't want. Adding syntax to the language is the normal solution, but we're looking for something open-ended (that embeds a string, not python code). We have to cheat somehow. One way is borrow from decorators: @foo blob: bar Another is to use $blob$ as a placeholder (only legal when the line is just an expression, not a statement): foo($blob$): bar
Such a sweeping change would only move them into the library, not remove them from the language. Indeed, because they're forced into an awkward over-generalized syntax, they become significantly harder to use.
Most programmers don't get it. They'd think reducing the axioms in math from 10 to 5 would make it simpler to learn algebra. Not only would this have no effect on how they use algebra, they probably can't even list the axioms anyway!
You're thinking about a generator function, not a generator expression. A generator expression is the same syntax as a list comprehension, but it uses () rather rather than []. Unless you meant to change how generator expressions work.. but that obviously wouldn't be compatible (nor would any change to a for-statement).
You're assuming statements and expressions won't continue to be done in different styles, which just isn't true. Even though a statement *could* return a value, most of the time it would be used as if it doesn't, so it remains as a special case to be remembered. There's also at least two different modes for a for-loop (lazy vs eager), assuming you drop the list-comp (and 3.0's set/dict comprehensions). That's what I meant by "abstract concept of complexity", although maybe I need a better label for it. You've recategorized thinks to be the same, yet we're expected to use the same way we always did (other than occasionally using new functionality). The categories don't determine complexity, how we use them does! Your changes actually add a substantial amount of complexity, as well as being unimplementable (ambiguous or incompatible behaviour). They never had a chance of being accepted, but I'm trying to explain why they don't work, so you and other budding language developers might give better suggestions in the future.
If you're not willing to solve it properly then don't come whining to us. Some problems *need* drastic solutions.
Depends how ugly you're willing to let it get. There's many ways to do a dispatcher dict, a couple of which have been mentioned already. -- Adam Olsen, aka Rhamphoryncus

On Wed, Sep 10, 2008 at 2:57 PM, Adam Olsen <rhamph@gmail.com> wrote:
class dispatcher(dispatch_dictionary): def x(value): ... def y(value): ... def int_23(value): ... With the proper dispatch_dictionary base class and it's metaclass, dispatcher would become a dictionary with functions that map from strings and integers to function handlers. Never underestimate the power of classes and metaclasses ;) . - Josiah

On Wed, 2008-09-10 at 11:43 -0700, Cliff Wells wrote:
Because I know that the question "what's your use case" will inevitably be forwarded, I will explain the exact use case that has driven me to distraction over this limitation. I developed and maintain a template engine written in Python: http://breve.twisty-industries.com/ As you can see from the example on the main page, Breve is written as an internal DSL (that is, it compiles directly to Python bytecode... Breve templates *are* Python). This means that any limitation in Python is inherently a limitation in Breve. There were several design decisions to be made when designing Breve, the first and foremost being "will Breve templates be full-blown Python programs or will they be limited to expressions?". I chose the latter for a few reasons: 1) Python expressions map more directly to the generated XML/HTML output. In other words, a Breve template has the same structure (if not syntax) as the desired result. 2) It would have required additional syntax within the template to allow for all of Python to be supported. Overall, I'm happy with my choice, but it forces some mental twister when trying to do particular operations in Breve, notably presentation-logic and looping. A loop must be presented as a list comprehension. "If" logic must be done using either 2.5's ternary if operator or using short-circuit logical operations, i.e. "test(value) and (this)". At one point I actually implemented custom tags for doing logical operations, but due to the inability to provide short-circuiting, they were less than satisfactory. This isn't fatal to Breve (albeit annoying), but it made me painfully aware of this self-imposed limitation in Python. It also made me wonder what other concepts this limitation has prevented me from grokking over this last decade. It's generally accepted that a language constrains not only the solutions to a problem, but also the *possible* solutions that a programmer will see. This is as much of a concern to me as any practical concerns today. Regards, Cliff

The only thing I'd like would be anonymous classes (like in java) and anonymous functions: class A(object): def some_method(self): pass def function_that_wants_a_A(a): ... function_that_wants_a_A(A(): def some_method(self): # I really like this feature of java # granted, it makes a lot more sense in the # context of a language like java. # there it's very handy print "this is my derived anonymous class" ) def function_that_wants_a_callback(callback): ... function_that_wants_a_callback(def(a,b,c): # do a lot of stuff that does not fit into a lambda # this reminds a bit of rubies closures ;) ) I don't think anything else concerning statement as expression is in any way necessary or handy.

On Fri, 2008-09-12 at 00:10 +0200, Mathias Panzenböck wrote:
The only thing I'd like would be anonymous classes (like in java) and anonymous functions:
I don't think anything else concerning statement as expression is in any way necessary or handy.
At one time I also perceived a shortcoming in lambda. You could continue to see it this way, or you could see it (like I now do) that it's not lambda that's limited (it effectively allows an arbitrary number of expressions), but rather the fact that a statement cannot be be used as an expression. For instance, the following is currently legal in Python: lambda x, y: ( x + y, x**2, y**2 ) But this is not: lambda x, y: ( if x > y: x else: y ) Whether or not it's illegal is due to lambda being limited or because statements are castrated expressions is all a matter of your point of view. Extending lambda to allow statements would certainly fix *lambda*, but making statements into expressions fixes Python (IMHO, of course). Regards, Cliff

Hey Cliff, I think the only way to move forward in this discussion is if you could write a complete parser for the variant of Python that you'd like to see. I expect that you'll say "but I don't know enough about parser technology to do so" -- but then my response is "then you don't know to understand the difficulty of the task, so shut up." If you want to continue this discussion you'll have to prove that it is possible. That in turn will force you to think about many of the details that you're currently brushing away as irrelevant -- and you'll find that they are actually highly relevant. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, 2008-09-11 at 18:38 -0700, Guido van Rossum wrote:
As I've pointed out, this has already been done (although the project appears dead): http://www.livelogix.net/logix/index.html http://www.livelogix.net/logix/tutorial/3-Introduction-For-Python-Folks.html Logix is actually much more than a Python interpreter, but that's tangential to this discussion. Logix also conveniently runs on the Python VM. It also removes the distinction between statements and expressions and is nearly code-compatible with Python (it has two significant shortcomings and two minor ones, none of which have to do with the topic at hand). http://www.livelogix.net/logix/tutorial/3-Introduction-For-Python-Folks.html
Or I can continue to say what I like and you can choose to ignore it. I obviously believe it's a worthwhile discussion and certainly enjoy the debate, but anyone who doesn't is best served by not joining it. I don't recall bringing this to python-dev as a serious proposal nor submitting it as a PEP, so I think an informal exchange of ideas without spending six months on a functional prototype is perfectly acceptable. That aside, it's already been proven that the syntax and concept is plausible in Logix. Whether or not it could be implemented in CPython is another matter, and whether or not the result would be accepted is yet a another matter.
I'm not brushing them away as irrelevant, I'm setting them aside until it's decided that it's even worth pursuing. I don't know about you, but I don't bother arguing with my wife about the color of a new cars until we're certain we need to buy one. I certainly agree that if I decided to move this beyond idle discussion that a prototype would be a critical aspect and I fully expect there would be many issues to be overcome. I don't believe such a change would be simple, but neither do I believe it's insurmountable. Regards, Cliff

On Thu, Sep 11, 2008 at 9:08 PM, Cliff Wells <cliff@develix.com> wrote:
Well, if you are asking for my personal opinion about the viability of this idea, I think it's not viable, but would like to be proven wrong (or right!) just so that this topic can be put to bed for good. I'm not swayed by the existence of Logix; it appears to be using a completely different way of interpreting whitespace, which seems incompatible with current Python, but they don't seem to give the exact rules, only some hints. Have you used it? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, 2008-09-11 at 21:36 -0700, Guido van Rossum wrote:
Well, that was actually my goal in bringing it up here: I've got something nagging me and the only way I'm going to put it to rest is to hash it out in a roomful of devil's advocates =)
I've imported Logix into a Pylons session and used some of it's features without issue. It's quite probable that this may have only been successful because Pylons itself was precompiled with CPython and the only code affected was the particular module I was compiling at the time. I'm curious which aspect of Logix' whitespace handling you suspect to be incompatible (aside from the line-continuation stuff). In any case, as fascinating as Logix is, I find it mostly interesting (to this debate) as an example of what such a syntax could appear like. Nevertheless, I think I have a compromise: in lieu of writing an interpreter, I'll investigate and properly document Logix (something I've considered in the past anyway). This way we can at least have an implementation to argue over without me needing to quit my day job to make a point ;-) Regards, Cliff

Cliff Wells wrote:
I'm sorry, but the points people are bringing up are not bikeshedding. They are very real points that are more analogous to whether the car has wheels or flies on pixie dust. I'm sorry, but you can't cast aside the subtleties of turning statements into expressions as bikeshedding arguments. Many of the people who gave you very genuine feedback pointed out particularly difficult to rectify problems (like how exactly do you notate a suite if you eliminate the required indent tokens?) and your response to them is that they are missing the point.. You want to change the language, the syntax of that change is *exactly* the point. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On Fri, 2008-09-12 at 02:40 -0400, Scott Dial wrote:
Fair enough. I simply think there's a larger philosophical issue: is the potential for deeply nested code a greater concern than an ever-expanding suite of special-case constructs? That needs to be agreed on before deciding on a specific way of addressing it. Even if we agreed on the larger issue I've forwarded, what I'm suggesting may not be the only way of addressing it.
To be clear, I don't suggest eliminating the required indentation for multi-line blocks (although it seems you could intentionally dodge it through judicious use of the ';' operator and parentheses, much like you can with expressions and operators today).
Well, I'll take the blame for not spending more time separating out the fact that I'm presenting both a problem and a potential solution which has led to a conflation of the two to some degree. I think the solution I've presented helps demonstrate that there *is* a problem and so I find it difficult to present the problem in the solution's absence. As I mentioned in my first post, the current situation has arisen because of a philosophy ("flat is better than nested"). This seems a valid position, but I'm arguing that it has proven over time to have as many pitfalls (in terms of complexity) as the problem it seeks to avoid. Hopefully that makes it more clear why I'm trying to get people to see (and hopefully agree) that there's a problem before bickering over the details of its solution. Regards, Cliff

On Sep 12, 2008, at 11:41 AM, Cliff Wells wrote:
I love some of the constructs of python like list comprehension but yes the new "if" construct don't strike me as a nice adition. And maybe your solution is a good one, but I will have to see it working. If you want to alter pypy to do it you will probably have to improve the parser generator that is being used right now, but you can also try to do it on ironpython or jython.

On Sep 12, 2008, at 11:41 AM, Cliff Wells wrote:
I love some of the constructs of python like list comprehension but yes the new "if" construct don't strike me as a nice adition. And maybe your solution is a good one, but I will have to see it working. If you want to alter pypy to do it you will probably have to improve the parser generator that is being used right now, but you can also try to do it on ironpython or jython.

If I understand this right, then this would become legal too: x = if cond(a): return b So what value is assigned to x when cond(a) is True, what value when it is False? What value does return return? The thing is that return does definitively not return anything, but as a statement, it can be placed everywhere in a block. -panzi

On Sun, 2008-09-14 at 23:34 +0200, Mathias Panzenböck wrote:
return would always be a syntax error outside of a function.
So what value is assigned to x when cond(a) is True, what value when it is False?
x = if False: 1 # evaluates to None x = if True: 1 # evalutates to True x = if True: return 1 # syntax error: return outside of function def foo(): x = if True: return 1 # exits the enclosing function. value of if-expression is moot as it's never reached.
What value does return return? The thing is that return does definitively not return anything, but as a statement, it can be placed everywhere in a block.
Not that it matters, but return always returns something, either a user-specified value or None. Regards, Cliff

Cliff Wells wrote:
Two thoughts: Please elaborate how you like to change the syntax of Python. I like to see some concrete examples how your syntax would look like. I also like to know how your are planing to implement features like lazy evaluation. The if else ternary operator statement is evaluated lazy. The same construct as expression wouldn't be lazy any more. Secondly any syntax change won't happen until we start planing Python 4000 ;) Christian

On Wed, 2008-09-10 at 21:46 +0200, Christian Heimes wrote:
No changes. Simply lifting of a particular restriction.
Why not? a = ( if a > 1 then: long_calculation() else: other_long_calculation() ) Clearly only one of these blocks would be evaluated at runtime.
Secondly any syntax change won't happen until we start planing Python 4000 ;)
Yes, that's my expectation, although hopefully PyPy will make some of these things possible to experiment with well before then =) Cliff

On Sep 10, 2008, at 6:22 PM, Cliff Wells wrote:
Two restrictions, both that statements can be used in place of expressions and that statements now return values. But please explain how to do it in a way that is clear. It's a plus if it is backwards compatible :)
And now ifs return a value
You can do that now in PyPy or any other python version... or maybe use Logix to show us a proof of concept version: http://www.livelogix.net/logix/ Now if I remeber correctly this is already the case of logix, look at: http://www.livelogix.net/logix/tutorial/3-Introduction-For-Python-Folks.html... Why would you need lambda then? couldn't you just write x = def (x,y): return x+y ?

On Wed, 2008-09-10 at 18:52 -0300, Leonardo Santagada wrote:
Well, one or two, depending on how you word it ;-) My position is to simply do away with statements and provide equivalent expressions. As far as backwards-compatibility, this is legal Python: 1 2 3 It doesn't do anything, but it is valid syntax. This demonstrates that expressions can be valid when used independent of a statement, therefore, the following is also legal (assuming "if" were now implemented as an expression): if (a==1): pass
Which you are not required to use, just as today: def foo(x): return x**2 foo(2)
Yes, Logix is quite interesting. Unfortunately the author has discontinued work on it (and was, in fact, unreachable when I last tried to contact him). He also mentioned retargeting to another VM (rather than the Python VM) which made it less appealing to me. It is interesting that you mention Logix, since it demonstrates that it's quite possible to be backward-compatible with this particular change.
Why would you need lambda then? couldn't you just write x = def (x,y): return x+y ?
Yes, although I suspect that this particular change to "def" would be more substantial and "lambda" is currently equivalent. Although this might satisfy GvR's inclination to dump lambda at some future point ;-) Cliff

Arnaud Delobelle schrieb:
OT: I attended a lecture called "abstract machines" and we had to write a little VM (parser, bytecode interpreter) on paper during the test. It was a expression based language and there was a "if ... then ... else ..." AND a "if ... then ..." construct. I asked the Prof. exactly the same question you did. He said: "Oh... well... strike that out." He really didn't notice that this is a problem when he wrote the test. *g* :P (Sorry, for the noise.) -panzi

On 13 Sep 2008, at 23:17, Cliff Wells wrote:
Assuming the return value of "None", I go back to an example I gave earlier: factors = for x in range(2, n): if n % x == 0: x This doesn't work as intended (filtering out the non-factors). How to make it work? The only way I can think of is to make (if 0: 1) return a special "non-value" which loops will then filter out. But then we all know what happens to non-values. So how would you solve this problem?
Cliff
-- Arnaud

On Sun, 2008-09-14 at 07:23 +0100, Arnaud Delobelle wrote:
By writing it properly ;-) factors = for x in range ( 2, n ): if n % x == 0: yield x As I mentioned previously, in order to merge the concept of generator with a for-expression would require bringing in the yield keyword, just as it does now for generator functions. The example you gave would evaluate to None (or perhaps an empty list or generator - that's a detail that would take more consideration before defining it). Regards, Cliff

On 14 Sep 2008, at 07:36, Cliff Wells wrote:
OK, but this seems to me incompatible with current Python: def chain(I, J): for i in I: yield i for j in J: yield j Currently
'-'.join(chain('spam', 'eggs')) 's-p-a-m-e-g-g-s'
With your proposal, the first *expression* (for i in I: yield i) will evaluate to something like iter(I) and then be discarded. Then the second *expression* (for j in J: yield j) will evaluate to something like iter(J) which will be discarded. So chain('spam', 'eggs') will return None.
Regards, Cliff
-- Arnaud

On Sun, 2008-09-14 at 08:36 +0100, Arnaud Delobelle wrote:
It seems you have me on this one. There's clearly other ways to do the same thing, but since backwards-compatibility is a prerequisite I'll have to concede the point. A potential solution would to be to use a different keyword than "yield" to separate current syntax from my proposed syntax (that is, distinguish expression-scoped yield from function-scoped yield), and just side-step the issue, but that seems unappealing to me. Regards, Cliff

On Sun, 2008-09-14 at 01:25 -0700, Cliff Wells wrote:
Actually, a few more minutes of pondering got me a solution I don't find abhorrent. Let yield mean what it currently does. Instead, let "continue" accept an optional parameter (like "return"). Then your above example continues to work and my proposed change would look like: for i in j: continue i I haven't considered this too deeply, but it is at least consistent with "return" syntax and doesn't add a new keyword. Regards, Cliff

On Sun, 2008-09-14 at 01:36 -0700, Cliff Wells wrote:
I'm probably replying way too fast (in fact, I know I am), but I have two thoughts on this: 1) it seems to alter the semantics of "continue" too much when considered against current syntax, but... 2) with the new syntax, it seems not too bad because j = range(3) for i in j: i # evaluates to [] for i in j: continue # evaluates to [] for i in j: continue i # evaluates to [0,1,2] Overall I'm a bit torn on the idea. Thoughts? Regards, Cliff

On Sun, 2008-09-14 at 01:44 -0700, Cliff Wells wrote:
Bah, I knew I was replying too fast. I'm thinking that "continue" would be redefined to mean "yield value and continue" which means that for i in j: continue # evaluates to [ None, None, None ] not [] would seem the most consistent, but I fear it might be less practical (as it would create problems trying to use for/continue inside other expressions, although the effect when for/continue is used as a statement remains fine). Cliff

On Sun, 2008-09-14 at 01:54 -0700, Cliff Wells wrote:
It would *have* to evaluate to an empty list otherwise this code: for i in range(10000000): continue would create a huge list as a side-effect. So the question is, does this seem too inconsistent? Clearly returning [None, None, None] fits nicely with how yield currently works but it's not going to work in this case. Cliff

On 14 Sep 2008, at 09:44, Cliff Wells wrote:
Let's not call it continue, but YIELD for now: for i in J: YIELD i Now this won't work for nested loops. E.g. in current python def flatten(I): for J in I: for j in J: yield j >>> '-'.join(flatten(['spam', 'eggs'])) 's-p-a-m-e-g-g-s' Now say you want to write that inline with a for-expression: '-'.join( for J in I: for j in J: YIELD j ) That won't work because the j's will be accumulated in the inner loop and the outer loop won't accumulate anything, therefore returning an empty iterable. The flatten() example above works because the the scope of the yield statement is clearly defined by the enclosing def statement. To make it work, not only you need a special yield expression but you also need a special for expression: '-'.join( FOR J in I: for j in J: YIELD j ) Here it is clear what happens: the YIELD accumulates values in the FOR loop. Not very coder friendly though :) Now compare with the current syntax: '-'.join(j for J in I for j in J)
Regards, Cliff
-- Arnaud

On Sun, 2008-09-14 at 10:08 +0100, Arnaud Delobelle wrote:
How about this way instead (since for-loop is now an expression): '-'.join( for j in ( for J in I: YIELD J ): YIELD j )
Now compare with the current syntax:
'-'.join(j for J in I for j in J)
Certainly more clear and concise, but since (luckily for me this time) we're maintaining backwards-compatibility, that form would still be available. Cliff

On Sun, Sep 14, 2008 at 11:28 AM, Arnaud Delobelle <arnodel@googlemail.com> wrote:
Agreed. For all of the semantic and syntactic gymnastics and discussion about how statements -> expressions would make Python a better language, all I can conclude from the above is "I'm glad Python doesn't do that." - Josiah

On Sun, 2008-09-14 at 12:36 -0700, Josiah Carlson wrote:
Well, realize that being able to do something doesn't make it the right thing to do in a particular situation. I don't think Arnaud's goal here is to show that functional programming is bad, rather he's forcing me to work out whether or not it could be done without breaking existing syntax (and a fine job he's doing too). The examples we've been working through are testing specific cases. It doesn't mean it would be the recommended idiom for these cases. When I was testing the macro feature for Breve (a bad name for the feature, I now realize), I wrote intentionally horrific code, simply to test that the solution was general enough: http://breve.twisty-industries.com/snippets/macro-madness Would I ever do that in real life? Only for testing =) Cliff

On Sun, 2008-09-14 at 13:02 -0700, Cliff Wells wrote:
Using Breve as an example again: by your reasoning, the mere existence of Breve should be reason enough not to support OO in Python (or at the very least, magic methods). Breve more or less a functional DSL that barely resembles Python but actually *is* Python (in fact, I've seen someone assert that Breve templates could not possibly be actual Python code). Even you were initially convinced that I was doing code-generation. Breve abuses classes and magic methods (albeit in an entirely legal way) to create the illusion of a declarative DSL. Clearly I think this ability is a good thing, but it could also be argued that what I've done is inscrutable and this type of code should not be allowed in Python. In fact, your own example of mimicking a dispatch "table" with a class is arguably object abuse. Personally I don't think it's a bad solution (especially in the absence of a better way), but you've basically mapped what is logically a functional problem onto an object. My point is that just because something *can* be abused, it isn't reason to throw the baby out with the proverbial bath water. The more general a tool is, the more able it is to be used incorrectly (as anyone who's pried a lid off a can of paint with a screwdriver can attest). Regards, Cliff

Cliff Wells wrote:
If it looks that little like Python, I'd say it really is a different language, and you have no right to expect to be able to use the Python compiler as-is to process it. Rather than twisting Python to make it possible to abuse it even further, you'd be better off writing a Breve compiler in Python that produces Python code (or maybe even translates it directly to Python bytecode). -- Greg

On Sep 14, 2008, at 10:35 PM, Greg Ewing wrote:
I was having fun reading the discussion, specially now that it got to the part were most people are starting to realise how ugly python would become, to put it lightly we got to the dificult part (the "how"). But all Cliff wanted is to have better support for DSL in python, that is his real use case. I say why don't we focus on it. The only clean way to support DSL in python in my view would be to support something like pyparsing or ANTLR and make it generate a parser and some simple form of compilation to python bytecodes. How can we support that? Is this even desired? -- Leonardo Santagada santagada at gmail.com

On Sun, 2008-09-14 at 22:58 -0300, Leonardo Santagada wrote:
Actually you can do quite a lot with plain objects and class magic. But if your DSL isn't purely procedural (Breve is declarative-functional) then you are left without much in the way of flow control as most flow control in Python is done via statements. Overall I've already made a decision about my programming future, but certainly better DSL support in Python would be a good thing. Regards, Cliff

Leonardo Santagada wrote:
The way to go is probably to have the parser build an AST. That way you get to reuse the compiler machinery for generating bytecode, without having to worry about the messy details of generating textual Python source. If the DSL->AST transformation is sufficiently context-free, it could probably be specified using some kind of declarative grammar. -- Greg

On Mon, 2008-09-15 at 14:23 +1200, Greg Ewing wrote:
This is the approach Logix took, which is probably why it no longer works under Python 2.5. This is exactly why I find this type of approach unappealing (vs writing an internal DSL). I'm not sure what approach EasyExtend takes, but it's another potential player in this field: http://www.fiber-space.de/EasyExtend/doc/EE.html Regards, Cliff

On Mon, 2008-09-15 at 13:35 +1200, Greg Ewing wrote:
Well *I* think it looks like Python, but then I understand (as can I think anyone familiar with Python's magic methods), but at least superficially it gives a different impression. In any case, Breve is mostly of interest in that it is the project that both made me appreciate the power of expressions and start bumping into Python's second-class support of them.
I think code generation is arguably worse than functional programming.
(or maybe even translates it directly to Python bytecode).
A seriously complicated and highly technical bit of code I'd likely have to fix every few years (and maintain multiple versions of to support multiple Python versions)? No thanks. In any case, I've already choked down my Python snobbery and ordered a Ruby book. Clearly where I want to go in programming isn't available in Python nor will it be any time soon. Regards, Cliff

On Sun, Sep 14, 2008 at 1:47 PM, Cliff Wells <cliff@develix.com> wrote:
Let me make it clear what I was expressing, because you have no idea. I have personal opinions on what I like and dislike. It became clear to me early on that we disagree on language aesthetics. All I was expressing was that I'm glad that Python didn't look like what you were writing. If you've got a problem with my expressing of that, that's fine, you already decided to move onto a different language. But don't claim to know what I was saying or implying about Python as a language and what it should or should not support. The fact is, Breve is implemented using OO in Python. Could it have been done using a functional approach? Sure, but then you would have had to use multi-calling semantics, closures (which is OO in disguise), or code copy/pasting. Breve is not proof that Python should or should not do something, and to claim otherwise (or believe someone is claiming otherwise) is silly.
I knew that Breve was Python, but I thought it was doing something smart by also doing transparent code generation (because there was discussion about the "speed" of Breve) a'la Cheetah/Spitfire, because I believed that you had worked more on the language than you actually had. I was wrong. But that doesn't make Breve a good example of a DSL, it makes it just another internal DSL using Python. As you say yourself, there are many.
Whom has it been argued by? I think the only argument is against adding syntax to make Python purely functional. Using Python syntax and semantics to write interesting features is part of what makes Python great. I've created SQL code generators using __getattr__ and the comparison operators plus __and__ and __or__ for where clauses. And you know what? They looked a million times better than embedding SQL in the code or using stored procedures. None of them would be possible without __magic__ methods.
You didn't even understand my example. I wasn't mapping it into an object, I was using the underlying class creation semantics (specifically metaclasses) to generate a dispatch dictionary. I'll be more clear this time. def dispatcher(name, bases, dict): import __builtin__ dispatch = {} for name, fcn in dict.iteritems(): if callable(name): if '_' in name: typnam, value in name.split('_')[:2] typ = getattr(__builtin__, typnam, None) if not typ: continue dispatch[typ(value)] = fcn else: dispatch[name] = fcn return dispatch If you set the __metaclass__ of a class to the above function, you don't get a class, you get a dictionary. A dispatch dictionary in particular. One that works with arbitrary types (the above only works for builtin types or those that have been inserted into the builtin module). This method shouldn't be new to anyone. I've been doing it since before metaclasses (using functions and locals() instead, because I liked the look better), and I've seen it used in various codebases. You can do similar things to build properties (I have), and any one of a number of other really useful things.
Ahh, but the example that I was complaining about *wasn't supposed to be an abuse*, it was supposed to be an example of "how can we make this really simple Python code work with an expression-only version of Python?" That the *simple* Python became so complicated and ugly in the translation to a non-existing expression-only version of Python is unfortunate. But again, I'm glad Python doesn't look like that. Again, don't try to tell me what my reasoning is, I was expressing an opinion on aesthetics. - Josiah

On Sun, 2008-09-14 at 20:26 -0700, Josiah Carlson wrote:
I clearly read more into it than you intended. My apologies. As for opinions on what Python should or shouldn't be, I'll stop having opinions on it when you do (or probably sooner, since it won't matter to me anyway).
I didn't claim Breve proved anything. I resort to it as an example because it's handy for doing so (and because there are so few such examples in Python).
If it did code generation, then it wouldn't be an internal DSL, it would be an external DSL that happened to target the Python language (Cheetah) or VM (Spitfire) as a runtime. That's why I was so careful to mention it several times.
I meant hypothetically ("could be argued"). In no way did I say that someone here on this list had taken that stance (it hadn't even come up).
Yep, I didn't get it. I'd assumed you were using __call__ (which would seem a much simpler way to achieve the same thing). Frankly this example only cements my belief that this is overly complicated given the problem to be solved.
Again I apologize. But given how many times you did the exact same thing to me here (and rather rudely at that) I think we can call it even. Anyway, I'm done. Regards, Cliff

[snip plenty of wordy arguments about Python, DSLs, misunderstandings, etc, with almost no code, and going nowhere in particular] I'm not a Python guru, I'm not even a professional programmer, worse than that I've never even been a CS student. But I think, humbly, that this thread is much too preoccupied with personal/philosophical opinions about the aesthetics of Python and not enough with *how* to make Python work assuming that statements become expressions. IMHO if we try to work out how, we will come to the conclusion that it would make it a horrible mess. But we can only establish this if we give it a try, and in the process we might learn something useful. And who knows, we might even discover that Python was really meant to be a functional language! After all, this list is called python-ideas. Sorry for the noise. -- Arnaud

On Sun, 2008-09-14 at 19:28 +0100, Arnaud Delobelle wrote:
Ha! I knew I should have gone to bed earlier =) I = [ 'spam', 'eggs' ] '-'.join ( for J in I: YIELD ( for j in J: YIELD j ) ) Now if only I'd followed Guido's suggestion I'd actually be able to test before I post ;-) Cliff

Cliff Wells wrote:
Noooooo..... this is getting worse and worse. You seem to be thinking of syntax issues as though they were purely technical puzzles. The're not -- they're at least as much human-factors issues.
But if you keep all the existing syntax as well, you haven't simplified anything. -- Greg

On Mon, 2008-09-15 at 13:00 +1200, Greg Ewing wrote:
Not that it's terribly relevant to what you say, but: '-'.join ( for J in I: YIELD ( for j in J: YIELD j ) ) is the corrected form (although YIELD continues to be a placeholder of course). In any case, my form does make it slightly more complicated for the simplest case, but makes it much less complicated for more complex cases (for the same reasons a plain for-statement can): x = [ J [ a ] for a in b if c ] vs x = for a in b: if c: continue J [ a ] The complexity has barely increased and yet the second form is already more readable (the formatting of the first reflects what I typically do as listcomps get more complex - overkill here but it demonstrates where I'm going). Given that the listcomp is the only direct form of looping available as an expression, I've written some really ugly looking ones. I've actually taken to commenting open/close brackets simply to help distinguish them. Not to mention, the listcomp's placement of the yielded result before the loop and condition only works well when it's a very simple expression. It doesn't scale well with complexity (not that I think it was meant to).
Yes, that's unfortunate. It might, however, obviate the need for newer ones. Cliff

Cliff Wells wrote:
No, I think you were right the first time. The above looks like it will generate a sequence of iterators, not a flat sequence of j values.
That's a matter of opinion. If you lay out the first one as x = [ J [ a ] for a in b if c ] there's not much difference between them.
You're right, it's not meant to. It's meant for the simple cases where all the syntactic overhead of a full for-statement and list appending code swamps the content. Given that, adding any extra syntactic baggage, even just a 'continue' keyword, reduces its effectiveness for its intended purpose. Also, I think it reads quite nicely with the expression at the beginning. It's modelled after the mathematical notation for describing sets: {x : some conditions on x} -- Greg

On Mon, 2008-09-15 at 14:11 +1200, Greg Ewing wrote:
join() takes care of flattening the final yielded iterator. The first one was actually wrong in that it didn't solve the presented problem.
I thought I'd been saying that all along ;-) I think listcomps are only a *clear* win when they are presented in their simplest form, otherwise it really is just preference. Get into nested listcomps and the readability (or more to the point the comprehensibility) pretty much vaporizes.
So then maybe what we could agree on is that there's a place for both?
Sure, I think listcomps have a place. I still maintain that they are logically redundant if you have if-expressions (not to mention less flexible), but if we decided they were justifiable syntactic sugar then that's fine too. Cliff

Cliff Wells wrote:
But the final iterator is yielding other iterators, not strings, unless I misunderstand the semantics you have in mind.
That depends on what you mean by "nested listcomps". I agree that nesting one entire listcomp inside another tends to look rather confusing: [f(a) for a in [g(b) for b in y]] But that's not the same thing as having nested *loops* within a single listcomp, which I don't think is particularly bad at all: [f(a, b) for a in x for b in y] or if you prefer, [f(a, b) for a in x for b in y]
Yes, but they're also logically redundant even if you don't have statement-expression equivalence, so that's not an argument for merging statements and expressions. -- Greg

On Tue, 2008-09-16 at 11:41 +1200, Greg Ewing wrote:
I think so. For the simplest case s = 'abc' x = for c in s: YIELD c # x = 'a', 'b', 'c' so for the previous example I = [ 'spam', 'eggs' ] for J in I: # J = 'spam' then 'eggs' YIELD ( # evaluate to an iterable for j in J: YIELD j # j is 's' then 'p' then ... ) so we get '-'.join( 's','p','a','m','e','g','g','s' ) maybe this is clearer as for J in I: tmp = for j in J: YIELD j YIELD tmp
Well part of the problem this entire thread has suffered is that there are several related issues being argued simultaneously. Perhaps if I were a better presenter this would have gone differently, but in any case what I was trying to get across is that Python has grown lots of extensions that could be considered redundant and will grow more unless what I see as the seed for these desired extensions is addressed (I believe it to be a desire for better FP support, which merged statements and expressions would largely address). I selected the ternary if-operator, generators and listcomps as examples but I may have overreached a bit. Regards, Cliff

Cliff Wells wrote:
s = 'abc' x = for c in s: YIELD c # x = 'a', 'b', 'c'
You mean that the value of (for c in s: YIELD c) is a tuple? Or that it's an iterator that produces that series of values?
I'm still not getting a clear idea of what semantics you intend for a for-loop-with-YIELD. Neither of the interpretations I suggested above (sequence or iterator) seems to produce this result.
No, that's not any clearer. Can you provide a translation into current, valid Python? -- Greg

On Tue, 2008-09-16 at 13:06 +1200, Greg Ewing wrote:
The latter, although I admit not seeing the importance of the distinction in this case.
My example was a translation of Arnaud's challenge: I = ['spam', 'eggs'] def flatten(I): for J in I: for j in J: yield j >>> '-'.join(flatten(['spam', 'eggs'])) 's-p-a-m-e-g-g-s' It's possible there's a nuance you're seeing that I'm not (and unfortunately we can't test). What are you predicting as the output? Cliff

On Tue, 2008-09-16 at 13:06 +1200, Greg Ewing wrote:
Actually, I think I see the issue. join() is getting a list of iterators (using list notation for simplicity): '-'.join ( [ [ 's','p','a','m'], ['e','g','g','s'] ] ) (I promise you I'm not being intentionally obtuse). Is this the conclusion you were coming to or something else? Cliff

Cliff Wells wrote:
Yes, that's right. Arnaud's original code is a single generator containing two nested loops, whereas your version consists of two nested generators. I think your first translation would have produced the right result, because it iterated over each of the inner generators and re-yielded their results in the outer generator. But that's a very expensive way of going about it. -- Greg

Cliff Wells wrote:
A potential solution would to be to use a different keyword than "yield" to separate current syntax from my proposed syntax
But inserting any kind of keyword into LCs is going to make them ugly and verbose. The point of having LCs in the first place is that they express certain kinds of things very concisely. Adding keywords and colons to them messes that up. Likewise with conditional expressions. It would be disappointing if, instead of x = a if b else c we had to write x = if a: b else: c Colons in the middle of expressions look ugly, IMO. -- Greg
participants (13)
-
Adam Olsen
-
Arnaud Delobelle
-
Blake Winton
-
Christian Heimes
-
Cliff Wells
-
George Sakkis
-
Greg Ewing
-
Guido van Rossum
-
Josiah Carlson
-
Leonardo Santagada
-
Mathias Panzenböck
-
Piotr Wysocki
-
Scott Dial