[Python-ideas] Statements vs Expressions... why?

[Cliff Wells]

On Wed, 2008-09-10 at 17:16 -0600, Adam Olsen wrote:
> On Wed, Sep 10, 2008 at 4:39 PM, Cliff Wells <cliff at develix.com> wrote:
> > On Wed, 2008-09-10 at 15:57 -0600, Adam Olsen wrote:
> >> On Wed, Sep 10, 2008 at 3:18 PM, Cliff Wells <cliff at develix.com> wrote:
> >> > Further, I feel that this limitation forces programmers into using hacks
> >> > and magic or overly spread-out code, which itself leads to readability
> >> > concerns.  Having used Python for around a decade, I'm quite aware of
> >> > the fact that you can happily write tons and tons of nice code with
> >> > Python in its current state.  However, because of the direction of my
> >> > work (developing a internal DSL in Python) I've suddenly become aware of
> >> > this glass ceiling.  I'd bumped into it before back when I was doing a
> >> > lot of GUI development, but blamed it on lambda rather than realizing
> >> > that it wasn't lambda so much as what I am bringing up now.
> >>
> >> Python is not intended for DSLs.  Really, don't do it.  Python is for
> >> python code.
> >
> > The DSL I work on *is* Python code.  And really, this is the first time
> > I've heard anyone assert anything like this.  Python is a
> > general-purpose language.  It's not VBA ;-)
> >
> > DSL's are an extremely useful concept.  To summarily dispatch the whole
> > of them with such an assertion is pretty much bolstering my argument:
> > you've just asserted that Python is inherently limited in scope.
> 
> I agree, DSL's are useful.  They're just not something python
> currently supports well.  

Well, this is where we agree.  Where we seem to disagree is whether or
not something needs to be done about it =)

At some level, the whole concept of OO programming *is* DSL support.  I
consider "DSL" to mean "mapping of abstract (language) constructs to
real-world (domain-specific) constructs".  This type of DSL construction
is quite-well supported by Python (and, in fact, is the way Breve is
constructed under the hood).   However, this support is hampered (to
some degree) by having a somewhat inflexible syntax.

> They're the use-case you need to justify
> against the substantial changes you propose. 

I guess I don't see it as substantial to people who don't wish to use it
(although it's quite substantial to people who do).  Overall, I think
this is why I feel the change doesn't require a huge amount of
justification: you aren't *forced* to use it, but if you need it, it's
huge.  It doesn't impose any significant stylistic change on people who
prefer the imperative style, but it opens vast doors for people wishing
to approach problems from a functional path.

>  Maybe it's worth it, or
> maybe it's better to add an indent-sensitive string literal that would
> then allow *arbitrary* DSL syntax?

Hm, I'd have to see an example (even contrived) of what you mean here.

> 
> >> If you want another language, write your own parser.  I
> >> hear lisp is simple to parse, and has no annoying statements to hold
> >> you back!
> >
> > Ah, except Python is the language I like in every way, *except* for this
> > one particular wart.   Really, had I not entered new programming domains
> > and tried to take Python with me, I'd probably never have had a
> > complaint.
> >
> > Also, external parsers defeat the entire reasoning behind internal DSL's
> > (taking advantage of an established VM/compiler, requiring users to
> > learn a new syntax in addition to their primary programming language).
> 
> I appreciate what you're saying here, and feel much the same way about
> my own pet-features, but this is a really poor argument.  Everybody
> has just one wafer-thin feature they'd like to add.

Again, this is only wafer-thin if you don't want to use it (and that is
part of its appeal - it's actually rather non-intrusive to classical
statement-oriented programming).  

What I see happening in Python is exactly what you appear to be arguing
against.  Little specialized features are added one after the other to
satisfy particular needs, when what is actually needed is one sweeping
change that would make those features redundant.

> >> Seriously though, there is an advantage to basing so much on
> >> statements rather than expressions.  We're specialized for one
> >> statement per line, which is the most common case,
> >
> > Clearly it's the most common case in existing Python code since nothing
> > else is allowed.   But frankly, even Javascript doesn't follow this
> > idiom anymore.   Expression-oriented languages have seen a rebirth for
> > good reasons (although I admit I'm none-to-fond of many of them, for
> > various reasons).
> 
> Again, "XYZ language has it" is an ineffective argument.

Not really what I was trying to say.  What I was saying is that
expression-oriented languages are rising (practically from the grave C
put them in) because they are inherently useful and typically more
powerful than their imperative counterparts.   Where they've tended to
suffer is the place where Python shines: readability.   What I'm
claiming is that Python's readability is not due to its imperative
nature, rather due to it's whitespace-oriented syntax and lack of
line-noise, so Python could literally become the best of both worlds
were it to shed its imperative roots.

> >>  and it allowed us
> >> to have extraneous semicolons, braces, or whatever.
> >
> > Not following this.  You mean to *not* have extraneous syntax?
> 
> Consider a loop in C vs in python
> 
> for (i = 0; i < 50; i++) {
>     a();
>     b();
> }
> 
> for i in range(50):
>     a()
>     b()
> 
> We avoid the semicolons and braces because the newline and indentation
> indicate the same thing. 

That's what I thought you meant.  You dropped the "not" in your original
statement.

>  That's what a statement it.  Perhaps you can
> retain this with your proposal, but only because a statement is still
> the default, with expression as a rarely used option.

Exactly.  My point remains that the imperative *style* might still be
preferred (and encouraged) because it *is* inherently simpler to
understand, but that it shouldn't be enforced because it's also
inherently limiting (not surprisingly).   I think the community as a
whole has been successful in forwarding "Pythonic" idioms even when the
language allows "unPythonic" coding without resorting to b&d tactics
within the language itself.  I think the same external discipline can be
applied to this concept without limiting Python's applicability in other
domains. 

> >> Readability benefits, style consistency benefits.
> >
> > I strongly disagree.  The artificial distinction between statements and
> > expressions is the definition of *inconsistent*.  Why do we have two
> > versions of the "if" conditional?  Why do we have "for" loops *and* list
> > comprehensions?  They express the same ideas, but the limitations in one
> > required growing the language another direction.   In short, we could
> > have had a single, more powerful construct in place of two lesser
> > constructs and simultaneously had less syntax to memorize and more
> > consistency across the board.
> 
> I was referring to consistency of the programs, not the language
> itself.  No silly arguments about brace position because they *are no
> braces*.

But you are assuming that eliminating statements somehow requires extra
syntax.  I think Logix, which was mentioned previously, adequately
demonstrates that this is not the case.

The only "extra" syntax that would need to be used would be parentheses
for logical grouping of expressions.  This syntax already exists in
Python and is already used for the same.

> There's actually a problem with trying to merge a for-loop and a list
> comprehension.  A generator expression is the canonical generic form,
> but [genexp] would create a list containing a single genexp object.
> 
> Likewise, a for-loop would become lazy, so without some extra syntax
> to evaluate it (and trigger the side effects!), your programs would
> cease to do anything.

I don't think so.  Think about how Python currently defines a generator:
the presence of the "yield" keyword within a function.  I think this
same logic could be applied to a for loop (but I'm willing to be
corrected if you see a problem).  A for loop without "yield" is more or
less what it is today (except it might evaluate to []).  A for loop with
"yield" is a generator (and by extension, useful as an expression).

> So you see, despite significant and obvious similarity between the
> features, there's some important differences as well.  These are so
> obvious from the contexts that you don't even think of them, so
> clearly the mental load of having 3 (soon to be 5) different forms of
> looping is not all that great.
> 
> Mental load is what really counts, not some abstract concept of complexity.

This isn't abstract.  It's a matter of countable constructs, rules, and
exceptions to rules.   Consider:

Rules in statement-oriented Python:

1) distinguishes between statements and expressions
2) expressions return a value, statements do not
3) expressions can be used inside other expressions and statements,
statements cannot
4) there is an if statement 
5) there is an if expression (ternary if operator)
6) there are for loop statements
7) there are list comprehensions
8) there are generators ("yield" defines when a function is a generator)

Equivalent rules in expression-oriented Python:

1) if expression returns a value (that can be ignored or used in an
expression)
2) for expression returns an empty list or a list if it's a generator
(has "yield" keyword) that can be ignored or used in an expression.

Which has more "mental load"?

> >> Now there are some use cases that suffer here, including the one you
> >> just gave: defining a dispatch dict with the functions inline.  The
> >> best I could do is define the dict first, then stick a decorator on
> >> each function to register them.  That's still ugly though.  A creative
> >> solution is needed, but none come to mind.
> >
> > That's because there is none.  And this is my fundamental problem: it's
> > not so much that it's hard to do in Python, it's that you *cannot* do it
> > in Python.  No amount of creativity, time, or experience will help, and
> > this is disappointing.
> 
> I didn't mean to do it in Python.  I meant to modify the language.

I don't consider modifying the language an acceptable solution for most
programmers.  It's a maintenance nightmare.

> >> An example where this has happened before is the with-statement, which
> >> is spectacularly successful IMO.  Now, you may notice it could have
> >> been done in a library rather than core syntax if generic anonymous
> >> blocks were allowed — so what?  The library is still part of the
> >> language!  It's still something that has to be learned.  And the
> >> syntax would be substantially uglier using a generic mechanism, rather
> >> than the specialized with-statement syntax.
> >
> > The "so what" is that it could *only* be implemented by the core devs.
> > It was not possible for an average (or even above-average) Python
> > programmer to write such a library, whereas it *could* have been had the
> > language not prohibited it.
> 
> As important as it is to extend the language via a library, somewhere
> you need to draw the line and start modifying the syntax or other
> fundamental builtins.  

Yes and no.  I believe it should be possible to prototype almost any
construct via a library.  Whether the language should then embrace the
concept embodied in that prototype to provide better integration,
performance, or simply syntactic sugar, can then be argued much more
fruitfully.  If a language prevents you from creating such prototypes
then I think 

> The is a universal tenant, applying even to
> lisp (which has very little syntax, rather than extensible syntax).
> 
> This is why we have a+b, rather than add(a, b).  More syntax when it's worth it.

But the language didn't prevent you from creating the hypothetical add()
function.  It merely provided syntactic sugar for making it more
readable.

I feel we're getting slightly side-tracked here =)

> So it's not that we don't want to allow extensibility - quite the
> opposite.  It's that we want the common statement to be simpler, and
> the extra syntax hasn't been justified for your use cases.

Once again, I am forwarding *zero* extra syntax.  In fact I am
suggesting *less* syntax overall and an accompanying reduction in rules
regarding the remaining syntax.  I am suggesting removing (or
deprecating) syntactic additions such as the ternary operator in favor
of extending the power (or more to the point, removing arbitrary
limitations) of the existing core language.

Regards,
Cliff