[Python-ideas] Statements vs Expressions... why?

[Adam Olsen]

On Wed, Sep 10, 2008 at 7:14 PM, Cliff Wells <cliff at develix.com> wrote:
> On Wed, 2008-09-10 at 17:16 -0600, Adam Olsen wrote:
>> On Wed, Sep 10, 2008 at 4:39 PM, Cliff Wells <cliff at develix.com> wrote:
>> > On Wed, 2008-09-10 at 15:57 -0600, Adam Olsen wrote:
>> >> On Wed, Sep 10, 2008 at 3:18 PM, Cliff Wells <cliff at develix.com> wrote:
>> >> > Further, I feel that this limitation forces programmers into using hacks
>> >> > and magic or overly spread-out code, which itself leads to readability
>> >> > concerns.  Having used Python for around a decade, I'm quite aware of
>> >> > the fact that you can happily write tons and tons of nice code with
>> >> > Python in its current state.  However, because of the direction of my
>> >> > work (developing a internal DSL in Python) I've suddenly become aware of
>> >> > this glass ceiling.  I'd bumped into it before back when I was doing a
>> >> > lot of GUI development, but blamed it on lambda rather than realizing
>> >> > that it wasn't lambda so much as what I am bringing up now.
>> >>
>> >> Python is not intended for DSLs.  Really, don't do it.  Python is for
>> >> python code.
>> >
>> > The DSL I work on *is* Python code.  And really, this is the first time
>> > I've heard anyone assert anything like this.  Python is a
>> > general-purpose language.  It's not VBA ;-)
>> >
>> > DSL's are an extremely useful concept.  To summarily dispatch the whole
>> > of them with such an assertion is pretty much bolstering my argument:
>> > you've just asserted that Python is inherently limited in scope.
>>
>> I agree, DSL's are useful.  They're just not something python
>> currently supports well.
>
> Well, this is where we agree.  Where we seem to disagree is whether or
> not something needs to be done about it =)
>
> At some level, the whole concept of OO programming *is* DSL support.  I
> consider "DSL" to mean "mapping of abstract (language) constructs to
> real-world (domain-specific) constructs".  This type of DSL construction
> is quite-well supported by Python (and, in fact, is the way Breve is
> constructed under the hood).   However, this support is hampered (to
> some degree) by having a somewhat inflexible syntax.

Aye, but.. it's hard to explain.  There's different levels upon which
you define a language.  At the lowest extreme a crude language with no
functions will show patterns in its structure, due to how you use it.
At the highest extreme you change how the underlying parser works, or
maybe even use a graphical language instead.  In between is a balance
of how predictable the language is (defining new functions rather than
a new way to *call* functions), with how powerful it is.

You can't simply have both.  If the syntax is too open you may as well
stick a new parser in every file, and hope they learn to read it.


>> They're the use-case you need to justify
>> against the substantial changes you propose.
>
> I guess I don't see it as substantial to people who don't wish to use it
> (although it's quite substantial to people who do).  Overall, I think
> this is why I feel the change doesn't require a huge amount of
> justification: you aren't *forced* to use it, but if you need it, it's
> huge.  It doesn't impose any significant stylistic change on people who
> prefer the imperative style, but it opens vast doors for people wishing
> to approach problems from a functional path.
>
>>  Maybe it's worth it, or
>> maybe it's better to add an indent-sensitive string literal that would
>> then allow *arbitrary* DSL syntax?
>
> Hm, I'd have to see an example (even contrived) of what you mean here.

Today, you'd have to use something like this:

    foo("""
        bar
    """)

It's ugly.  It uses the equivalent of braces to tell the parser
something that's obvious to us from the indentation.  We want it to
look more like a normal block:

    foo:
        bar

However, this is no longer a function call.  You can't put a value
inside parenthesis without moving the parenthesis to the last line,
which is what we don't want.  Adding syntax to the language is the
normal solution, but we're looking for something open-ended (that
embeds a string, not python code).  We have to cheat somehow.  One way
is borrow from decorators:

    @foo
    blob:
        bar

Another is to use $blob$ as a placeholder (only legal when the line is
just an expression, not a statement):

    foo($blob$):
        bar


>> >> If you want another language, write your own parser.  I
>> >> hear lisp is simple to parse, and has no annoying statements to hold
>> >> you back!
>> >
>> > Ah, except Python is the language I like in every way, *except* for this
>> > one particular wart.   Really, had I not entered new programming domains
>> > and tried to take Python with me, I'd probably never have had a
>> > complaint.
>> >
>> > Also, external parsers defeat the entire reasoning behind internal DSL's
>> > (taking advantage of an established VM/compiler, requiring users to
>> > learn a new syntax in addition to their primary programming language).
>>
>> I appreciate what you're saying here, and feel much the same way about
>> my own pet-features, but this is a really poor argument.  Everybody
>> has just one wafer-thin feature they'd like to add.
>
> Again, this is only wafer-thin if you don't want to use it (and that is
> part of its appeal - it's actually rather non-intrusive to classical
> statement-oriented programming).
>
> What I see happening in Python is exactly what you appear to be arguing
> against.  Little specialized features are added one after the other to
> satisfy particular needs, when what is actually needed is one sweeping
> change that would make those features redundant.

Such a sweeping change would only move them into the library, not
remove them from the language.  Indeed, because they're forced into an
awkward over-generalized syntax, they become significantly harder to
use.


>> >> Seriously though, there is an advantage to basing so much on
>> >> statements rather than expressions.  We're specialized for one
>> >> statement per line, which is the most common case,
>> >
>> > Clearly it's the most common case in existing Python code since nothing
>> > else is allowed.   But frankly, even Javascript doesn't follow this
>> > idiom anymore.   Expression-oriented languages have seen a rebirth for
>> > good reasons (although I admit I'm none-to-fond of many of them, for
>> > various reasons).
>>
>> Again, "XYZ language has it" is an ineffective argument.
>
> Not really what I was trying to say.  What I was saying is that
> expression-oriented languages are rising (practically from the grave C
> put them in) because they are inherently useful and typically more
> powerful than their imperative counterparts.   Where they've tended to
> suffer is the place where Python shines: readability.   What I'm
> claiming is that Python's readability is not due to its imperative
> nature, rather due to it's whitespace-oriented syntax and lack of
> line-noise, so Python could literally become the best of both worlds
> were it to shed its imperative roots.

Most programmers don't get it.  They'd think reducing the axioms in
math from 10 to 5 would make it simpler to learn algebra.  Not only
would this have no effect on how they use algebra, they probably can't
even list the axioms anyway!


>> There's actually a problem with trying to merge a for-loop and a list
>> comprehension.  A generator expression is the canonical generic form,
>> but [genexp] would create a list containing a single genexp object.
>>
>> Likewise, a for-loop would become lazy, so without some extra syntax
>> to evaluate it (and trigger the side effects!), your programs would
>> cease to do anything.
>
> I don't think so.  Think about how Python currently defines a generator:
> the presence of the "yield" keyword within a function.  I think this
> same logic could be applied to a for loop (but I'm willing to be
> corrected if you see a problem).  A for loop without "yield" is more or
> less what it is today (except it might evaluate to []).  A for loop with
> "yield" is a generator (and by extension, useful as an expression).

You're thinking about a generator function, not a generator
expression.  A generator expression is the same syntax as a list
comprehension, but it uses () rather rather than [].

Unless you meant to change how generator expressions work.. but that
obviously wouldn't be compatible (nor would any change to a
for-statement).


>> So you see, despite significant and obvious similarity between the
>> features, there's some important differences as well.  These are so
>> obvious from the contexts that you don't even think of them, so
>> clearly the mental load of having 3 (soon to be 5) different forms of
>> looping is not all that great.
>>
>> Mental load is what really counts, not some abstract concept of complexity.
>
> This isn't abstract.  It's a matter of countable constructs, rules, and
> exceptions to rules.   Consider:
>
> Rules in statement-oriented Python:
>
> 1) distinguishes between statements and expressions
> 2) expressions return a value, statements do not
> 3) expressions can be used inside other expressions and statements,
> statements cannot
> 4) there is an if statement
> 5) there is an if expression (ternary if operator)
> 6) there are for loop statements
> 7) there are list comprehensions
> 8) there are generators ("yield" defines when a function is a generator)
>
> Equivalent rules in expression-oriented Python:
>
> 1) if expression returns a value (that can be ignored or used in an
> expression)
> 2) for expression returns an empty list or a list if it's a generator
> (has "yield" keyword) that can be ignored or used in an expression.
>
> Which has more "mental load"?

You're assuming statements and expressions won't continue to be done
in different styles, which just isn't true.  Even though a statement
*could* return a value, most of the time it would be used as if it
doesn't, so it remains as a special case to be remembered.

There's also at least two different modes for a for-loop (lazy vs
eager), assuming you drop the list-comp (and 3.0's set/dict
comprehensions).

That's what I meant by "abstract concept of complexity", although
maybe I need a better label for it.  You've recategorized thinks to be
the same, yet we're expected to use the same way we always did (other
than occasionally using new functionality).  The categories don't
determine complexity, how we use them does!

Your changes actually add a substantial amount of complexity, as well
as being unimplementable (ambiguous or incompatible behaviour).  They
never had a chance of being accepted, but I'm trying to explain why
they don't work, so you and other budding language developers might
give better suggestions in the future.


>> >> Now there are some use cases that suffer here, including the one you
>> >> just gave: defining a dispatch dict with the functions inline.  The
>> >> best I could do is define the dict first, then stick a decorator on
>> >> each function to register them.  That's still ugly though.  A creative
>> >> solution is needed, but none come to mind.
>> >
>> > That's because there is none.  And this is my fundamental problem: it's
>> > not so much that it's hard to do in Python, it's that you *cannot* do it
>> > in Python.  No amount of creativity, time, or experience will help, and
>> > this is disappointing.
>>
>> I didn't mean to do it in Python.  I meant to modify the language.
>
> I don't consider modifying the language an acceptable solution for most
> programmers.  It's a maintenance nightmare.

If you're not willing to solve it properly then don't come whining to
us.  Some problems *need* drastic solutions.


>> >> An example where this has happened before is the with-statement, which
>> >> is spectacularly successful IMO.  Now, you may notice it could have
>> >> been done in a library rather than core syntax if generic anonymous
>> >> blocks were allowed — so what?  The library is still part of the
>> >> language!  It's still something that has to be learned.  And the
>> >> syntax would be substantially uglier using a generic mechanism, rather
>> >> than the specialized with-statement syntax.
>> >
>> > The "so what" is that it could *only* be implemented by the core devs.
>> > It was not possible for an average (or even above-average) Python
>> > programmer to write such a library, whereas it *could* have been had the
>> > language not prohibited it.
>>
>> As important as it is to extend the language via a library, somewhere
>> you need to draw the line and start modifying the syntax or other
>> fundamental builtins.
>
> Yes and no.  I believe it should be possible to prototype almost any
> construct via a library.  Whether the language should then embrace the
> concept embodied in that prototype to provide better integration,
> performance, or simply syntactic sugar, can then be argued much more
> fruitfully.  If a language prevents you from creating such prototypes
> then I think

Depends how ugly you're willing to let it get.  There's many ways to
do a dispatcher dict, a couple of which have been mentioned already.


-- 
Adam Olsen, aka Rhamphoryncus