[Python-ideas] Statements vs Expressions... why?

Thu Sep 11 11:05:25 CEST 2008

On Wed, 2008-09-10 at 22:47 -0700, Josiah Carlson wrote:
> On Wed, Sep 10, 2008 at 9:34 PM, Cliff Wells <cliff at develix.com> wrote:
> > On Wed, 2008-09-10 at 18:30 -0700, Josiah Carlson wrote:
> >> On Wed, Sep 10, 2008 at 6:14 PM, Cliff Wells <cliff at develix.com> wrote:

> > Again I assert the opposite: that Python is currently forced to explain
> > the arbitrary distinction that currently exists and that a significant
> > amount of extra syntax and language features exist to work around that
> > distinction.  People may not clamor for the change but there's quite a
> > few newcomers to Python who must have the distinction explained to them.
> 
> For people who have gone through the Python tutorial or any one of the
> many good 'learning Python' texts (online, paper, free, and paid-for)
> the distinction has already been explained.  For those who refuse (for
> some reason) to go through one of those resources, that's not an issue
> with Python, that's an issue with the user.

You won't catch me disagreeing with this, but I was trying (clumsily) to
point out that the distinction violates (albeit subtly) the principal of
least surprise and so requires explanation (even if the explanation is
very simple).

> As a data point; everyone that I've taught Python over the years
> (undergrads and interns who only ever used Java before to 30-year
> programming veterans of apl/cobol/fortran), not a single one of them
> has ever had a problem with the difference between statements and
> expressions.  It is possible that I've only ever worked with gifted
> students and coworkers, though this is unlikely.

Without any supporting data, my hunch is that most people who migrate to
Python come from other imperative languages (myself included) and so
don't find the distinction foreign. 

> What I've found (historically) is that any time I find myself trying
> to do something that is not available within the syntax of Python,
> it's usually because it's a bad idea.  In your case, you want (for
> example) the following to be valid Python...
> 
> lfcn = lambda x: (
>     if x:
>         True
>     else:
>         False)
> 
> Similarly...
> 
> lfcn = lambda x: (
>     for i in xrange(x):
>         yield fcn(i, x, ofcn(i))
>     )
> 
> Why?  Initially it is for a templating language that you have designed
> that makes these kinds of things difficult to do during an automatic
> translation from working Python source code into compiled source code.
>  Presumably something of the form...
> 
> html [ body [ a (href='http://python.org') ["Python!"] ] ]
> 
> To be converted into something like...
> 
> def generate_html(out):
>     out.write('''<html>
>   <body>
>     <a href="http://python.org">Python!</a>
>   </body>
> </html>''')

Actually, it's much simpler than that.   The HTML tags are objects which
simply get serialized into their text representation.  I specifically do
not generate Python source.  In fact, you can simply do something like
this in the interactive interpreter:

>>> print html [ body [ a (href='http://python.org') ["Python!"] ] ]
>>>
<html><body><a href="http://python.org">Python!</a></body></html>

This is more or less how Nevow Stan does it as well.

> But I digress.  At some point, you acknowledge that your templating
> language is constrained by your programming language of choice (as is
> the case for systems that use the base programming language for
> syntax, because of the convenience of parsing), or really, your
> templating language is constrained because you refuse to write your
> own parser.  Don't get me wrong, the convenience of writing a
> templating language in Python syntax is amazing...but you've noticed
> that your compilation steps are hampered by Python's immediate
> execution of code.  In fact, your own renderers are a work-around for
> something that is /relatively/ natural in Cheetah/Spitfire:
> 
>     $for $link in $links:
>         <a href="$link.url">$link.label</a>
> 
> Which is generally compiled into the following (with a bit of extra
> garbage, but you get the idea)...
> 
>     for link in links:
>         out.write('''<a href="%s">%s</a>'''%(link.url, link.label))
> 
> Now, I loathe writing Cheetah/Spitfire because I loathe writing html,
> which is why I wrote my own templating system, but there are
> work-arounds.  For example, if you want to stick with Python syntax...
> 
> For("$link", "$links", "<a href="$link.url">$link.label</a")
> 
> Now you just need to write your For factory and flattener, which
> produces the right code (easy), and with a little work, you can even
> make for loops composable, add ifs, etc.  I know, it's not as elegant
> (for you) in this situation as statements becoming expressions, but it
> will work today. (note that pie-in-the-sky best-case scenario is 18+
> months for Python 3.1, but more likely 15 years for Python 4 ;) )

I've actually employed several methods for working around this issue in
Breve: 

For looping:
 - list comprehensions in place of for loops (arguably the natural choice in 
   Python, but list comps tend toward unreadability quickly). 
 - tag multiplication e.g. 
    li(class_='$class')['$value'] * [{'class': 'even', 'value': 1 }, {'class': 'odd', 'value': 2}]
    results in <li class="even">1</li><li class="odd">2</li>

For conditionals:
 - ternary if-operator for Python >=2.5
 - test(val) and [ block ] in place of if-statement (for <2.5 support)
   e.g. test (session.logged_in) and div['hello', session.user]

For both:
 - renderers - push the logic back into Python proper (as you suggested)

At one point, I even wrote a set of custom tag classes that allowed things like this:

switch ( x ) [
    case ( 1 ) [ 'x is 1' ],
    case ( 2 ) [ 'x is 2' ],
    case ( 3 ) [ 'x is 3' ],
    default [ 'x is not in list' ]
]

While these worked superficially, they were ill-fated due to lack of
short-circuiting/lazy evaluation (which rendered them both inefficient
and incorrect).

At the end of the day, while these sorts of methods work, they feel like
what they are: workarounds (although 2.5 helps a bit with the ternary
if-operator).  This might be nit-picking on my part, but I don't think
I'm especially unusual in my desire to do the "Right Thing".

Ultimately though, whether/how I solved these issues can be considered
two ways: from a practical point-of-view (what you are espousing) or
from a purist's point-of-view (which I know is not a popular view
amongst Pythonistas).  I mentally weighed the pros and cons of having
statements and ended up finding the pros lacking (ultimately coming down
to enforcing an imperative style).

> That's not necessary.  You've already solved your problem with
> renderers, and I've pointed out another method for getting some
> behavior in-line (If you are careful, you could probably build all of
> Python out of constructs similar to the For above).  Of course there's
> always the option of just using some version of the output of the
> compiler/ast (I have done as much to pull out class definitions,
> function definitions, etc., in the editor that I have written).  You
> could even write a tool for doing the conversion ;) .

I've considered this, but frankly find AST manipulation a bit daunting
(and even more of a workaround than the above pure-Python solutions).
I've also considered trying out EasyExtend to accomplish my goals but it
still doesn't feel quite right.

> > The fact that lambda was so bitterly defended against GvR's strong
> > desire to remove it should be hint enough that a significant number of
> > Pythonistas are interested in functional and expression-oriented
> > programming.
> 
> Indeed!  I have a penchant for the use of list comprehensions,
> generator expressions, map, etc. - when they are reasonably
> applicable.  The question is always: what is reasonably applicable.

As you might expect, I subscribe to the "consenting adult" perspective.
I think "reasonably applicable" is a determination best left to the
programmer managing a particular piece of code rather than the
programming language.  Readability matters most to those who have to
read it (which is, more often than not, the person who wrote it).  What
is readable to one isn't always equally readable to someone else.  
For example, list comprehensions used to appear completely inside-out to
me whereas map/filter/reduce was as simple as could be, and yet I've
hear GvR claim on several occasions that the exact opposite is true for
him.  Most likely we'd differ just as much on any sufficiently
non-trivial bit of code.  But at the end of the day, GvR probably won't
need to worry about how readable any bit of my code is, it only matters
to me.  And even if someone else were to need to read my code, it's a
toss-up which way he'd find more readable.  Ultimately I don't think
there's a clear win here either way, but I feel that Python needlessly
enforces a single vision to the exclusion of other equally valid
visions.

> > In any case, I actually got the response I expected and ultimately I
> > expect the discussion here was probably far more enlightened than I
> > would expect on c.l.py.
> >
> > It seems most Pythoneers work in particular domains where this isn't an
> > issue, are satisfied with the workarounds, or simply are unable to see
> > there's a vast world of elegant solutions beyond what imperative
> > languages can describe.  Unfortunately this only confirms my (rather
> > reluctant) expectation that I'll be forced to move to a more expressive
> > language to satisfy my programming itches.
> 
> Pythoneers do see that there are /alternate/ solutions to what is
> currently available within Python, hence why people tend to like to
> write parsers in Python ;) .  One thing you may want to look into is
> that you aren't using the full expressive power of Python in your
> templating language, and because of this, it is actually quite easy to
> write a parser for your subset, adding all of the necessary behavior
> and translation.  I used DParser in Python for a few years and loved
> it, though it seems like the home page for it is no longer available.

Well, as far as Breve goes, I think there are adequate workarounds.
What has disturbed me was finding I that I actually needed workarounds
(something I haven't needed a lot of in my years of Python programming).
This got me to thinking about the dichotomy of statements and
expressions in Python and I was irrevocably drawn to the conclusion that
this distinction is not as inconsequential as I had once believed.

> On the one hand, yes, you are currently constrained by the limits of
> Python's syntax.  And yes, you could move to Ruby, Boo, etc.  Or you
> could try alternate constructs (For, If, Def, etc.), write your own
> parser (which isn't hard), or shrug and admit to yourself that the
> limitations really aren't all that bad.  Inconvenient, sure.  The end
> of breve in Python?  Not necessarily.

Admittedly, this is largely a matter of taste.  But of course, many
seemingly critical decisions about programming are.  Unfortunately, once
I'd had my eyes opened to this limitation, it's now impossible for me
not to notice. 

For example, when I see how much more natural Twisted's deferreds appear
in MochiKit than in Python (where they originated), I can't help but
believe that Twisted's failure to reach a larger audience is at least
partially because that approach isn't as easily expressed in Python as
it would be in a more expression-oriented language.   In this case, it's
not even that it can't be done (it clearly can), it's that the approach
feels forced rather than natural.  

Overall, I'm left with the nagging suspicion that because of my language
choice, I'm missing an important paradigm that would elevate my
programming to the next level.

In any case, I expect I'll stew in my current situation for a while
longer at least.  The unfortunate fact remains that most of the
languages that speak to me (e.g. Boo and Io) don't have the broad range
of libraries and frameworks available that Python does and ultimately
this tends to outweigh my esoteric desires.

Regards,
Cliff