Code block literals

[Alex Martelli]

Dave Benjamin wrote:
   ...
>>> for line in file('input.txt'):
>>>      do_something_with(line)
>> 
>> I don't see that there's anything "implicit" in the concept that a
>> special operation works as indicated by its syntax.  I.e., I do not
   ...
> What's implicit to me is that the use of an iterator is never specified.

It could be, but it would be redundant, because the for-in syntax
_always_ uses iterators -- so,
    for x in y:
and
    for x in iter(y):
and
    for x in iter(iter(y)):
and so on, ad nauseam, are all equivalent.  All except the first form
are also redundant.

> For instance, we could (and I'm *not* suggesting this) do this:
> 
> iterator = file('input.txt')

You forgot to call iter (which the for loop does), so, depending
on whether a file object HAS-AN iterator (as in 2.2) or IS-AN
iterator, you could be out of luck -- e.g.:

[alex at lancelot src23x]$ python2.2
Python 2.2.2 (#1, Oct 24 2002, 11:43:01)
   ...
>>> f=file('/tmp/ba')
>>> print f.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'file' object has no attribute 'next'
>>>

(in 2.3, you'd be lucky;-).

> while iterator.has_next():
>     line = iterator.next()
>     do_something_with(line)

That wouldn't be the definition of the for loop (and in general it
would be hellish to make it work -- that "predictive" has_next might
be a real bear to implement, with sane semantics, on certain
iterators which "consume" data and can't put it back).

However, you COULD write something like:

iterator = iter(file('/tmp/ba'))
while 1:
    try: line = iterator.next()
    except StopIteration: break
    else: do_something_with(line)

i.e., expand the definition of "for line in..." into the constructs it's
defined in terms of -- same results with different goofy syntax.

But such expansion wouldn't be any more "explicit" than USING the
definition!!!  Any more than, say, x*5 is "less explicit" than x+x+x+x+x --
once multiplication is DEFINED in terms of repeated addition, it can
perfectly well be used, and be just as explicit as writing out the
repeated addition itself would be.

I don't think it's possible to make any halfway-sensible case that
USING a language's constructs directly is "less explicit" than goofily
spelling them out in terms of other constructs they may be defined
as equivalent to.


>> other forms of suites, e.g. hypothetically:
>> 
>> with <object>:
>>     <suite>
>> 
>> meaning to call the object (or some given method[s] in it, whatever)
>> with the suite as its argument, it would be just as explicit as, e.g.:
   ...
> This would be an interesting alternative, but it's back to being a special
> case, like Ruby has. I think it'd be more flexible as a literal that
> returns a callable.

Yes, by being a statement it would be less general than having it as an
expression would be -- an expression can be used anywhere a statement
can be used, but not viceversa.  What I'm uncertain about is whether
there IS a *good* syntax for a Python expression that can embody arbitrary
statements -- I'm not really enthusiastic about any proposal I've seen,
including mine own.


>>> This is not to say that I dislike that behavior; in fact, I find it
>>> *beneficial* that the manner of looping is *implicit* because you can
>>> substitute a generator for a sequence without changing the usage. But
>> 
>> You could do so even if you HAD to say iter(<object>) instead of
>> just <object> after every "for <name> in" -- it wouldn't be any
>> more "explicit", just more verbose (redundant, boiler-platey).  So
>> I do not agree with your motivation for liking "for x in y:" either;-).
> 
> Well, let's just say I've been on the Java side of the fence for a little
> while, and it has redefined my personal definition of explicit. One of the
> reasons Python code is so much smaller than Java code is that a lot of
> things are implicit that are required to be explicit in Java. I see this
> as a good thing.

I think you're using 'explicit' as a synonym for 'redundant', then.  But
it's not useful nor sensible usage, and insisting on it would be very
humpty-dumptyish.  It's not that in Python "a lot of things are implicit":
rather, the constructs are defined to _explicitly_ map to typical usage,
often (not invariably, but close) without boilerplate and redundance.


>>> there's little readability difference, IMHO, between that and:
>>> 
>>> file('input.txt').each_line({ |line|
>>>      do_something_with(line)
>>> })
>> 
>> Not huge, but the abundance of ({ | &c here hurts a little bit.
> 
> Well, we all __pick__ our __poisons__...

Sure, Ruby's "def add" is more readable than Python's "def __add__" --
we weren't discussing that, though, nor making general comparisons
about the two languages, but rather talking about the readability of
*iterator usage* in the two languages.


>> if you could not call file directly but rather than do say, e.g.:
>> 
>> file.open_for_reading_only('foo.txt').write('wot?')
> 
> Nah, I'm not arguing for redundancy at all. I'm saying that there is some
> voodoo going on here. When the *constructor* for a file object behaves
> like a generator that loops over newline-delimited lines of a text field,

The constructor constructs the object -- no more, no less.  The *object* is
an iterator (in 2.3; in 2.2 it HAS one instead), so of course you can 
iterate on it -- no voodoo whatsoever.

> doesn't that seem like it's been specialized for a particular domain in an
> unobvious way? Why lines? Why not bytes, words, unicode characters? I

Usage frequency: looping on lines is by far the use case that dominates in
frequency (no formal statistics, but a rare consensus of many experienced
users on python-dev at the time).

> mean, it's more convenient for people that do a lot of text processing,
> but I don't see anything specific to text or lines in the phrase
> "file('foo.txt')". That's all I'm saying.

There is nothing (except the accidental 'txt' substring) that directly
mention text, nor lines, bytes, words, etc; why, if you used the 'open'
synonym of 'file', there would be nothing mentioning files either.  But
then, there is nothing in the character '+' that even remotely suggests
addition -- except habit.  Do you therefore think that using '+' to denote
addition is *not explicit*?  Since '+' is DEFINED to perform addition,
by a convention that is quite familiar to most users, I'd disagree.  And
similarly for looping on a file object -- as there is one dominant use case,
it was deemed important enough to rapidly become familiar to most
users, and thereby eschew the "in the face of ambiguity, resist the
temptation to guess" principle's application.

Actually I was caught out that way while helping out on help at python.org
today -- a user was asking why "for line in afile.readlines(): 
md5.update(line)" was eating up all of his memory on processing a huge 
binary file, and I made the facile suggestion of moving to "for line in 
afile:" -- fortunately Danny caught me out (the good thing of having
several volunteers on the helpline -- not quite as good as c.l.py that
way, but still) and pointed out that the binary file MIGHT lack even a
single '\n' byte in its huge body, after all, so there was no _safety_ in
this approach (easily fixed with a generator, once spotted).  So, maybe I
shouldn't feel so tranquil that the "face of ambiguity" was *correctly*
handled.  But note it's most emphatically NOT about implicit or explicit:
the user originally had a PERFECTLY explicit readlines call -- just a
wrong one (which I mentally translated to just omit the .readlines() part,
which was my mistake).  It's about ambiguity and the temptation to
guess -- files don't provide methods to loop by ANYTHING BUT lines,
and that partly reflects the fact that other loops are easier to write as
generators, but also partly a "guess" (in the face of ambiguity...) that
a typical file is read as text, not as binary (for writing, the writelines
method has excellent semantics for consuming a finite iterator -- it's
just badly misnamed, leaving the misleading impression that it has
anything to do with lines... so much for 'explicitness' I guess...:-).


>>>    - types of variables
>> 
>> There are none, so how could such a nonexisting thing be EITHER implicit
>> OR explicit?  Variables don't HAVE types -- OBJECTS do.
> 
> The very fact that variables to not have types, and following that, that
> variables do not have manifest types, is an example of implicit being
> chosen over explicit. I know your argument, and I understand that Python
> variables are Post-It sticky notes and all of that, but please, just try
> to look at it from a non-Python-centric perspective. Other languages (like

Ok, as long as you let me choose Ruby as the non-Python in question.
Once I've done that, I see no difference: still sticky notes without
manifest per-variable types;-).

"x=5" is just as explicit as "int x=5", because the type of the object is
manifest.  The difference is not between implicit and explicit: it's
about attaching a type at all to that name 'x' or not.

> C++, which I hear you are vaguely familiar with ;) require you to be
> explicit about what type of thing you're defining and sending where.

Yes, and in that it's distinguished from e.g. Haskell, which also has
typed-names but allows you to NOT explicitly (or redundantly rather)
state them out and let it do inference instead.  Now THAT kind of
distinction is about explicitness or redundance.  Deeming that a name
has NO type is quite different.  I know of no programming language
that requires nor allows you to state the color of your cat's hair (except
in comments and the like): it's not a matter of your cat's color being
left "implicit", it's rather that those languages don't HAVE the concept
of a cat, its hair, and the hair's color.  Just think of variables as cats,
rather than sticky notes, and the issue may get clearer.


>> Etc, etc -- can't spend another 1000 lines to explain why your "lots of
>> things" do not indicate violations of "explicit is better than implicit".
> 
> They're not *violations*. Correct me if I'm wrong, but the Zen of Python
> is not the LAW! It's a poem! It's very beautiful, very concise, inspiring,
> and thoughtful, but it's not the 10 commandments! I just get very tired of

It's a set of principles humorously and perceptively expressed, of course.

> every idea getting shot down because of some rule from Tim Peters. I
> really don't think he intended for it to be used to prove the validity of
> ideas.

No doubt some people quote it routinely, but, when e.g. I do, it's just
because it's often a very concise, expressive and clear way to express
what I could alternatively take a few hours to laboriously set down in
excruciating detail (of course, I can and do perform BOTH too:-).
Those principles that are NOT all that common in other programming
languages are particularly valuable because they express Python's
community consensus where it most needs to be expressed.


> The implicit/explicit thing is one of the most abused, in my opinion,
> because it can quite frankly be used to shut down any attempt at creating
> abstraction. In fact, for that reason alone, I'm tempted to say "Implicit
> is better than explicit". Say what you want, not how you want it. Be
> abstract, not concrete.

...but please do so in other languages than Python; there are several
excellent ones for the purpose (Ruby isn't really one -- many of its 
"implicitness" factors are in fact legacy holdovers -- for example, I've 
often seen the optional omission of parentheses in method calls deemed iffy 
by experienced Rubytes, even more for the implicit $_, etc, etc).  Python 
is quite concrete and pragmatical, as is Ruby, at their cores.  I think
functional languages encourage and nudge you to work at very high
levels of abstraction, very VERY far from the machine, in ways that no
language with modifiable data truly does.  If "be abstract, not concrete"
is really the motto you want to live by, I think Erlang or Haskell or ML
might make you much happier than Python or Ruby would.

Most of us actually work better on a more concrete level, I believe --
and that is why, for me (and 'us':-) languages with the concreteness
(and explicitness) of Perl and Ruby are more productive (though the
mind-expanding experience of studying FP is a great visit, I'm not sure
I'd want to live there).

But note that this about actual abstraction, and has nothing to do
with implicit vs explicit.  If you think that a language's rules and
definitions -- just like, say, the definition of multiplication -- "are
implicit", or using them _makes_ your programs "implicit", then I think
your individual and idiosyncratic definition of words can make discourse
with you simply impossible.


>>> This is nothing like APL... if anything, it's like Smalltalk, a language
>>> designed to be readable by children!
>> 
>> Cite pls?  I knew that Logo and ABC had been specifically designed
>> with children in mind, but didn't know that of Smalltalk.
> 
> http://ei.cs.vt.edu/~history/GASCH.KAY.HTML
> http://www.cosc.canterbury.ac.nz/~wolfgang/cosc205/smalltalk1.html
> http://www.cs.washington.edu/homes/dugan/history.html

Thanks!  Quite instructive.


> Well, it was a rough analogy, and I've never done any APL myself, but
> here's my justification, FWIW:
> 
>  - APL provides syntactical constructs for high-level array processing

Yep, a huge array of weird symbols for the purpose (operators infix
or prefix, w/o priority differences among the set of infix ones).

>  - List comprehensions do this also

Not really: just an expression to create a list (array), with deliberate
use of keywords (vs Haskell's symbols for the same role).  This is
like saying that, e.g., range(5) is like APL's <iota>5... no, it's even
"farther fetched" IMHO.

>  - Code blocks have nothing inherently to do with array processing

Neither do many of APL's many symbols, e.g. left-pointing arrow
for assignment, right-pointing for goto (only control flow statement being
a computed goto), etc.


> But I agree that neither resemble APL as I've seen. I guess it's like
> saying a carrot is more like APL than a rutabega.

It does seem to be on a similar plane of "similitude", yes.


Alex