Python syntax in Lisp and Scheme

[prunesquallor at comcast.net]

Alexander Schmolck <a.schmolck at gmx.net> writes:

> prunesquallor at comcast.net writes:
>> Suppose I cut just one arm of a conditional.  When I paste, it is
>> unclear whether I intend for the code after the paste to be part of
>> that arm, part of the else, or simply part of the same block.
>
> Sorry, I have difficulties understanding what exactly you mean again. 

Let me back up here.  I originally said:

>> Every line in a block doesn't encode just its depth relative to the
>> immediately surrounding context, but its absolute depth relative to
>> the global context.

To which you replied:

> I really don't understand why this is a problem, since its trivial to
> transform python's 'globally context' dependent indentation block structure
> markup into into C/Pascal-style delimiter pair block structure markup.

> Significantly, AFAICT you can easily do this unambiguously and *locally*, for
> example your editor can trivially perform this operation on cutting a piece of
> python code and its inverse on pasting (so that you only cut-and-paste the
> 'local' indentation).

Consider this python code (lines numbered for exposition):

 1 def dump(st):
 2     mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime = st
 3     print "- size:", size, "bytes"
 4     print "- owner:", uid, gid
 5     print "- created:", time.ctime(ctime)
 6     print "- last accessed:", time.ctime(atime)
 7     print "- last modified:", time.ctime(mtime)
 8     print "- mode:", oct(mode)
 9     print "- inode/dev:", ino, dev
10
11 def index(directory):
12     # like os.listdir, but traverses directory trees
13     stack = [directory]
14     files = []
15     while stack:
16         directory = stack.pop()
17         for file in os.listdir(directory):
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
22     return files

This code is to provide verisimilitude, not to actually run.  I wish
to show that local information is insufficient for cutting and pasting
under some circumstances.

If we were to cut lines 18 and 19 and to insert them between lines
4 and 5, we'd have this result: 

 3     print "- size:", size, "bytes"
 4     print "- owner:", uid, gid
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
 5     print "- created:", time.ctime(ctime)
 6     print "- last accessed:", time.ctime(atime)

Where we can clearly see that the pasted code is at the wrong
indentation level.  It is also clear that in this case, the
editor could easily have determined the correct indentation.

But let us consider cutting lines 6 and 7 and putting them
between lines 21 and 22.  We get this: 

15     while stack:
16         directory = stack.pop()
17         for file in os.listdir(directory):
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
 6     print "- last accessed:", time.ctime(atime)
 7     print "- last modified:", time.ctime(mtime)
22     return files

But it is unclear whether the intent was to be outside the while,
or outside the for, or part of the if.  All of these are valid: 

15     while stack:
16         directory = stack.pop()
17         for file in os.listdir(directory):
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
 6         print "- last accessed:", time.ctime(atime)
 7         print "- last modified:", time.ctime(mtime)
22     return files

15     while stack:
16         directory = stack.pop()
17         for file in os.listdir(directory):
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
 6             print "- last accessed:", time.ctime(atime)
 7             print "- last modified:", time.ctime(mtime)
22     return files

15     while stack:
16         directory = stack.pop()
17         for file in os.listdir(directory):
18             fullname = os.path.join(directory, file)
19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
 6                 print "- last accessed:", time.ctime(atime)
 7                 print "- last modified:", time.ctime(mtime)
22     return files

Now consider this `pseudo-equivalent' parenthesized code:

 1 (def dump (st)
 2    (destructuring-bind (mode ino dev nlink uid gid size atime mtime ctime) st
 3       (print "- size:" size "bytes")
 4       (print "- owner:" uid gid)
 5       (print "- created:" (time.ctime ctime))
 6       (print "- last accessed:" (time.ctime atime))
 7       (print "- last modified:" (time.ctime mtime))
 8       (print "- mode:" (oct mode))
 9       (print "- inode/dev:" ino dev)))
10
11 (def index (directory)
12     ;; like os.listdir, but traverses directory trees
13     (let ((stack directory)
14           (files '()))
15       (while stack
16         (setq directory (stack-pop))
17         (dolist (file (os-listdir directory))
18            (let ((fullname (os-path-join directory file)))
19              (push fullname files)
20              (if (and (os-path-isdir fullname) (not (os-path-islink fullname)))
21                  (push fullname stack)))))
22       files))

If we cut lines 6 and 7 with the intent of inserting them
in the vicinity of line 21, we have several options (as in python),
but rather than insert them incorrectly and then fix them, we have
the option of inserting them into the correct place to begin with.
In the line `(push fullname stack)))))', there are several close
parens that indicate the closing of the WHILE, DOLIST, LET, and IF,
assuming we wanted to include the lines in the DOLIST, but not
in the LET or IF, we'd insert here:
                                            V
21                  (push fullname stack)))   ))

The resulting code is ugly:

11 (def index (directory)
12     ;; like os.listdir, but traverses directory trees
13     (let ((stack directory)
14           (files '()))
15       (while stack
16         (setq directory (stack-pop))
17         (dolist (file (os-listdir directory))
18            (let ((fullname (os-path-join directory file)))
19              (push fullname files)
20              (if (and (os-path-isdir fullname) (not (os-path-islink fullname)))
21                  (push fullname stack)))
6       (print "- last accessed:" (time.ctime atime))
7       (print "- last modified:" (time.ctime mtime))))
22       files))

But it is correct.

(Incidentally inserting at that point is easy:  you move the cursor over
the parens until the matching one at the beginning of the DOLIST begins
to blink.  At this point, you know that you are at the same syntactic level
as the dolist.)

>> >> The fact that the information is replicated, and that there is nothing
>> >> but programmer discipline keeping it consistent is a source of errors.

Let me expand on this point.  The lines I cut are very similar to each
other, and very different from the lines where I placed them.  But
suppose they were not, and I had ended up with this:

19             files.append(fullname)
20             if os.path.isdir(fullname) and not os.path.islink(fullname):
21                 stack.append(fullname)
 6     print "- last accessed:", time.ctime(atime)
 7     print "- last modified:", time.ctime(mtime)
22     print "- copacetic"
23     return files

Now you can see that lines 6 and 7 ought to be re-indented, but line 22 should
not.  It would be rather easy to either accidentally group line seven with
line 22, or conversely line 22 with line 7. 

>> > Sure there is. Your editor and immediate visual feedback (no need to remember
>> > to reindent after making the semantic changes).
>> 
>> `immediate visual feedback' = programmer discipline
>> Laxness at this point is a source of errors.
>
> You got it backwards. 
> Not forgetting to press 'M-C-\' = programmer discipline.
> Laxness at this point is a source of errors. 

Forgetting to indent properly in a lisp program does not yield
erroneous code.

> And indeed, people *do* have to be educated not to be lax when editing lisp -
> newbies frequently get told in c.l.l or c.l.s that they should have reindented
> their code because then they would have seen that they got their parens mixed
> up.

This is correct.  But what is recommended here is to use a simple tool to
enhance readability and do a trivial syntactic check. 

> OTOH, if you make an edit in python the result of this edit is immediately
> obvious -- no mismatch between what you think it means and what your computer
> thinks it means and thus no (extra) programmer discipline required.

Would that this were the case.  Lisp code that is poorly indented will still
run.  Python code that is poorly indented will not.  I have seen people write
lisp code like this:

(defun factorial (x)
(if (> x 0)
x
(*
(factorial (- x 1))
x
)))

I still tell them to re-indent it.  A beginner writing python in this manner
would be unable to make the code run.

> Of course you need *some* basic level of discipline to not screw up your
> source code when making edits -- but for all I can see at the moment (and know
> from personal experience) it is *less* than what's required when you edit lisp
> (I have provided a suggested way to edit this particular example in emacs for
> python in my previous post -- you haven't provided an analoguous editing
> operation for lisp with an explanation why it would be less error-prone)).

Ok.  For any sort of semantic error (one in which a statement is
associated with an incorrect group) one could make in python, there is
an analagous one in lisp, and vice versa.  This is simply because both
have unambiguous parse trees.

However, there is a class of *syntactic* error that is possible in
python, but is not possible in lisp (or C or any language with
balanced delimiters).  Moreover, this class of error is common,
frequently encountered during editing, and it cannot be detected
mechanically.

Consider this thought experiment:  pick a character (like parenthesis
for example) go to a random line in a lisp file and insert four of them.
Is the result syntactically correct?  No.  Could a naive user find them?
Trivially.  Could I program Emacs to find them?  Sure.

Now go to a random line in a python file and insert four spaces.  Is
the result syntactically correct?  Likely.  Could a naive user find
them?  Unlikely.  Could you write a program to find them?  No.

Delete four adjacent parens in a Lisp file.  Will it still compile?  No.
Will it even be parsable?  No.

Delete four adjacent spaces in a Python file.  Will it still compile?
Likely.

>> >> >> Yet the visual representation is not only identical between all of these, it
>> >> >> cannot even be displayed.
>> >> >
>> >> > I don't understand what you mean. Could you maybe give a concrete example of
>> >> > the information that can't be displayed? 
>> >> 
>> >> Sure.  Here are five parens )))))  How much whitespace is there here:          
>> >
>> > 10 spaces (which BTW I counted in emacs in just the same way that I'd count a
>> > similar number of parens) -- but what has counting random trailing whitespace
>> > got to do with anything? 
>> 
>> It is simply an illustration that there is no obvious glyph associated
>> with whitespace, and you wanted a concrete example of something that can't
>> be displayed.
>
> No, I didn't want just *any* example of something that can't be displayed; I
> wanted an example of something that can't be displayed and is *pertinent* to
> our discussion (based on the Quinean assumption that you wouldn't have brought
> up "things that can't be displayed" if they were completely besides the
> point).

I thought that whitespace was significant to Python.

My computer does not display whitespace.  I understand that most
computers do not.  There are few fonts that have glyphs at the space
character.

Since having the correct amount of whitespace is *vital* to the
correct operation of a Python program, it seems that the task of
maintaining it is made that much more difficult because it is only
conspicuous by its absence.

> me:
>> >> > People can't "read" '))))))))'.
> [more dialog snipped]
>> I cannot read Abelson and Sussman's minds, but neither of them are
>> ignorant of the vast variety of computer languages in the world.
>> Nonetheless, given the opportunity to choose any of them for
>> exposition, they have chosen lisp.  Sussman went so far as to
>> introduce lisp syntax into his book on classical mechanics.
>
> Well the version of SICM *I've* seen predeominantly seems to use (infixy) math
> notation, so maybe Sussman is a little less confident in the perspicuousness
> of his brainchild than you (also cf. Iverson)?

Perhaps you are looking at the wrong book.  The full title is
`Structure and Interpretation of Classical Mechanics' by Gerald Jay
Sussman and Jack Wisdom with Meinhard E. Mayer, and it is published by
MIT Press.  Every computational example in the book, and there are
many, is written in Scheme.

Sussman is careful to separate the equations of classical mechanics
from the *implementation* of those equations in the computer, the
former are written using a functional mathematical notation similar to
that used by Spivak, the latter in Scheme.  The two appendixes give
the details.  Sussman, however, notes ``For very complicated
expressions the prefix notation of Scheme is often better''

> I don't personally think (properly formated) lisp reads that badly at all
> (compared to say C++ or java) and you sure got the word-seperators right. But
> to claim that using lisp-style parens are in better conformance with the
> dictum above than python-style indentation frankly strikes me as a bit silly
> (whatever other strengths and weaknesses these respective syntaxes might
> have).

And where did I claim that?  You originally stated:

> Still, I'm sure you're familiar with the following quote (with which I most
> heartily agree):
>
>  "[P]rograms must be written for people to read, and only incidentally for
>   machines to execute."
>
> People can't "read" '))))))))'.

Quoting Sussman and Abelson as a prelude to stating that parenthesis are
unreadable is hardly going to be convincing to anyone.

>> Obviously the indentation.  
>> But I'd notice the mismatch.
>
> (Hmm, you or emacs?)

Does it matter?

>> If I gave you a piece of python code jotted down on paper that (as these
>> hypothetical examples usually are) for some reason was of vital importance
>> but I accidentally misplaced the indentation -- how would you know?
>
> Excellent point. But -- wait! Were it Lisp, how would I know that you didn't
> intend e.g.
>
>   (if (bar watz) foo)
>
> instead of 
>
>   (if (bar) watz foo)

You are presupposing *two* errors of two different kinds here:  the
accidental inclusion of an extra parenthesis after bar *and* the
accidental omission of a parenthesis after watz.

The kind of error I am talking about with Python code is a single
error of either omission or inclusion.

> Moral: I really think your (stereoptypical) argument that the possibility of
> inconsistency between "user interpretation" and "machine interpretation" of a
> certain syntax is a feature (because it introduces redundancy that can can be
> used for error detection) requires a bit more work.

I could hardly care less.