[Edu-sig] Reloading code (was Re: OLPC: first thoughts)

[Paul D. Fernhout]

Michael-

Thanks for the update. An entire Python compiler to make it work right?
Wow. I don't use many class methods so I've never noticed this problem.
Thanks for putting this up, and under the Python license (according to
the readme).
  http://svn.brownspencer.com/pycompiler/branches/new_ast/readme.txt
Is this in any ways related to PyPy?
  http://codespeak.net/pypy/dist/pypy/doc/news.html

In the Jython reload case I have noticed a different problem, which is
that a method with a call to super__ will fail. super__MethodName is
Jython's convention for calling a superclass method, which is especially
important to use if the Java method in a Java base class is "protected"
as using the alternative BaseClassName.method(self, args) will not work
then. It turns out that after a reload in Jython, the newly compiled
methods have a slightly different signature than the old ones as I think
there is a unique number added to the superclass name, causing the
reference to the old super class wrapper to be invalid somehow on a
second load

Also, if you change the number of arguments to a function, or add a new
function, the reload feature also fails. I think these problems may all
be related to Jython's internal behind the scenes dispatching using some
forms of name mangling which change from load to load.

As I write this, I am wondering if those problems only occur when the
base class is a Java class as opposed to a Jython base class, though
many if not most of my base classes are Java classes (typically JPanel
or JFrame), so it is a frequent problem.

Nonetheless, I find the reload functionality very valuable, even with
these problem of having to restart when adding a function, changing the
arguments to a function. A typically way to use this is adding print
statements to a function operating unexpectedly for use in debugging.
Also I just avoid using super__ use in modules in various ways (which is
sometimes difficult as it is the only way I know to call a Java
protected method).

I don't believe CPython reloading has any of those problems, since it is
not doing the kind of reflection Jython does.

On the JythonDev list, I submitted this issue as one that might interest
an intern for the Google Summer of Code.
http://www.nabble.com/Jython-and-Summer-of-Code-tf3350563.html#a9367861

When even Visual Basic or C# or Ruby can do this "edit and continue", I
am worried about Python's future if it does not gain this feature. See
for example:
  http://www.codinghorror.com/blog/archives/000507.html

While I prefer Smalltalk keyword syntax for being self-documenting over
function calls with position arguments, I can weight that against
Python's clean syntax using indentation, and say Python syntax and
Smalltalk syntax are close in usability. And even if Smalltalk will
always also have an edge in edge in easily adding features like new
iterators or control structures that require changes to the Python
compiler, I can weigh that against Python's broad acceptability in
looking a lot like C or Java, especially how easily Jython can
interoperate with Java, given the similarity in syntax. BUT, when I
weigh being much more productive in an environment like Smalltalk that
lets you "edit and continue", Python is the clear loser there. It is
only its other virtues (licensing, borad libraries, ease in doing small
projects with it, and so on) which save the day there for it IMHO. But
that's a little sad -- to see such a great dynamic language tied down
somehow over an issue which you would think it would excel at. Now that
Guido is at Google, and Google uses Python and does large projects (e.g.
the recent AI announcement)
http://www.oreillynet.com/onlamp/blog/2005/11/google_library_will_train_goog.html
which presumably require modifying long running training processes while
they run if they encounter, say, a typo in a variable reference, I would
hope building in these features might someday become more of a priority,
nudge, nudge. :-) But really, this is the kind of deep thing in Python I
would expect Guido would have to be involved in changing; it's probably
not like a minor change or add-on in the system -- it might have
ramifications throughout the codebase. I doubt anyone would have enough
comprehension of Python internals to do as good a job as he could.

Note that I think that even if it was impossible in Python to resume
from an exception (without major surgery in Python's innards), being
able to break on the creation of any exception in the debugger, and then
one by one tell the debugger to ignore some of them based on the type of
exception or the module it is created in, would probably still be OK
enough if you could then edit and continue before the exception was
actually raised. This weaker form of "Edit and continue" would be harder
to use, but probably still worthwhile given the right IDE tools. It
would be more feasible for a Summer of Code student or other contributor
to do as a first step. (Generally, you only care about the exceptions
that bubble up to the top, since a lot of Python code routinely handles
exceptions like for not opening a file or when using exceptions to check
the duck typing of something, which is why this weaker form would entail
extra work). A lot of IDEs for other languages (like C++) let you
control what exceptions the debugger breaks on, so this is not that
unfeasible.

--Paul Fernhout

Michael Spencer wrote:
> Paul D. Fernhout <pdfernhout <at> kurtz-fernhout.com> writes:
> 
> 
>> This xreload module Guido supplies is similar to the approach I used for 
>> Jython in my earlier link.
>> http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
>> If you look at that thread, I acknowledge Michael Spencer as having 
>> supplied some of the more sophisticated reload logic which is similar to 
>> what Guido has posted here (not sure where Michael got it from of course, 
>> nor where Guido got xreload from). See Michael's first related post here:
>> http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.html
>>
> 
> 
> 
> FWIW, I still tinker with the reloading issue from time to time, in pursuit of a
> fully-interactive environment.  My latest module is at:
> http://svn.brownspencer.com/pynote/trunk/Mupdaters.py.  It's not documented for
> standalone use, but may contain some useful ideas, and I'd be happy to discuss
> it if anyone were interested.
> 
> Incidentally, class bodies provide one of the hard problems for reloading.
> Ideally, for reloading, one would execute the class body in a proxy of the
> existing dict, as for modules, but this can't be achieved with the same exec
> <modulecode> in <proxyofexistingdict> approach.   In order to work-around this,
> I wrote a pure-python compiler.  With this, it becomes possible to modify the
> class definition bytecode to execute the class definitions in the existing class
> dict rather than a new one.  The compiler itself is at:
> http://svn.brownspencer.com/pycompiler/branches/new_ast/ .  However, I have not
> yet connected it into the reloading module.
> 
> Michael