[Edu-sig] Reloading code (was Re: OLPC: first thoughts)

[Paul D. Fernhout]

Andreas-

Here is one post on this issue with code I made previously for Jython:
   "ReloaderWindow 0.2 (improvements to selective reloading)"
http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
This is a different (earlier) approach than for PataPata (PataPata uses 
prototypes and overriding __getattr__) and the linked code works with 
regular Python classes.

The important thing to understand is that a Python VM manages a world of 
objects somewhat like a Smalltalk image, lumping all the objects from all 
the loaded modules into the same objectspace. You can just reference some 
of them (typically classes) by the way the current hierarchical Python 
namespace is set up. When you reload a module, it creates completely new 
class objects which just happen to have the same name as the old one and 
the namespace now points to those new objects (sort of like if every time 
you changed a method in Smalltalk the SystemDictionary Smalltalk was made 
to point to entirely new classes, leaving the old ones floating, which 
would obviously be a silly thing to do). So any existing instances of 
original classes point still to the old classes (which are no longer 
referenceable from the namespace).

The various approaches try to get around this in various ways, such as in 
my approach above by loading a changed module into a temporary namespace 
and then copying the methods from it on top of the old classes. Also, 
things get a little more complicate than that, since Python often uses 
method pointers, equivalent in some way to Smalltalk blocks, to hook up 
GUI systems. So even if you keep the class the same and replace the 
methods, your running GUI window typically has bindings by reference to 
the old overwritten functions. So, as this approach I outlines does, you 
need to also keep the shell of the old functions and just overwrite what 
they actually do -- Kind of like if you kept CompiledMethods around and 
just overwrote their code. It's an awkward way to do things. Now, 
obviously, Smalltalk sometimes fails at updating GUIs when you tie actions 
to blocks, where recompiling methods produces new blocks (a big problem in 
older VisualWorks apps, not sure of the current status of VisualWorks or 
Squeak). Of course, in Smalltalk you could make the block call a method 
using a selector, which as least gives you a point of intervention. 
Similarly in Python, if you modify code which the bound function 
references, that newer code will be called by the old function's code as 
dispatch works differently depending whether you have a function pointer 
as opposed to an inline reference to another function in a Python 
function. In Python, I can get around this by wrapping any function 
pointer I give to a GUI callback using an object (like Java :-)  where the 
callback then calls by name. PataPata did some of that internally, though 
obviously it adds extra layers of overhead to every GUI call, plus the 
programmer had to proactively add a wrapper, making code uglier, and so on.

In Python, people often build specific hooks into their application to 
explicitly reload modules, including sometimes closing and reopening only 
one window inside a large application, to try to make the best of this. 
There are other attempts to do a more selective reloading like the 
approach I give above. Consider for example:
   "[Patches] [ python-Patches-1212921 ] option to allow reload to affect 
existing instances"
   http://mail.python.org/pipermail/patches/2005-September/018060.html
There is some discussion at that link of the issue. And a reference there 
to another approach:
   http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

 From the first patch link: "Brett and I work at the same company. He made 
this patch  because we were wasting hundreds of hours debugging reload 
problems or just quitting the application and restarting. I've
been using this patch now for 3 months. It has saved our  company hundreds 
of man-hours and we didn't have to change one line of existing code to 
make it work. If the community doesn't want to use it, I'm fine with that. 
I  just want to make sure that you fully understand what you are giving up."

I'm not sure what the disposition of that patch is. But, you can see from 
the responses there is a bit of a cultural problem here getting the value 
of the idea across.

I did not look at the patch, but from the description I wonder if it still 
might suffer from the issue of allowing identities for functions to change 
(and so GUIs with actions previously linked to functions would still use 
the old behavior).

There is also a more subtle issue for module reloading. Smalltalk is 
essentially a declarative language, while Python is essentially an 
imperative language (maybe there is a better word?). Essentially, 
compiling a Smalltalk class has no side effects (other than perhaps 
defining the class). But a Python module on loading can produce lots of 
side effects from defining metaclasses to affect future class loading to 
executing random bits of code (sort of like Forth in this sense). So, only 
modules without these side effects can be easily reloaded. Consider for 
example what if your module makes a singleton instance of something -- 
when you reload it, should it make a new singleton instance and leave the 
other one to still perhaps be used by old instances, or should the 
singleton instance get upgraded to the new class? Just one of many issues 
that happen when you are using Python's textual module reloading paradigm. 
Ideally, when you change a method in a text file you probably almost 
always want only that specific method to change with everything else being 
left alone. But that breaks the Python paradigm. Still, even with this 
problem, easier class reloading like I outlined above still gets you great 
value when developing new code, especially if you slightly alter how you 
layout your new modules with this limitation in mind. But reloading older 
code may present you with problems -- making maintenance using this 
technique harder.

--Paul Fernhout

Andreas Raab wrote:
> Paul D. Fernhout wrote:
> 
>>This "edit and continue" support is much more fine grained and 
>>easier to use than reloading Python modules, even with the variants I and 
>>others have created. 
> 
> Where can I find out more about these variants and what they do? We are 
> currently trying to enable a workflow that centers around a life 
> environment and these issues become important for us. Are there any 
> other life environments (besides Patapata which I have looked at) that 
> are worth studying?