[Edu-sig] Reloading code (was Re: OLPC: first thoughts)
Paul D. Fernhout
pdfernhout at kurtz-fernhout.com
Sat Feb 24 15:35:00 CET 2007
Here is one post on this issue with code I made previously for Jython:
"ReloaderWindow 0.2 (improvements to selective reloading)"
This is a different (earlier) approach than for PataPata (PataPata uses
prototypes and overriding __getattr__) and the linked code works with
regular Python classes.
The important thing to understand is that a Python VM manages a world of
objects somewhat like a Smalltalk image, lumping all the objects from all
the loaded modules into the same objectspace. You can just reference some
of them (typically classes) by the way the current hierarchical Python
namespace is set up. When you reload a module, it creates completely new
class objects which just happen to have the same name as the old one and
the namespace now points to those new objects (sort of like if every time
you changed a method in Smalltalk the SystemDictionary Smalltalk was made
to point to entirely new classes, leaving the old ones floating, which
would obviously be a silly thing to do). So any existing instances of
original classes point still to the old classes (which are no longer
referenceable from the namespace).
The various approaches try to get around this in various ways, such as in
my approach above by loading a changed module into a temporary namespace
and then copying the methods from it on top of the old classes. Also,
things get a little more complicate than that, since Python often uses
method pointers, equivalent in some way to Smalltalk blocks, to hook up
GUI systems. So even if you keep the class the same and replace the
methods, your running GUI window typically has bindings by reference to
the old overwritten functions. So, as this approach I outlines does, you
need to also keep the shell of the old functions and just overwrite what
they actually do -- Kind of like if you kept CompiledMethods around and
just overwrote their code. It's an awkward way to do things. Now,
obviously, Smalltalk sometimes fails at updating GUIs when you tie actions
to blocks, where recompiling methods produces new blocks (a big problem in
older VisualWorks apps, not sure of the current status of VisualWorks or
Squeak). Of course, in Smalltalk you could make the block call a method
using a selector, which as least gives you a point of intervention.
Similarly in Python, if you modify code which the bound function
references, that newer code will be called by the old function's code as
dispatch works differently depending whether you have a function pointer
as opposed to an inline reference to another function in a Python
function. In Python, I can get around this by wrapping any function
pointer I give to a GUI callback using an object (like Java :-) where the
callback then calls by name. PataPata did some of that internally, though
obviously it adds extra layers of overhead to every GUI call, plus the
programmer had to proactively add a wrapper, making code uglier, and so on.
In Python, people often build specific hooks into their application to
explicitly reload modules, including sometimes closing and reopening only
one window inside a large application, to try to make the best of this.
There are other attempts to do a more selective reloading like the
approach I give above. Consider for example:
"[Patches] [ python-Patches-1212921 ] option to allow reload to affect
There is some discussion at that link of the issue. And a reference there
to another approach:
From the first patch link: "Brett and I work at the same company. He made
this patch because we were wasting hundreds of hours debugging reload
problems or just quitting the application and restarting. I've
been using this patch now for 3 months. It has saved our company hundreds
of man-hours and we didn't have to change one line of existing code to
make it work. If the community doesn't want to use it, I'm fine with that.
I just want to make sure that you fully understand what you are giving up."
I'm not sure what the disposition of that patch is. But, you can see from
the responses there is a bit of a cultural problem here getting the value
of the idea across.
I did not look at the patch, but from the description I wonder if it still
might suffer from the issue of allowing identities for functions to change
(and so GUIs with actions previously linked to functions would still use
the old behavior).
There is also a more subtle issue for module reloading. Smalltalk is
essentially a declarative language, while Python is essentially an
imperative language (maybe there is a better word?). Essentially,
compiling a Smalltalk class has no side effects (other than perhaps
defining the class). But a Python module on loading can produce lots of
side effects from defining metaclasses to affect future class loading to
executing random bits of code (sort of like Forth in this sense). So, only
modules without these side effects can be easily reloaded. Consider for
example what if your module makes a singleton instance of something --
when you reload it, should it make a new singleton instance and leave the
other one to still perhaps be used by old instances, or should the
singleton instance get upgraded to the new class? Just one of many issues
that happen when you are using Python's textual module reloading paradigm.
Ideally, when you change a method in a text file you probably almost
always want only that specific method to change with everything else being
left alone. But that breaks the Python paradigm. Still, even with this
problem, easier class reloading like I outlined above still gets you great
value when developing new code, especially if you slightly alter how you
layout your new modules with this limitation in mind. But reloading older
code may present you with problems -- making maintenance using this
Andreas Raab wrote:
> Paul D. Fernhout wrote:
>>This "edit and continue" support is much more fine grained and
>>easier to use than reloading Python modules, even with the variants I and
>>others have created.
> Where can I find out more about these variants and what they do? We are
> currently trying to enable a workflow that centers around a life
> environment and these issues become important for us. Are there any
> other life environments (besides Patapata which I have looked at) that
> are worth studying?
More information about the Edu-sig