As far as I know, there will be some screencasts/recordings available. I haven't seen a link to the slides so far and was planning to find out. On 2/23/07, kirby urner <kirby.urner@gmail.com> wrote:

Thanks for the update André.

At Pycons I've attended @ GWU in DC, we were promised screencasts at least if the speaker signed a waiver allowing it.

Was it your impression that this keynote will have an URL, maybe has one already?

I was unaware of the 'view source' button. Must be a different brand.

Kirby

On 2/23/07, Vern Ceder <vceder@canterburyschool.org> wrote:

I would second Andre's assessment. It just may have been one of the best keynotes of any PyCon, whatever you think of OLPC. And I think most people in the room ended up with a pretty positive feeling for what the OLPC folks want to do.

Vern

I haven't met anyone "anti" OLPC, just some who've felt their plans to further computerize the curriculum were not on hold while we waited for such hardware to materialize. Mark Shuttleworth for example. The forward progress of that South African process was in no way contingent upon the success of failure (e.g. perpetual postponement) of the projected OLPC result (one laptop per child). For example, given TuxLabs, shared equipment, it's already possible to do Squeak-based robotics or whatever. Kirby

On 2/23/07, francois schnell <francois.schnell@gmail.com> wrote:

I hope someone recorded that if so it looks like it should appear in the future on this page (items):

http://us.pycon.org/zope/talks/2007/fri/track3/004/talkDetails

Well, still nothing. I imagine it will be available in weeks or months to come. During the same time frame there was another OLPC presentation in Bruxelles at the FOSDEM 2007 ('Free and Open source Software Developers' European Meeting'). The speaker was Jime Gettys, http://en.wikipedia.org/wiki/Jim_Gettys and the video is available here: http://ftp.belnet.be/mirrors/FOSDEM/2007/FOSDEM2007-OLPC.ogg The video format is Ogg/Theora. It will play fine in VLC player for example. http://www.videolan.org/ -- francois schnell

On 2/23/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

Anyway, there is no inherent reason Python can not do what Smalltalk does -- although it might require some minor changes to the VM to debug and restart exceptions; Python just does not do it, I suspect in part because key Pythoneers may just not understand what they are missing and think module reloading is the same thing. :-) I predict if Guido added this one set of related features to Python, it would be a very short time before everyone wondered how they lived without it before. :-)

You may be right, although if I understand your aright, it's neither the syntax nor semantics of Python that are at issue, but the IDE and its ability to permit debugging on the fly, using thrown exceptions as restart points of some kind. Sounds cool. Don't have to wait for Guido, might be able to retrofit it for earlier Pythons. Kirby

On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with is specifically in how you write your own code, but that is not a general solution.

Not sure I follow yet. A module loads top to bottom, with lower defs premised on those previously mentioned. Is that what you mean? Once everything is loaded, it's more like a __dict__, i.e. the namespace of the module of accessible, either via dot notation, or directly if the names are top level.

See my previous note to Andreas for more details, including a related Python patch someone submitted in 2005.

--Paul Fernhout

I'll wait for the IDE that passes muster as a "what you meant". In no such IDE is forthcoming, I'm not inclined to blame Python the language yet, as your improvements sound tangential to the design concepts. So far. Maybe I'm missing something, so keep persuading me if you like. Kirby

On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

kirby urner wrote:

...

On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

...

There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with [it] specifically in how you write your own code, but that is not a general solution.

Not sure I follow yet. A module loads top to bottom, with lower defs premised on those previously mentioned. Is that what you mean? Once everything is loaded, it's more like a __dict__, i.e. the namespace of the module of accessible, either via dot notation, or directly if the names are top level.

To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).

Unfortunately, the paradigm used by every Python IDE I've tried is to reload an entire textual module (or more typically, entire program) at a time for even the slightest change to one function. Embedded Python shells generally allow you to redefine a function if you have a copy of the code, but they offer no way to save the code. Most Smalltalks uses a different paradigm, where code is presented to the user one function at a time in a browser and is compiled one function at a time. Yes, there are cases where people "filein" Smalltalk code defining a complex program, but such fileins are generally considered an *interchange* format, not a preferred program representation for editing unlike as is usually the case with Python.

Consider the meaning of an arbitrary piece of Python code near the bottom of a textual module. Essentially, you have no idea what it means if the original author has used some Python bells and whistles. For example, he or she could have defined a metaclass where every nested "def" under a class was converted to, say, an uppercase string and stored under a key that was the numerical hash of the function name (with no functions actually defined for that class perhaps). The specific metaclass behavior may even hinge on the current state of a global which has been modified several times during the course of loading the module. So essentially, you have no way of knowing for sure what any apparent Python code really means by isolated inspection. And because any module can run any arbitrary Python code, without actually running the Python program (or doing the equivalent analysis), you can never be sure what side effects loading a module has. Now, Smalltalk has metaclasses too, but in practice, because of the way code is presented to the user and edited and recompiled one method/function at a time, the context makes fairly clear what is going to happen when that snippet of code you just changed is compiled. The big difference is really the effective unit of compilation -- the complex module in Python or the simple method/function in Smalltalk.

Now, this is rarely a problem the *first* time a module is loaded, but it generally becomes a problem when a module is *reloaded*. If you only treated as module as an *interchange* format, and then modified the live classes using a tool which only works on regular classes (like PataPata does), there is no need to reload the module, so this potential problem related to parsing a modules meaning via an IDE tool remains only potential, and also avoided is the possibility reloading a module might have side effects. (In practice, anything still depends on mapping from a function back to its source text, and this may go wrong for various reasons... :-)

Naturally, this kind of major redefinition is rarely done, and it would create lots of confusion, but it is possible, so IDE tools that do not support it are incomplete. This is a perrenial problem with, say, C, where you can make all sorts of macros and so never know just exactly what arbitrary C code out of context does (see the obfuscated code contests...). And it means that you can't get a simple one-to-one mapping of a section of a file that looks like it defines a function and an actual function reliably without analyzing the entire program. Yes, 99.99% of the time Python code does the obvious thing, but it is not 100% certain. The same is true for Forth -- in theory any isolated snippet of Forth can mean anything, since it is trivially easy to modify how the compiler interprets text -- something that make Forth very powerful but at the same time potentially very confusing for a code maintainer. I don't have the link offhand, but a while back I came across a blog post suggesting you tend to either have a powerful language or powerful tools -- but not at the same time (except perhaps for Smalltalk :-). That is because if the language is very flexible, it becomes almost impossible to write IDE tools that can keep up with it in all its generality.

Now, since almost all Python code is written in a straightforward manner, one can still make such tools and find them useful. But likely there will aways be gotchas in such systems as long as they tie their operation closely to the notion of compiling one module at a time, compared to Smalltalk which ties itself to compiling one method/function at a time.

One of the things PataPata tried to do, and succeeded to some extent, was breaking the link between reloading a textual module and modifying a running Python program, yet it was still able to use a textual Python module as both an interchange format and also an image format (something even no Smalltalk has done to my knowledge, as all Smalltalk images I know of are binary, not human editable text).

One idea I have wanted to try for Python but never got around to it is to create a Smalltalk-like browser and build and modify classes on the fly by changing their objects and compiling only individual functions as they are changed; I could store the textual representation of functions in a repository with version control. Then, I could also still use Python modules as an interchange format, sort of like PataPata did but without prototypes. You would lose some of the generality of coding in Python (setting globals in a module and such) but you would essentially have a somewhat Smalltalk like environment to mess with (ignoring restarting from exceptions, which is very important in Smalltalk development, where much code ends up being written in the debugger as often as not; I'm not sure whether that part could be simulated with plain Python or whether it would require a VM change).

--Paul Fernhout

# xreload.py. """Alternative to reload(). This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """ import imp import sys import types def xreload(mod): """Reload a module in place, updating classes, methods and functions. Args: mod: a module object Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj. If oldobj is immutable, this simply returns newobj. Args: oldobj: the object to be updated newobj: the object used as the source for the update Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Oh, what a giveaway! Did you hear that? Did you hear that, eh? That's what I'm on about!" :-) This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm... Notice xreload's update is much smaller than the one I made with Michael's additional inspiration and ideas: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... In some ways, xreload's is better by including a general "Provide a hook for updating" concept. However xreload just ignores what to do with top level objects like lists or dicts -- xreload apparently throws away the updates to them. Notice it is not recursive -- for example does it handle nested classes? I'm not sure of all the reasons for the other differences -- it was tricky code I admit I never fully understood how to write thoroughly, mucking about with Python/Jython internals I only hazily understood. And even for the reload code I supplied, it is not always clear what to do with changes to top level objects -- do you want to update them or replace them? If you do update them, how deeply do you want the update to go? Notice xreload still has sections with comments like "Not something we recognize, just give up" (that's throwing away changes to top level lists and dicts and singletons and such), or "Cop-out: if the type changed, give up", and it can not handle in general the issue of side effects I mentioned (e.g. is a singleton instance recreated?). And of course you can't use it to restart an exception after making a change (though it is perhaps a piece of that puzzle). It does however deal with updating pointers to existing function as far as I can tell (so previously set GUI callbacks will still call the old code) -- and does it in the same way the previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code). So, the xreload module Guido supplies is nifty piece of code to have (assuming Python licensed?). Using it properly will make you more productive. However, it still requires building hooks into your application explicitly to call it for each module of interest; the Jython version I supplied includes a brief GUI control window listing all modules which could be made to work similarly under wx or tk. Still, elegant as it is (including the reload hook), even xreload does not handle all the important cases -- just (forgive me :-) the *easy* ones. When Guido supplies an xrestart.py :-) python 2.5 code module that lets you restart exceptions after having modified one of the functions in the stack trace, then I will admit it is something I have never seen before in Python and be suitably impressed. Is it even possible without modifying the VM? :-) There will remain the semantic problem of what to do with top level objects, but as I said, one can avoid that somewhat by just never expecting to reload an entire module at a time -- so using the core of these ideas to reload a method or class (which is what reload does, as it does not update other globals than code related ones). And then, a next bigger step is getting both xreload and xrestart (or related code) into general use into common Python IDEs like IDLE or PyDev. Otherwise, how is a beginner ever going to understand why you would want this xreload piece of code, or how it works, or when to use it? It's one thing to have a bit of code that suggest how something can be done -- it is another to weave an idea throughout a community and a set of application implementations. And the prevalence of those ideas of fine grained changes and fixing code using the debugger and then restarting is a big reason Smalltalk coders are more productive than Python coders, all other things being equal (which of course they are not given licensing, availability, libraries, community, and so on, which is why I use Python instead of Smalltalk. Can't stop pining for the fjords, I guess. :-). One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference. That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes. --Paul Fernhout Guido van Rossum wrote:

On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

...

To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).

# xreload.py.

"""Alternative to reload().

This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """

import imp import sys import types

def xreload(mod): """Reload a module in place, updating classes, methods and functions.

Args: mod: a module object

Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod

def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj.

If oldobj is immutable, this simply returns newobj.

Args: oldobj: the object to be updated newobj: the object used as the source for the update

Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj

def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc

def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth

def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass

On 2/25/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm...

I think I was inspired by something I heard someone say about Twisted's rebuild (?) module. But I've never seen the code.

Notice xreload's update is much smaller than the one I made with Michael's additional inspiration and ideas: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm...

Well it needs work. The version I checked into the py3k branch is a bit better and I hope to continue to develop it there.

In some ways, xreload's is better by including a general "Provide a hook for updating" concept. However xreload just ignores what to do with top level objects like lists or dicts -- xreload apparently throws away the updates to them.

No it just replaces the whole object. I was just punting on this; what are the required semantics?

Notice it is not recursive -- for example does it handle nested classes?

This was an omission. The svn version is, and should handle nested classes just fine.

I'm not sure of all the reasons for the other differences -- it was tricky code I admit I never fully understood how to write thoroughly, mucking about with Python/Jython internals I only hazily understood. And even for the reload code I supplied, it is not always clear what to do with changes to top level objects -- do you want to update them or replace them? If you do update them, how deeply do you want the update to go?

Right. I offer clean if limited semantics so you can write your code to work with it -- and a hook so you can extend it. A registration mechanism keyed on the class could easily be added (or using generic functions :-).

Notice xreload still has sections with comments like "Not something we recognize, just give up" (that's throwing away changes to top level lists and dicts and singletons and such),

No it throws away the old version.

or "Cop-out: if the type changed, give up", and it can not handle in general the issue of side effects I mentioned (e.g. is a singleton instance recreated?). And of course you can't use it to restart an exception after making a change (though it is perhaps a piece of that puzzle). It does however deal with updating pointers to existing function as far as I can tell (so previously set GUI callbacks will still call the old code) -- and does it in the same way the previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code).

There just is no way to handle *all* possibilities. How would you handle renamings?

So, the xreload module Guido supplies is nifty piece of code to have (assuming Python licensed?). Using it properly will make you more productive. However, it still requires building hooks into your application explicitly to call it for each module of interest; the Jython version I supplied includes a brief GUI control window listing all modules which could be made to work similarly under wx or tk.

The GUI part isn't supposed to be included. There's also the issue of what to do with dependencies; if x imports y, and y is reloaded, x probably needs to be reloaded too...

Still, elegant as it is (including the reload hook), even xreload does not handle all the important cases -- just (forgive me :-) the *easy* ones.

When Guido supplies an xrestart.py :-) python 2.5 code module that lets you restart exceptions after having modified one of the functions in the stack trace, then I will admit it is something I have never seen before in Python and be suitably impressed. Is it even possible without modifying the VM? :-)

It is impossible because we don't know whether the exception has a handler (an except block that traps it) until we've bubbled all the way up. I suggest it's pointless to try to emulate every single common lisp feature in Python. The resulting language would *be* common lisp and nobody would want it.

There will remain the semantic problem of what to do with top level objects, but as I said, one can avoid that somewhat by just never expecting to reload an entire module at a time -- so using the core of these ideas to reload a method or class (which is what reload does, as it does not update other globals than code related ones).

You seem to be building a whole body of misunderstandings on this one misconception.

And then, a next bigger step is getting both xreload and xrestart (or related code) into general use into common Python IDEs like IDLE or PyDev.

Well that's up to the IDE authors right? [Irrelevant paragraph snipped]

One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference.

Can you clarify this for someone who is not familiar with Smalltalk? (Your habit of explaining every idea by comparing it to how that idea was implemented in Smalltalk doesn't help the audience understand you -- you're preaching to the choir when you do that, and the rest of the congregation is lost.)

That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes.

Why do you care about avoidung VM changes? The VM changes (incrementally) at each minor Python release. --Guido

--Paul Fernhout

Guido van Rossum wrote:

...

On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:

...

To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).

# xreload.py.

"""Alternative to reload().

This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """

import imp import sys import types

def xreload(mod): """Reload a module in place, updating classes, methods and functions.

Args: mod: a module object

Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod

def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj.

If oldobj is immutable, this simply returns newobj.

Args: oldobj: the object to be updated newobj: the object used as the source for the update

Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj

def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc

def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth

def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass

_______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Paul D. Fernhout <pdfernhout <at> kurtz-fernhout.com> writes:

This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm...

FWIW, I still tinker with the reloading issue from time to time, in pursuit of a fully-interactive environment. My latest module is at: http://svn.brownspencer.com/pynote/trunk/Mupdaters.py. It's not documented for standalone use, but may contain some useful ideas, and I'd be happy to discuss it if anyone were interested. Incidentally, class bodies provide one of the hard problems for reloading. Ideally, for reloading, one would execute the class body in a proxy of the existing dict, as for modules, but this can't be achieved with the same exec <modulecode> in <proxyofexistingdict> approach. In order to work-around this, I wrote a pure-python compiler. With this, it becomes possible to modify the class definition bytecode to execute the class definitions in the existing class dict rather than a new one. The compiler itself is at: http://svn.brownspencer.com/pycompiler/branches/new_ast/ . However, I have not yet connected it into the reloading module. Michael

Paul D. Fernhout wrote:

Michael-

Thanks for the update. An entire Python compiler to make it work right?

No, the module I linked to is self-contained, although hardly user-friendly, I concede. I believe that custom-compilation *will* be required to tackle some corner cases, but I haven't implemented it.

Wow. I don't use many class methods so I've never noticed this problem. Thanks for putting this up, and under the Python license (according to the readme). http://svn.brownspencer.com/pycompiler/branches/new_ast/readme.txt

Yep, Python license.

Is this in any ways related to PyPy? http://codespeak.net/pypy/dist/pypy/doc/news.html

No. It's just an update/replacement for the stdlib compiler package which has several known problems, and which does not support the native Python 2.5 AST.

In the Jython reload case I have noticed a different problem, which is that a method with a call to super__ will fail. ...

Unfortunately, I don't know any details about Jython's implementation, but this doesn't surprise me.

I don't believe CPython reloading has any of those problems, since it is not doing the kind of reflection Jython does.

No, although it's pretty easy to come up with corner cases to confuse CPython reloading too.

When even Visual Basic or C# or Ruby can do this "edit and continue", I am worried about Python's future if it does not gain this feature. See for example: http://www.codinghorror.com/blog/archives/000507.html

I strongly agree. Edit and continue is intuitive for beginners, productive for experts, and anyway the standard set by competitors. ...

Note that I think that even if it was impossible in Python to resume from an exception (without major surgery in Python's innards), being able to break on the creation of any exception in the debugger, and then one by one tell the debugger to ignore some of them based on the type of exception or the module it is created in, would probably still be OK enough if you could then edit and continue before the exception was actually raised. This weaker form of "Edit and continue" would be harder to use, but probably still worthwhile given the right IDE tools.

I suspect that this requires a customized interpreter, and that pypy might be the vehicle to make this happen. Michael

Andreas- Here is one post on this issue with code I made previously for Jython: "ReloaderWindow 0.2 (improvements to selective reloading)" http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... This is a different (earlier) approach than for PataPata (PataPata uses prototypes and overriding __getattr__) and the linked code works with regular Python classes. The important thing to understand is that a Python VM manages a world of objects somewhat like a Smalltalk image, lumping all the objects from all the loaded modules into the same objectspace. You can just reference some of them (typically classes) by the way the current hierarchical Python namespace is set up. When you reload a module, it creates completely new class objects which just happen to have the same name as the old one and the namespace now points to those new objects (sort of like if every time you changed a method in Smalltalk the SystemDictionary Smalltalk was made to point to entirely new classes, leaving the old ones floating, which would obviously be a silly thing to do). So any existing instances of original classes point still to the old classes (which are no longer referenceable from the namespace). The various approaches try to get around this in various ways, such as in my approach above by loading a changed module into a temporary namespace and then copying the methods from it on top of the old classes. Also, things get a little more complicate than that, since Python often uses method pointers, equivalent in some way to Smalltalk blocks, to hook up GUI systems. So even if you keep the class the same and replace the methods, your running GUI window typically has bindings by reference to the old overwritten functions. So, as this approach I outlines does, you need to also keep the shell of the old functions and just overwrite what they actually do -- Kind of like if you kept CompiledMethods around and just overwrote their code. It's an awkward way to do things. Now, obviously, Smalltalk sometimes fails at updating GUIs when you tie actions to blocks, where recompiling methods produces new blocks (a big problem in older VisualWorks apps, not sure of the current status of VisualWorks or Squeak). Of course, in Smalltalk you could make the block call a method using a selector, which as least gives you a point of intervention. Similarly in Python, if you modify code which the bound function references, that newer code will be called by the old function's code as dispatch works differently depending whether you have a function pointer as opposed to an inline reference to another function in a Python function. In Python, I can get around this by wrapping any function pointer I give to a GUI callback using an object (like Java :-) where the callback then calls by name. PataPata did some of that internally, though obviously it adds extra layers of overhead to every GUI call, plus the programmer had to proactively add a wrapper, making code uglier, and so on. In Python, people often build specific hooks into their application to explicitly reload modules, including sometimes closing and reopening only one window inside a large application, to try to make the best of this. There are other attempts to do a more selective reloading like the approach I give above. Consider for example: "[Patches] [ python-Patches-1212921 ] option to allow reload to affect existing instances" http://mail.python.org/pipermail/patches/2005-September/018060.html There is some discussion at that link of the issue. And a reference there to another approach: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 From the first patch link: "Brett and I work at the same company. He made this patch because we were wasting hundreds of hours debugging reload problems or just quitting the application and restarting. I've been using this patch now for 3 months. It has saved our company hundreds of man-hours and we didn't have to change one line of existing code to make it work. If the community doesn't want to use it, I'm fine with that. I just want to make sure that you fully understand what you are giving up." I'm not sure what the disposition of that patch is. But, you can see from the responses there is a bit of a cultural problem here getting the value of the idea across. I did not look at the patch, but from the description I wonder if it still might suffer from the issue of allowing identities for functions to change (and so GUIs with actions previously linked to functions would still use the old behavior). There is also a more subtle issue for module reloading. Smalltalk is essentially a declarative language, while Python is essentially an imperative language (maybe there is a better word?). Essentially, compiling a Smalltalk class has no side effects (other than perhaps defining the class). But a Python module on loading can produce lots of side effects from defining metaclasses to affect future class loading to executing random bits of code (sort of like Forth in this sense). So, only modules without these side effects can be easily reloaded. Consider for example what if your module makes a singleton instance of something -- when you reload it, should it make a new singleton instance and leave the other one to still perhaps be used by old instances, or should the singleton instance get upgraded to the new class? Just one of many issues that happen when you are using Python's textual module reloading paradigm. Ideally, when you change a method in a text file you probably almost always want only that specific method to change with everything else being left alone. But that breaks the Python paradigm. Still, even with this problem, easier class reloading like I outlined above still gets you great value when developing new code, especially if you slightly alter how you layout your new modules with this limitation in mind. But reloading older code may present you with problems -- making maintenance using this technique harder. --Paul Fernhout Andreas Raab wrote:

Paul D. Fernhout wrote:

...

This "edit and continue" support is much more fine grained and easier to use than reloading Python modules, even with the variants I and others have created.

Where can I find out more about these variants and what they do? We are currently trying to enable a workflow that centers around a life environment and these issues become important for us. Are there any other life environments (besides Patapata which I have looked at) that are worth studying?