[Edu-sig] Reloading code (was Re: OLPC: first thoughts)
Guido van Rossum
guido at python.org
Sun Feb 25 21:25:11 CET 2007
On 2/25/07, Paul D. Fernhout <pdfernhout at kurtz-fernhout.com> wrote:
> This xreload module Guido supplies is similar to the approach I used for
> Jython in my earlier link.
> http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
> If you look at that thread, I acknowledge Michael Spencer as having
> supplied some of the more sophisticated reload logic which is similar to
> what Guido has posted here (not sure where Michael got it from of course,
> nor where Guido got xreload from). See Michael's first related post here:
> http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.html
I think I was inspired by something I heard someone say about
Twisted's rebuild (?) module. But I've never seen the code.
> Notice xreload's update is much smaller than the one I made with Michael's
> additional inspiration and ideas:
> http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
Well it needs work. The version I checked into the py3k branch is a
bit better and I hope to continue to develop it there.
> In some ways, xreload's is better by including a general "Provide a hook
> for updating" concept. However xreload just ignores what to do with top
> level objects like lists or dicts -- xreload apparently throws away the
> updates to them.
No it just replaces the whole object. I was just punting on this; what
are the required semantics?
> Notice it is not recursive -- for example does it handle
> nested classes?
This was an omission. The svn version is, and should handle nested
classes just fine.
> I'm not sure of all the reasons for the other differences
> -- it was tricky code I admit I never fully understood how to write
> thoroughly, mucking about with Python/Jython internals I only hazily
> understood. And even for the reload code I supplied, it is not always
> clear what to do with changes to top level objects -- do you want to
> update them or replace them? If you do update them, how deeply do you want
> the update to go?
Right. I offer clean if limited semantics so you can write your code
to work with it -- and a hook so you can extend it. A registration
mechanism keyed on the class could easily be added (or using generic
functions :-).
> Notice xreload still has sections with comments like "Not something we
> recognize, just give up" (that's throwing away changes to top level lists
> and dicts and singletons and such),
No it throws away the old version.
> or "Cop-out: if the type changed, give
> up", and it can not handle in general the issue of side effects I
> mentioned (e.g. is a singleton instance recreated?). And of course you
> can't use it to restart an exception after making a change (though it is
> perhaps a piece of that puzzle). It does however deal with updating
> pointers to existing function as far as I can tell (so previously set GUI
> callbacks will still call the old code) -- and does it in the same way the
> previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code).
There just is no way to handle *all* possibilities. How would you
handle renamings?
> So, the xreload module Guido supplies is nifty piece of code to have
> (assuming Python licensed?). Using it properly will make you more
> productive. However, it still requires building hooks into your
> application explicitly to call it for each module of interest; the Jython
> version I supplied includes a brief GUI control window listing all modules
> which could be made to work similarly under wx or tk.
The GUI part isn't supposed to be included. There's also the issue of
what to do with dependencies; if x imports y, and y is reloaded, x
probably needs to be reloaded too...
> Still, elegant as it is (including the reload hook), even xreload does not
> handle all the important cases -- just (forgive me :-) the *easy* ones.
>
> When Guido supplies an xrestart.py :-) python 2.5 code module that lets
> you restart exceptions after having modified one of the functions in the
> stack trace, then I will admit it is something I have never seen before in
> Python and be suitably impressed. Is it even possible without modifying
> the VM? :-)
It is impossible because we don't know whether the exception has a
handler (an except block that traps it) until we've bubbled all the
way up.
I suggest it's pointless to try to emulate every single common lisp
feature in Python. The resulting language would *be* common lisp and
nobody would want it.
> There will remain the semantic problem of what to do with top level
> objects, but as I said, one can avoid that somewhat by just never
> expecting to reload an entire module at a time -- so using the core of
> these ideas to reload a method or class (which is what reload does, as it
> does not update other globals than code related ones).
You seem to be building a whole body of misunderstandings on this one
misconception.
> And then, a next bigger step is getting both xreload and xrestart (or
> related code) into general use into common Python IDEs like IDLE or PyDev.
Well that's up to the IDE authors right?
[Irrelevant paragraph snipped]
> One impressive thing about the Python design which I liked from all this
> was how Python separates the notion of execution code from a pointer
> reference.
Can you clarify this for someone who is not familiar with Smalltalk?
(Your habit of explaining every idea by comparing it to how that idea
was implemented in Smalltalk doesn't help the audience understand you
-- you're preaching to the choir when you do that, and the rest of the
congregation is lost.)
> That is what makes all these reloading tricks possible. And my
> hat goes off to Guido for having included the extra level of indirection
> which makes this feasible. I can hope that generality and late bindingness
> might also make possible restarting an exception without VM changes.
Why do you care about avoidung VM changes? The VM changes
(incrementally) at each minor Python release.
--Guido
> --Paul Fernhout
>
> Guido van Rossum wrote:
> > On 2/24/07, Paul D. Fernhout <pdfernhout at kurtz-fernhout.com> wrote:
> >>To step back for a minute, the fundamental problem here is that for
> >>whatever reason a programmer wants to modify just one method of an already
> >>loaded Python class (which came from a textual module which was loaded
> >>already), save the change somewhere so it can be reloaded later
> >>(overwriting part of the textual module?), and also have the program start
> >>using the new behavior for existing instances without any other side
> >>effects arising from recompiling this one change. In practice, this is
> >>trivial to do in almost any Smalltalk system; it is hard if not impossible
> >>to do in any widely used Python IDE or program (even when a Python shell
> >>is embedded).
> >
> > # xreload.py.
> >
> > """Alternative to reload().
> >
> > This works by executing the module in a scratch namespace, and then
> > patching classes, methods and functions. This avoids the need to
> > patch instances. New objects are copied into the target namespace.
> > """
> >
> > import imp
> > import sys
> > import types
> >
> >
> > def xreload(mod):
> > """Reload a module in place, updating classes, methods and functions.
> >
> > Args:
> > mod: a module object
> >
> > Returns:
> > The (updated) input object itself.
> > """
> > # Get the module name, e.g. 'foo.bar.whatever'
> > modname = mod.__name__
> > # Get the module namespace (dict) early; this is part of the type check
> > modns = mod.__dict__
> > # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever'
> > i = modname.rfind(".")
> > if i >= 0:
> > pkgname, modname = modname[:i], modname[i+1:]
> > else:
> > pkgname = None
> > # Compute the search path
> > if pkgname:
> > # We're not reloading the package, only the module in it
> > pkg = sys.modules[pkgname]
> > path = pkg.__path__ # Search inside the package
> > else:
> > # Search the top-level module path
> > pkg = None
> > path = None # Make find_module() uses the default search path
> > # Find the module; may raise ImportError
> > (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path)
> > # Turn it into a code object
> > try:
> > # Is it Python source code or byte code read from a file?
> > # XXX Could handle frozen modules, zip-import modules
> > if kind not in (imp.PY_COMPILED, imp.PY_SOURCE):
> > # Fall back to built-in reload()
> > return reload(mod)
> > if kind == imp.PY_SOURCE:
> > source = stream.read()
> > code = compile(source, filename, "exec")
> > else:
> > code = marshal.load(stream)
> > finally:
> > if stream:
> > stream.close()
> > # Execute the code im a temporary namespace; if this fails, no changes
> > tmpns = {}
> > exec(code, tmpns)
> > # Now we get to the hard part
> > oldnames = set(modns)
> > newnames = set(tmpns)
> > # Add newly introduced names
> > for name in newnames - oldnames:
> > modns[name] = tmpns[name]
> > # Delete names that are no longer current
> > for name in oldnames - newnames - set(["__name__"]):
> > del modns[name]
> > # Now update the rest in place
> > for name in oldnames & newnames:
> > modns[name] = _update(modns[name], tmpns[name])
> > # Done!
> > return mod
> >
> >
> > def _update(oldobj, newobj):
> > """Update oldobj, if possible in place, with newobj.
> >
> > If oldobj is immutable, this simply returns newobj.
> >
> > Args:
> > oldobj: the object to be updated
> > newobj: the object used as the source for the update
> >
> > Returns:
> > either oldobj, updated in place, or newobj.
> > """
> > if type(oldobj) is not type(newobj):
> > # Cop-out: if the type changed, give up
> > return newobj
> > if hasattr(newobj, "__reload_update__"):
> > # Provide a hook for updating
> > return newobj.__reload_update__(oldobj)
> > if isinstance(newobj, types.ClassType):
> > return _update_class(oldobj, newobj)
> > if isinstance(newobj, types.FunctionType):
> > return _update_function(oldobj, newobj)
> > if isinstance(newobj, types.MethodType):
> > return _update_method(oldobj, newobj)
> > # XXX Support class methods, static methods, other decorators
> > # Not something we recognize, just give up
> > return newobj
> >
> >
> > def _update_function(oldfunc, newfunc):
> > """Update a function object."""
> > oldfunc.__doc__ = newfunc.__doc__
> > oldfunc.__dict__.update(newfunc.__dict__)
> > oldfunc.func_code = newfunc.func_code
> > oldfunc.func_defaults = newfunc.func_defaults
> > # XXX What else?
> > return oldfunc
> >
> >
> > def _update_method(oldmeth, newmeth):
> > """Update a method object."""
> > # XXX What if im_func is not a function?
> > _update_function(oldmeth.im_func, newmeth.im_func)
> > return oldmeth
> >
> >
> > def _update_class(oldclass, newclass):
> > """Update a class object."""
> > # XXX What about __slots__?
> > olddict = oldclass.__dict__
> > newdict = newclass.__dict__
> > oldnames = set(olddict)
> > newnames = set(newdict)
> > for name in newnames - oldnames:
> > setattr(oldclass, name, newdict[name])
> > for name in oldnames - newnames:
> > delattr(oldclass, name)
> > for name in oldnames & newnames - set(["__dict__", "__doc__"]):
> > setattr(oldclass, name, newdict[name])
> > return oldclass
> >
> _______________________________________________
> Edu-sig mailing list
> Edu-sig at python.org
> http://mail.python.org/mailman/listinfo/edu-sig
>
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
More information about the Edu-sig
mailing list