OLPC: first thoughts on the first keynote at Pycon
I just attended a great talk on OLPC. I thought I should share my first reactions with folks not fortunate enough to be able to attend. BTW, if anyone feels this is a bit of an abuse of this list, feel free to let me know (I won't be offended). I recognize that these are not well polished notes and perhaps should rather find their way on my blog. The first part was about school and learning - great stuff. Goal: changing the way kids learn (no mention as this point about laptops). Art would have been pleasantly surprised. This was done in a "non-preachy" way, unlike the way some home-schooling advocates present their stuff. (no offense meant to anyone on this list). A big chunk of the talk dealt with technical/hardware issues - all done in a very relevant way, but not exactly relevant for edu-sig. Some excellent reasons were given as to why Python was chosen as *the* development language. There is actually a button on the keyboard which is meant to "display the source"; click on it and you're presented with the actual Python source code used to run the application being viewed/used. The user can then modify the source (like smalltalk I guess, like Paul F. [?] sometimes mentions on this list) and run the modified version. One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rurple within crunchy). There is a real need to have more people involved in the development. This project presents a real opportunity to "make a real difference" to a lot of kids out there. Overall, I was extremely impressed with the talk - it was a job well done and a fantastic start to the conference. André
Thanks for the update André. At Pycons I've attended @ GWU in DC, we were promised screencasts at least if the speaker signed a waiver allowing it. Was it your impression that this keynote will have an URL, maybe has one already? I was unaware of the 'view source' button. Must be a different brand. Kirby
As far as I know, there will be some screencasts/recordings available. I haven't seen a link to the slides so far and was planning to find out. On 2/23/07, kirby urner <kirby.urner@gmail.com> wrote:
Thanks for the update André.
At Pycons I've attended @ GWU in DC, we were promised screencasts at least if the speaker signed a waiver allowing it.
Was it your impression that this keynote will have an URL, maybe has one already?
I was unaware of the 'view source' button. Must be a different brand.
Kirby
I would second Andre's assessment. It just may have been one of the best keynotes of any PyCon, whatever you think of OLPC. And I think most people in the room ended up with a pretty positive feeling for what the OLPC folks want to do. Vern Andre Roberge wrote:
I just attended a great talk on OLPC. I thought I should share my first reactions with folks not fortunate enough to be able to attend.
BTW, if anyone feels this is a bit of an abuse of this list, feel free to let me know (I won't be offended). I recognize that these are not well polished notes and perhaps should rather find their way on my blog.
The first part was about school and learning - great stuff. Goal: changing the way kids learn (no mention as this point about laptops). Art would have been pleasantly surprised. This was done in a "non-preachy" way, unlike the way some home-schooling advocates present their stuff. (no offense meant to anyone on this list).
A big chunk of the talk dealt with technical/hardware issues - all done in a very relevant way, but not exactly relevant for edu-sig.
Some excellent reasons were given as to why Python was chosen as *the* development language. There is actually a button on the keyboard which is meant to "display the source"; click on it and you're presented with the actual Python source code used to run the application being viewed/used. The user can then modify the source (like smalltalk I guess, like Paul F. [?] sometimes mentions on this list) and run the modified version.
One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rurple within crunchy).
There is a real need to have more people involved in the development. This project presents a real opportunity to "make a real difference" to a lot of kids out there.
Overall, I was extremely impressed with the talk - it was a job well done and a fantastic start to the conference.
André _______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig
On 2/23/07, Vern Ceder <vceder@canterburyschool.org> wrote:
I would second Andre's assessment. It just may have been one of the best keynotes of any PyCon, whatever you think of OLPC. And I think most people in the room ended up with a pretty positive feeling for what the OLPC folks want to do.
Vern
I haven't met anyone "anti" OLPC, just some who've felt their plans to further computerize the curriculum were not on hold while we waited for such hardware to materialize. Mark Shuttleworth for example. The forward progress of that South African process was in no way contingent upon the success of failure (e.g. perpetual postponement) of the projected OLPC result (one laptop per child). For example, given TuxLabs, shared equipment, it's already possible to do Squeak-based robotics or whatever. Kirby
Thanks Andre for the update I really appreciate. I hope someone recorded that if so it looks like it should appear in the future on this page (items): http://us.pycon.org/zope/talks/2007/fri/track3/004/talkDetails I'm also hoping your talk for crunchy will be recorded. It would also be a good occasion to have an interview with Ron Stephens for the Python 411 podcast (I'm sure he will be interested). francois On 2/23/07, Andre Roberge <andre.roberge@gmail.com> wrote:
I just attended a great talk on OLPC. I thought I should share my first reactions with folks not fortunate enough to be able to attend.
BTW, if anyone feels this is a bit of an abuse of this list, feel free to let me know (I won't be offended). I recognize that these are not well polished notes and perhaps should rather find their way on my blog.
The first part was about school and learning - great stuff. Goal: changing the way kids learn (no mention as this point about laptops). Art would have been pleasantly surprised. This was done in a "non-preachy" way, unlike the way some home-schooling advocates present their stuff. (no offense meant to anyone on this list).
A big chunk of the talk dealt with technical/hardware issues - all done in a very relevant way, but not exactly relevant for edu-sig.
Some excellent reasons were given as to why Python was chosen as *the* development language. There is actually a button on the keyboard which is meant to "display the source"; click on it and you're presented with the actual Python source code used to run the application being viewed/used. The user can then modify the source (like smalltalk I guess, like Paul F. [?] sometimes mentions on this list) and run the modified version.
One open issue (as I understand it) is that of finding the "best practice" for plugins. The idea is that the core programs should be as small as possible but easy to extend via plugins. I thought that there already was a "well known and best way" to design plugins - and it was on my list of things to learn about (to eventually incorporate rurple within crunchy).
There is a real need to have more people involved in the development. This project presents a real opportunity to "make a real difference" to a lot of kids out there.
Overall, I was extremely impressed with the talk - it was a job well done and a fantastic start to the conference.
André _______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig
On 2/23/07, francois schnell <francois.schnell@gmail.com> wrote:
I hope someone recorded that if so it looks like it should appear in the future on this page (items):
http://us.pycon.org/zope/talks/2007/fri/track3/004/talkDetails
Well, still nothing. I imagine it will be available in weeks or months to come. During the same time frame there was another OLPC presentation in Bruxelles at the FOSDEM 2007 ('Free and Open source Software Developers' European Meeting'). The speaker was Jime Gettys, http://en.wikipedia.org/wiki/Jim_Gettys and the video is available here: http://ftp.belnet.be/mirrors/FOSDEM/2007/FOSDEM2007-OLPC.ogg The video format is Ogg/Theora. It will play fine in VLC player for example. http://www.videolan.org/ -- francois schnell
I doubt this key and related process works anything like Smalltalk, based on what you describe, though I have not seen it in action. In Smalltalk, you can *easily* modify the source of a small part of a running program (including most windows callbacks [except for blocks on some systems]) and have the changes affect the running program (so no restart is required). For small programs this is not that noticeable, but if you are developing a large complex application, where it can sometimes take minutes to get the program back to the state you are testing for (especially if it needs to process large amounts of data), or where exceptions may be hard to repeat, this can be a *huge* time saver. So to when you are working on a large application which is always running 24X7 (large financial and simulation and embedded and server projects are often like this). This "edit and continue" support is much more fine grained and easier to use than reloading Python modules, even with the variants I and others have created. Also, if you get an exception in most Smalltalk, you can modify the offending code right there and then and restart just the method, retaining the entire calling history and values for the call such as it was like the exception never happened, without needing to restart the entire program (try that in Python. :-) That is why I feel that Smalltalk programming all other things being equal is several times more productive than Python. Of course, all things are not always equal, so libraries, other programmer's familiarity with C syntax, installed interpreters, relevant libraries, licensing, and so on effect the decision. That is why I use Python and typically feel more productive *overall* than working in a typical Smalltalk despite knowing that when I actually do write and debug my own code that effort by itself is much less productive otherwise than doing the equivalent in Smalltalk (if all other things were equal, which they are not, especially typically in relation to licensing, which may change). Well, I'll say Python's indentational syntax is also a big plus which I prefer over Smalltalk's, while I otherwise prefer Smalltalk's keyword syntax for being more self-documenting. :-). Anyway, there is no inherent reason Python can not do what Smalltalk does -- although it might require some minor changes to the VM to debug and restart exceptions; Python just does not do it, I suspect in part because key Pythoneers may just not understand what they are missing and think module reloading is the same thing. :-) I predict if Guido added this one set of related features to Python, it would be a very short time before everyone wondered how they lived without it before. :-) However, it would also need to interact with the difference between a system hosted in files and one hosted in an image, so that part of it would be somewhat harder to manage that the internal mechanics. Thanks in any case for the report! --Paul Fernhout Andre Roberge wrote:
Some excellent reasons were given as to why Python was chosen as *the* development language. There is actually a button on the keyboard which is meant to "display the source"; click on it and you're presented with the actual Python source code used to run the application being viewed/used. The user can then modify the source (like smalltalk I guess, like Paul F. [?] sometimes mentions on this list) and run the modified version.
On 2/23/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
Anyway, there is no inherent reason Python can not do what Smalltalk does -- although it might require some minor changes to the VM to debug and restart exceptions; Python just does not do it, I suspect in part because key Pythoneers may just not understand what they are missing and think module reloading is the same thing. :-) I predict if Guido added this one set of related features to Python, it would be a very short time before everyone wondered how they lived without it before. :-)
You may be right, although if I understand your aright, it's neither the syntax nor semantics of Python that are at issue, but the IDE and its ability to permit debugging on the fly, using thrown exceptions as restart points of some kind. Sounds cool. Don't have to wait for Guido, might be able to retrofit it for earlier Pythons. Kirby
There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with is specifically in how you write your own code, but that is not a general solution. See my previous note to Andreas for more details, including a related Python patch someone submitted in 2005. --Paul Fernhout kirby urner wrote:
On 2/23/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
Anyway, there is no inherent reason Python can not do what Smalltalk does -- although it might require some minor changes to the VM to debug and restart exceptions; Python just does not do it, I suspect in part because key Pythoneers may just not understand what they are missing and think module reloading is the same thing. :-) I predict if Guido added this one set of related features to Python, it would be a very short time before everyone wondered how they lived without it before. :-)
You may be right, although if I understand your aright, it's neither the syntax nor semantics of Python that are at issue, but the IDE and its ability to permit debugging on the fly, using thrown exceptions as restart points of some kind. Sounds cool. Don't have to wait for Guido, might be able to retrofit it for earlier Pythons.
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with is specifically in how you write your own code, but that is not a general solution.
Not sure I follow yet. A module loads top to bottom, with lower defs premised on those previously mentioned. Is that what you mean? Once everything is loaded, it's more like a __dict__, i.e. the namespace of the module of accessible, either via dot notation, or directly if the names are top level.
See my previous note to Andreas for more details, including a related Python patch someone submitted in 2005.
--Paul Fernhout
I'll wait for the IDE that passes muster as a "what you meant". In no such IDE is forthcoming, I'm not inclined to blame Python the language yet, as your improvements sound tangential to the design concepts. So far. Maybe I'm missing something, so keep persuading me if you like. Kirby
kirby urner wrote:
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with [it] specifically in how you write your own code, but that is not a general solution.
Not sure I follow yet. A module loads top to bottom, with lower defs premised on those previously mentioned. Is that what you mean? Once everything is loaded, it's more like a __dict__, i.e. the namespace of the module of accessible, either via dot notation, or directly if the names are top level.
To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded). Unfortunately, the paradigm used by every Python IDE I've tried is to reload an entire textual module (or more typically, entire program) at a time for even the slightest change to one function. Embedded Python shells generally allow you to redefine a function if you have a copy of the code, but they offer no way to save the code. Most Smalltalks uses a different paradigm, where code is presented to the user one function at a time in a browser and is compiled one function at a time. Yes, there are cases where people "filein" Smalltalk code defining a complex program, but such fileins are generally considered an *interchange* format, not a preferred program representation for editing unlike as is usually the case with Python. Consider the meaning of an arbitrary piece of Python code near the bottom of a textual module. Essentially, you have no idea what it means if the original author has used some Python bells and whistles. For example, he or she could have defined a metaclass where every nested "def" under a class was converted to, say, an uppercase string and stored under a key that was the numerical hash of the function name (with no functions actually defined for that class perhaps). The specific metaclass behavior may even hinge on the current state of a global which has been modified several times during the course of loading the module. So essentially, you have no way of knowing for sure what any apparent Python code really means by isolated inspection. And because any module can run any arbitrary Python code, without actually running the Python program (or doing the equivalent analysis), you can never be sure what side effects loading a module has. Now, Smalltalk has metaclasses too, but in practice, because of the way code is presented to the user and edited and recompiled one method/function at a time, the context makes fairly clear what is going to happen when that snippet of code you just changed is compiled. The big difference is really the effective unit of compilation -- the complex module in Python or the simple method/function in Smalltalk. Now, this is rarely a problem the *first* time a module is loaded, but it generally becomes a problem when a module is *reloaded*. If you only treated as module as an *interchange* format, and then modified the live classes using a tool which only works on regular classes (like PataPata does), there is no need to reload the module, so this potential problem related to parsing a modules meaning via an IDE tool remains only potential, and also avoided is the possibility reloading a module might have side effects. (In practice, anything still depends on mapping from a function back to its source text, and this may go wrong for various reasons... :-) Naturally, this kind of major redefinition is rarely done, and it would create lots of confusion, but it is possible, so IDE tools that do not support it are incomplete. This is a perrenial problem with, say, C, where you can make all sorts of macros and so never know just exactly what arbitrary C code out of context does (see the obfuscated code contests...). And it means that you can't get a simple one-to-one mapping of a section of a file that looks like it defines a function and an actual function reliably without analyzing the entire program. Yes, 99.99% of the time Python code does the obvious thing, but it is not 100% certain. The same is true for Forth -- in theory any isolated snippet of Forth can mean anything, since it is trivially easy to modify how the compiler interprets text -- something that make Forth very powerful but at the same time potentially very confusing for a code maintainer. I don't have the link offhand, but a while back I came across a blog post suggesting you tend to either have a powerful language or powerful tools -- but not at the same time (except perhaps for Smalltalk :-). That is because if the language is very flexible, it becomes almost impossible to write IDE tools that can keep up with it in all its generality. Now, since almost all Python code is written in a straightforward manner, one can still make such tools and find them useful. But likely there will aways be gotchas in such systems as long as they tie their operation closely to the notion of compiling one module at a time, compared to Smalltalk which ties itself to compiling one method/function at a time. One of the things PataPata tried to do, and succeeded to some extent, was breaking the link between reloading a textual module and modifying a running Python program, yet it was still able to use a textual Python module as both an interchange format and also an image format (something even no Smalltalk has done to my knowledge, as all Smalltalk images I know of are binary, not human editable text). One idea I have wanted to try for Python but never got around to it is to create a Smalltalk-like browser and build and modify classes on the fly by changing their objects and compiling only individual functions as they are changed; I could store the textual representation of functions in a repository with version control. Then, I could also still use Python modules as an interchange format, sort of like PataPata did but without prototypes. You would lose some of the generality of coding in Python (setting globals in a module and such) but you would essentially have a somewhat Smalltalk like environment to mess with (ignoring restarting from exceptions, which is very important in Smalltalk development, where much code ends up being written in the debugger as often as not; I'm not sure whether that part could be simulated with plain Python or whether it would require a VM change). --Paul Fernhout
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
kirby urner wrote:
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
There may be one major semantical issue, in terms of the meaning of side effects when loading a module (e.g. defining singletons, opening files, etc.) which is hard to deal with generically with Python. You can deal with [it] specifically in how you write your own code, but that is not a general solution.
Not sure I follow yet. A module loads top to bottom, with lower defs premised on those previously mentioned. Is that what you mean? Once everything is loaded, it's more like a __dict__, i.e. the namespace of the module of accessible, either via dot notation, or directly if the names are top level.
To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).
Unfortunately, the paradigm used by every Python IDE I've tried is to reload an entire textual module (or more typically, entire program) at a time for even the slightest change to one function. Embedded Python shells generally allow you to redefine a function if you have a copy of the code, but they offer no way to save the code. Most Smalltalks uses a different paradigm, where code is presented to the user one function at a time in a browser and is compiled one function at a time. Yes, there are cases where people "filein" Smalltalk code defining a complex program, but such fileins are generally considered an *interchange* format, not a preferred program representation for editing unlike as is usually the case with Python.
Consider the meaning of an arbitrary piece of Python code near the bottom of a textual module. Essentially, you have no idea what it means if the original author has used some Python bells and whistles. For example, he or she could have defined a metaclass where every nested "def" under a class was converted to, say, an uppercase string and stored under a key that was the numerical hash of the function name (with no functions actually defined for that class perhaps). The specific metaclass behavior may even hinge on the current state of a global which has been modified several times during the course of loading the module. So essentially, you have no way of knowing for sure what any apparent Python code really means by isolated inspection. And because any module can run any arbitrary Python code, without actually running the Python program (or doing the equivalent analysis), you can never be sure what side effects loading a module has. Now, Smalltalk has metaclasses too, but in practice, because of the way code is presented to the user and edited and recompiled one method/function at a time, the context makes fairly clear what is going to happen when that snippet of code you just changed is compiled. The big difference is really the effective unit of compilation -- the complex module in Python or the simple method/function in Smalltalk.
Now, this is rarely a problem the *first* time a module is loaded, but it generally becomes a problem when a module is *reloaded*. If you only treated as module as an *interchange* format, and then modified the live classes using a tool which only works on regular classes (like PataPata does), there is no need to reload the module, so this potential problem related to parsing a modules meaning via an IDE tool remains only potential, and also avoided is the possibility reloading a module might have side effects. (In practice, anything still depends on mapping from a function back to its source text, and this may go wrong for various reasons... :-)
Naturally, this kind of major redefinition is rarely done, and it would create lots of confusion, but it is possible, so IDE tools that do not support it are incomplete. This is a perrenial problem with, say, C, where you can make all sorts of macros and so never know just exactly what arbitrary C code out of context does (see the obfuscated code contests...). And it means that you can't get a simple one-to-one mapping of a section of a file that looks like it defines a function and an actual function reliably without analyzing the entire program. Yes, 99.99% of the time Python code does the obvious thing, but it is not 100% certain. The same is true for Forth -- in theory any isolated snippet of Forth can mean anything, since it is trivially easy to modify how the compiler interprets text -- something that make Forth very powerful but at the same time potentially very confusing for a code maintainer. I don't have the link offhand, but a while back I came across a blog post suggesting you tend to either have a powerful language or powerful tools -- but not at the same time (except perhaps for Smalltalk :-). That is because if the language is very flexible, it becomes almost impossible to write IDE tools that can keep up with it in all its generality.
Now, since almost all Python code is written in a straightforward manner, one can still make such tools and find them useful. But likely there will aways be gotchas in such systems as long as they tie their operation closely to the notion of compiling one module at a time, compared to Smalltalk which ties itself to compiling one method/function at a time.
One of the things PataPata tried to do, and succeeded to some extent, was breaking the link between reloading a textual module and modifying a running Python program, yet it was still able to use a textual Python module as both an interchange format and also an image format (something even no Smalltalk has done to my knowledge, as all Smalltalk images I know of are binary, not human editable text).
One idea I have wanted to try for Python but never got around to it is to create a Smalltalk-like browser and build and modify classes on the fly by changing their objects and compiling only individual functions as they are changed; I could store the textual representation of functions in a repository with version control. Then, I could also still use Python modules as an interchange format, sort of like PataPata did but without prototypes. You would lose some of the generality of coding in Python (setting globals in a module and such) but you would essentially have a somewhat Smalltalk like environment to mess with (ignoring restarting from exceptions, which is very important in Smalltalk development, where much code ends up being written in the debugger as often as not; I'm not sure whether that part could be simulated with plain Python or whether it would require a VM change).
--Paul Fernhout
# xreload.py. """Alternative to reload(). This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """ import imp import sys import types def xreload(mod): """Reload a module in place, updating classes, methods and functions. Args: mod: a module object Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj. If oldobj is immutable, this simply returns newobj. Args: oldobj: the object to be updated newobj: the object used as the source for the update Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
# xreload.py.
"""Alternative to reload().
This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """
Sweeeet! Thanks a zillion. Cheers, - Andreas
"Oh, what a giveaway! Did you hear that? Did you hear that, eh? That's what I'm on about!" :-) This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm... Notice xreload's update is much smaller than the one I made with Michael's additional inspiration and ideas: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... In some ways, xreload's is better by including a general "Provide a hook for updating" concept. However xreload just ignores what to do with top level objects like lists or dicts -- xreload apparently throws away the updates to them. Notice it is not recursive -- for example does it handle nested classes? I'm not sure of all the reasons for the other differences -- it was tricky code I admit I never fully understood how to write thoroughly, mucking about with Python/Jython internals I only hazily understood. And even for the reload code I supplied, it is not always clear what to do with changes to top level objects -- do you want to update them or replace them? If you do update them, how deeply do you want the update to go? Notice xreload still has sections with comments like "Not something we recognize, just give up" (that's throwing away changes to top level lists and dicts and singletons and such), or "Cop-out: if the type changed, give up", and it can not handle in general the issue of side effects I mentioned (e.g. is a singleton instance recreated?). And of course you can't use it to restart an exception after making a change (though it is perhaps a piece of that puzzle). It does however deal with updating pointers to existing function as far as I can tell (so previously set GUI callbacks will still call the old code) -- and does it in the same way the previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code). So, the xreload module Guido supplies is nifty piece of code to have (assuming Python licensed?). Using it properly will make you more productive. However, it still requires building hooks into your application explicitly to call it for each module of interest; the Jython version I supplied includes a brief GUI control window listing all modules which could be made to work similarly under wx or tk. Still, elegant as it is (including the reload hook), even xreload does not handle all the important cases -- just (forgive me :-) the *easy* ones. When Guido supplies an xrestart.py :-) python 2.5 code module that lets you restart exceptions after having modified one of the functions in the stack trace, then I will admit it is something I have never seen before in Python and be suitably impressed. Is it even possible without modifying the VM? :-) There will remain the semantic problem of what to do with top level objects, but as I said, one can avoid that somewhat by just never expecting to reload an entire module at a time -- so using the core of these ideas to reload a method or class (which is what reload does, as it does not update other globals than code related ones). And then, a next bigger step is getting both xreload and xrestart (or related code) into general use into common Python IDEs like IDLE or PyDev. Otherwise, how is a beginner ever going to understand why you would want this xreload piece of code, or how it works, or when to use it? It's one thing to have a bit of code that suggest how something can be done -- it is another to weave an idea throughout a community and a set of application implementations. And the prevalence of those ideas of fine grained changes and fixing code using the debugger and then restarting is a big reason Smalltalk coders are more productive than Python coders, all other things being equal (which of course they are not given licensing, availability, libraries, community, and so on, which is why I use Python instead of Smalltalk. Can't stop pining for the fjords, I guess. :-). One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference. That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes. --Paul Fernhout Guido van Rossum wrote:
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).
# xreload.py.
"""Alternative to reload().
This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """
import imp import sys import types
def xreload(mod): """Reload a module in place, updating classes, methods and functions.
Args: mod: a module object
Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod
def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj.
If oldobj is immutable, this simply returns newobj.
Args: oldobj: the object to be updated newobj: the object used as the source for the update
Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj
def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc
def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth
def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass
One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference. That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes.
--Paul Fernhout
You might want to raise your ideas on the IronPython list as well. As you know from vast Jython experience, there's no one Python VM, but any number, depending on which people decide which system language to use for that purpose. So far, C, Java and C# have been among the primary choices, but maybe those cell phones are using something else. I've seen demos of the Microsoft debugger stepping through Python then switching to C# as it follows the source to a deeper layer, then back out to Python again. But I don't recall if the IDE permitted editing on the fly or not, if paused at exceptions. Kirby
On 2/25/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm...
I think I was inspired by something I heard someone say about Twisted's rebuild (?) module. But I've never seen the code.
Notice xreload's update is much smaller than the one I made with Michael's additional inspiration and ideas: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm...
Well it needs work. The version I checked into the py3k branch is a bit better and I hope to continue to develop it there.
In some ways, xreload's is better by including a general "Provide a hook for updating" concept. However xreload just ignores what to do with top level objects like lists or dicts -- xreload apparently throws away the updates to them.
No it just replaces the whole object. I was just punting on this; what are the required semantics?
Notice it is not recursive -- for example does it handle nested classes?
This was an omission. The svn version is, and should handle nested classes just fine.
I'm not sure of all the reasons for the other differences -- it was tricky code I admit I never fully understood how to write thoroughly, mucking about with Python/Jython internals I only hazily understood. And even for the reload code I supplied, it is not always clear what to do with changes to top level objects -- do you want to update them or replace them? If you do update them, how deeply do you want the update to go?
Right. I offer clean if limited semantics so you can write your code to work with it -- and a hook so you can extend it. A registration mechanism keyed on the class could easily be added (or using generic functions :-).
Notice xreload still has sections with comments like "Not something we recognize, just give up" (that's throwing away changes to top level lists and dicts and singletons and such),
No it throws away the old version.
or "Cop-out: if the type changed, give up", and it can not handle in general the issue of side effects I mentioned (e.g. is a singleton instance recreated?). And of course you can't use it to restart an exception after making a change (though it is perhaps a piece of that puzzle). It does however deal with updating pointers to existing function as far as I can tell (so previously set GUI callbacks will still call the old code) -- and does it in the same way the previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code).
There just is no way to handle *all* possibilities. How would you handle renamings?
So, the xreload module Guido supplies is nifty piece of code to have (assuming Python licensed?). Using it properly will make you more productive. However, it still requires building hooks into your application explicitly to call it for each module of interest; the Jython version I supplied includes a brief GUI control window listing all modules which could be made to work similarly under wx or tk.
The GUI part isn't supposed to be included. There's also the issue of what to do with dependencies; if x imports y, and y is reloaded, x probably needs to be reloaded too...
Still, elegant as it is (including the reload hook), even xreload does not handle all the important cases -- just (forgive me :-) the *easy* ones.
When Guido supplies an xrestart.py :-) python 2.5 code module that lets you restart exceptions after having modified one of the functions in the stack trace, then I will admit it is something I have never seen before in Python and be suitably impressed. Is it even possible without modifying the VM? :-)
It is impossible because we don't know whether the exception has a handler (an except block that traps it) until we've bubbled all the way up. I suggest it's pointless to try to emulate every single common lisp feature in Python. The resulting language would *be* common lisp and nobody would want it.
There will remain the semantic problem of what to do with top level objects, but as I said, one can avoid that somewhat by just never expecting to reload an entire module at a time -- so using the core of these ideas to reload a method or class (which is what reload does, as it does not update other globals than code related ones).
You seem to be building a whole body of misunderstandings on this one misconception.
And then, a next bigger step is getting both xreload and xrestart (or related code) into general use into common Python IDEs like IDLE or PyDev.
Well that's up to the IDE authors right? [Irrelevant paragraph snipped]
One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference.
Can you clarify this for someone who is not familiar with Smalltalk? (Your habit of explaining every idea by comparing it to how that idea was implemented in Smalltalk doesn't help the audience understand you -- you're preaching to the choir when you do that, and the rest of the congregation is lost.)
That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes.
Why do you care about avoidung VM changes? The VM changes (incrementally) at each minor Python release. --Guido
--Paul Fernhout
Guido van Rossum wrote:
On 2/24/07, Paul D. Fernhout <pdfernhout@kurtz-fernhout.com> wrote:
To step back for a minute, the fundamental problem here is that for whatever reason a programmer wants to modify just one method of an already loaded Python class (which came from a textual module which was loaded already), save the change somewhere so it can be reloaded later (overwriting part of the textual module?), and also have the program start using the new behavior for existing instances without any other side effects arising from recompiling this one change. In practice, this is trivial to do in almost any Smalltalk system; it is hard if not impossible to do in any widely used Python IDE or program (even when a Python shell is embedded).
# xreload.py.
"""Alternative to reload().
This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. """
import imp import sys import types
def xreload(mod): """Reload a module in place, updating classes, methods and functions.
Args: mod: a module object
Returns: The (updated) input object itself. """ # Get the module name, e.g. 'foo.bar.whatever' modname = mod.__name__ # Get the module namespace (dict) early; this is part of the type check modns = mod.__dict__ # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever' i = modname.rfind(".") if i >= 0: pkgname, modname = modname[:i], modname[i+1:] else: pkgname = None # Compute the search path if pkgname: # We're not reloading the package, only the module in it pkg = sys.modules[pkgname] path = pkg.__path__ # Search inside the package else: # Search the top-level module path pkg = None path = None # Make find_module() uses the default search path # Find the module; may raise ImportError (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path) # Turn it into a code object try: # Is it Python source code or byte code read from a file? # XXX Could handle frozen modules, zip-import modules if kind not in (imp.PY_COMPILED, imp.PY_SOURCE): # Fall back to built-in reload() return reload(mod) if kind == imp.PY_SOURCE: source = stream.read() code = compile(source, filename, "exec") else: code = marshal.load(stream) finally: if stream: stream.close() # Execute the code im a temporary namespace; if this fails, no changes tmpns = {} exec(code, tmpns) # Now we get to the hard part oldnames = set(modns) newnames = set(tmpns) # Add newly introduced names for name in newnames - oldnames: modns[name] = tmpns[name] # Delete names that are no longer current for name in oldnames - newnames - set(["__name__"]): del modns[name] # Now update the rest in place for name in oldnames & newnames: modns[name] = _update(modns[name], tmpns[name]) # Done! return mod
def _update(oldobj, newobj): """Update oldobj, if possible in place, with newobj.
If oldobj is immutable, this simply returns newobj.
Args: oldobj: the object to be updated newobj: the object used as the source for the update
Returns: either oldobj, updated in place, or newobj. """ if type(oldobj) is not type(newobj): # Cop-out: if the type changed, give up return newobj if hasattr(newobj, "__reload_update__"): # Provide a hook for updating return newobj.__reload_update__(oldobj) if isinstance(newobj, types.ClassType): return _update_class(oldobj, newobj) if isinstance(newobj, types.FunctionType): return _update_function(oldobj, newobj) if isinstance(newobj, types.MethodType): return _update_method(oldobj, newobj) # XXX Support class methods, static methods, other decorators # Not something we recognize, just give up return newobj
def _update_function(oldfunc, newfunc): """Update a function object.""" oldfunc.__doc__ = newfunc.__doc__ oldfunc.__dict__.update(newfunc.__dict__) oldfunc.func_code = newfunc.func_code oldfunc.func_defaults = newfunc.func_defaults # XXX What else? return oldfunc
def _update_method(oldmeth, newmeth): """Update a method object.""" # XXX What if im_func is not a function? _update_function(oldmeth.im_func, newmeth.im_func) return oldmeth
def _update_class(oldclass, newclass): """Update a class object.""" # XXX What about __slots__? olddict = oldclass.__dict__ newdict = newclass.__dict__ oldnames = set(olddict) newnames = set(newdict) for name in newnames - oldnames: setattr(oldclass, name, newdict[name]) for name in oldnames - newnames: delattr(oldclass, name) for name in oldnames & newnames - set(["__dict__", "__doc__"]): setattr(oldclass, name, newdict[name]) return oldclass
_______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
In some ways, xreload's is better by including a general "Provide a hook for updating" concept. However xreload just ignores what to do with top level objects like lists or dicts -- xreload apparently throws away the updates to them.
No it just replaces the whole object. I was just punting on this; what are the required semantics?
You're right; I read this wrong. The point about required semantics for global objects when reloading their modules is that there are none. As you say, one might imagine different situations requiring different effects when you reload a module. Perhaps you could make a pragma directive, or define a reload hook like you set up, but even then the need might vary depending on how you are developing (i.e. is the point of the reload because you changed a method or because you changed that specific global data structure?). The point is that reloading an entire module forces you to think about this -- reloading just a method lets you ignore the problem.
There just is no way to handle *all* possibilities. How would you handle renamings?
Selectively in the IDE? :-)
The GUI part isn't supposed to be included. There's also the issue of what to do with dependencies; if x imports y, and y is reloaded, x probably needs to be reloaded too...
Yes -- another reason not to want to reload an entire module if it can be avoided. Although in practice, this seems a bigger problem when doing a regular reload rather than when doing an xreload type thing (since other modules which are handing onto the classes get the new behavior).
Still, elegant as it is (including the reload hook), even xreload does not handle all the important cases -- just (forgive me :-) the *easy* ones.
When Guido supplies an xrestart.py :-) python 2.5 code module that lets you restart exceptions after having modified one of the functions in the stack trace, then I will admit it is something I have never seen before in Python and be suitably impressed. Is it even possible without modifying the VM? :-)
It is impossible because we don't know whether the exception has a handler (an except block that traps it) until we've bubbled all the way up.
OK, impossible as things are now. Why not keep the stack of internal objects (exception handlers?) around somehow so that after you have bubbled up, you can present that stack of objects to the debugger? The point is not to go into the debugger on every exception. The point is that when you are debugging you can make changes -- either while stepping, or when an otherwise unhanded exception occurs. Essentially, think of it as a having default exception handler that throws you into the debugger at the point the exception is created (but before any related handling is done, like in finally clauses). I don't know the CPython internals on exception handling, so I do not know what would be easy to do. Perhaps you can not look up the stack of exception handlers to see if anyone would catch it without processing each handler's finally clause first? And even if you can, you would still need to hold that entire exception processing stack around so that if someone closes the debugger window (or entire application) without restarting the exception those finally clauses get done. It may require adding an entire level of abstraction to exception handling if it is impossible now? Also, in the case of Smalltalk, one can always add a "Self halt" that throws an exception for sure which almost certainly reaches the debugger. It is common to create a method function with just "self halt" and when that is called, then fill out the method in the debugger. Coding in the debugger is an entirely different style of software development. Sometimes it is very nice. http://www.google.com/search?hl=en&q=smalltalk+%22coding+in+the+debugger%22 Again, for small programs this may not matter, but when writing, say, a simulation which may run for hours, having to restart when you get an unhanded exception (like from a minor typo) can really harm productivity.
One impressive thing about the Python design which I liked from all this was how Python separates the notion of execution code from a pointer reference.
Can you clarify this for someone who is not familiar with Smalltalk? (Your habit of explaining every idea by comparing it to how that idea was implemented in Smalltalk doesn't help the audience understand you -- you're preaching to the choir when you do that, and the rest of the congregation is lost.)
Smalltalk has a separation too of behavior (a method) from invocation using a "selector" or name of a method by sending a message -- I was just thinking how much better this was, than, say C, where a function's actual code location is tightly tied to the pointer to the function. So you can't take a C function pointer and just assign new code to it.
That is what makes all these reloading tricks possible. And my hat goes off to Guido for having included the extra level of indirection which makes this feasible. I can hope that generality and late bindingness might also make possible restarting an exception without VM changes.
Why do you care about avoidung VM changes? The VM changes (incrementally) at each minor Python release.
Just so everyone (especially *me* :-) can use start using it right now, including in older versions of Python (like 2.4 etc.). Of course, I would have to break that bad habit I've gotten into of restarting the program every time I make a change -- and it's hard to break that habit, even when I use a supplemental tool that lets me reload Jython modules selectively -- I keep thinking -- I did not have to restart. :-) Still, even with all this, if you are making a GUI and modify the function that defines the window, you generally still need to close and open the window again. So there remain limits (unless you move to GUIs defined interactively like Morphic or PataPata). Just talking about getting most of the benefit. --Paul Fernhout
Paul D. Fernhout <pdfernhout <at> kurtz-fernhout.com> writes:
This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm...
FWIW, I still tinker with the reloading issue from time to time, in pursuit of a fully-interactive environment. My latest module is at: http://svn.brownspencer.com/pynote/trunk/Mupdaters.py. It's not documented for standalone use, but may contain some useful ideas, and I'd be happy to discuss it if anyone were interested. Incidentally, class bodies provide one of the hard problems for reloading. Ideally, for reloading, one would execute the class body in a proxy of the existing dict, as for modules, but this can't be achieved with the same exec <modulecode> in <proxyofexistingdict> approach. In order to work-around this, I wrote a pure-python compiler. With this, it becomes possible to modify the class definition bytecode to execute the class definitions in the existing class dict rather than a new one. The compiler itself is at: http://svn.brownspencer.com/pycompiler/branches/new_ast/ . However, I have not yet connected it into the reloading module. Michael
Michael- Thanks for the update. An entire Python compiler to make it work right? Wow. I don't use many class methods so I've never noticed this problem. Thanks for putting this up, and under the Python license (according to the readme). http://svn.brownspencer.com/pycompiler/branches/new_ast/readme.txt Is this in any ways related to PyPy? http://codespeak.net/pypy/dist/pypy/doc/news.html In the Jython reload case I have noticed a different problem, which is that a method with a call to super__ will fail. super__MethodName is Jython's convention for calling a superclass method, which is especially important to use if the Java method in a Java base class is "protected" as using the alternative BaseClassName.method(self, args) will not work then. It turns out that after a reload in Jython, the newly compiled methods have a slightly different signature than the old ones as I think there is a unique number added to the superclass name, causing the reference to the old super class wrapper to be invalid somehow on a second load Also, if you change the number of arguments to a function, or add a new function, the reload feature also fails. I think these problems may all be related to Jython's internal behind the scenes dispatching using some forms of name mangling which change from load to load. As I write this, I am wondering if those problems only occur when the base class is a Java class as opposed to a Jython base class, though many if not most of my base classes are Java classes (typically JPanel or JFrame), so it is a frequent problem. Nonetheless, I find the reload functionality very valuable, even with these problem of having to restart when adding a function, changing the arguments to a function. A typically way to use this is adding print statements to a function operating unexpectedly for use in debugging. Also I just avoid using super__ use in modules in various ways (which is sometimes difficult as it is the only way I know to call a Java protected method). I don't believe CPython reloading has any of those problems, since it is not doing the kind of reflection Jython does. On the JythonDev list, I submitted this issue as one that might interest an intern for the Google Summer of Code. http://www.nabble.com/Jython-and-Summer-of-Code-tf3350563.html#a9367861 When even Visual Basic or C# or Ruby can do this "edit and continue", I am worried about Python's future if it does not gain this feature. See for example: http://www.codinghorror.com/blog/archives/000507.html While I prefer Smalltalk keyword syntax for being self-documenting over function calls with position arguments, I can weight that against Python's clean syntax using indentation, and say Python syntax and Smalltalk syntax are close in usability. And even if Smalltalk will always also have an edge in edge in easily adding features like new iterators or control structures that require changes to the Python compiler, I can weigh that against Python's broad acceptability in looking a lot like C or Java, especially how easily Jython can interoperate with Java, given the similarity in syntax. BUT, when I weigh being much more productive in an environment like Smalltalk that lets you "edit and continue", Python is the clear loser there. It is only its other virtues (licensing, borad libraries, ease in doing small projects with it, and so on) which save the day there for it IMHO. But that's a little sad -- to see such a great dynamic language tied down somehow over an issue which you would think it would excel at. Now that Guido is at Google, and Google uses Python and does large projects (e.g. the recent AI announcement) http://www.oreillynet.com/onlamp/blog/2005/11/google_library_will_train_goog... which presumably require modifying long running training processes while they run if they encounter, say, a typo in a variable reference, I would hope building in these features might someday become more of a priority, nudge, nudge. :-) But really, this is the kind of deep thing in Python I would expect Guido would have to be involved in changing; it's probably not like a minor change or add-on in the system -- it might have ramifications throughout the codebase. I doubt anyone would have enough comprehension of Python internals to do as good a job as he could. Note that I think that even if it was impossible in Python to resume from an exception (without major surgery in Python's innards), being able to break on the creation of any exception in the debugger, and then one by one tell the debugger to ignore some of them based on the type of exception or the module it is created in, would probably still be OK enough if you could then edit and continue before the exception was actually raised. This weaker form of "Edit and continue" would be harder to use, but probably still worthwhile given the right IDE tools. It would be more feasible for a Summer of Code student or other contributor to do as a first step. (Generally, you only care about the exceptions that bubble up to the top, since a lot of Python code routinely handles exceptions like for not opening a file or when using exceptions to check the duck typing of something, which is why this weaker form would entail extra work). A lot of IDEs for other languages (like C++) let you control what exceptions the debugger breaks on, so this is not that unfeasible. --Paul Fernhout Michael Spencer wrote:
Paul D. Fernhout <pdfernhout <at> kurtz-fernhout.com> writes:
This xreload module Guido supplies is similar to the approach I used for Jython in my earlier link. http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... If you look at that thread, I acknowledge Michael Spencer as having supplied some of the more sophisticated reload logic which is similar to what Guido has posted here (not sure where Michael got it from of course, nor where Guido got xreload from). See Michael's first related post here: http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.htm...
FWIW, I still tinker with the reloading issue from time to time, in pursuit of a fully-interactive environment. My latest module is at: http://svn.brownspencer.com/pynote/trunk/Mupdaters.py. It's not documented for standalone use, but may contain some useful ideas, and I'd be happy to discuss it if anyone were interested.
Incidentally, class bodies provide one of the hard problems for reloading. Ideally, for reloading, one would execute the class body in a proxy of the existing dict, as for modules, but this can't be achieved with the same exec <modulecode> in <proxyofexistingdict> approach. In order to work-around this, I wrote a pure-python compiler. With this, it becomes possible to modify the class definition bytecode to execute the class definitions in the existing class dict rather than a new one. The compiler itself is at: http://svn.brownspencer.com/pycompiler/branches/new_ast/ . However, I have not yet connected it into the reloading module.
Michael
Paul D. Fernhout wrote:
Michael-
Thanks for the update. An entire Python compiler to make it work right?
No, the module I linked to is self-contained, although hardly user-friendly, I concede. I believe that custom-compilation *will* be required to tackle some corner cases, but I haven't implemented it.
Wow. I don't use many class methods so I've never noticed this problem. Thanks for putting this up, and under the Python license (according to the readme). http://svn.brownspencer.com/pycompiler/branches/new_ast/readme.txt
Yep, Python license.
Is this in any ways related to PyPy? http://codespeak.net/pypy/dist/pypy/doc/news.html
No. It's just an update/replacement for the stdlib compiler package which has several known problems, and which does not support the native Python 2.5 AST.
In the Jython reload case I have noticed a different problem, which is that a method with a call to super__ will fail. ...
Unfortunately, I don't know any details about Jython's implementation, but this doesn't surprise me.
I don't believe CPython reloading has any of those problems, since it is not doing the kind of reflection Jython does.
No, although it's pretty easy to come up with corner cases to confuse CPython reloading too.
When even Visual Basic or C# or Ruby can do this "edit and continue", I am worried about Python's future if it does not gain this feature. See for example: http://www.codinghorror.com/blog/archives/000507.html
I strongly agree. Edit and continue is intuitive for beginners, productive for experts, and anyway the standard set by competitors. ...
Note that I think that even if it was impossible in Python to resume from an exception (without major surgery in Python's innards), being able to break on the creation of any exception in the debugger, and then one by one tell the debugger to ignore some of them based on the type of exception or the module it is created in, would probably still be OK enough if you could then edit and continue before the exception was actually raised. This weaker form of "Edit and continue" would be harder to use, but probably still worthwhile given the right IDE tools.
I suspect that this requires a customized interpreter, and that pypy might be the vehicle to make this happen. Michael
Paul D. Fernhout wrote:
This "edit and continue" support is much more fine grained and easier to use than reloading Python modules, even with the variants I and others have created.
Where can I find out more about these variants and what they do? We are currently trying to enable a workflow that centers around a life environment and these issues become important for us. Are there any other life environments (besides Patapata which I have looked at) that are worth studying? Cheers, - Andreas
Andreas- Here is one post on this issue with code I made previously for Jython: "ReloaderWindow 0.2 (improvements to selective reloading)" http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.htm... This is a different (earlier) approach than for PataPata (PataPata uses prototypes and overriding __getattr__) and the linked code works with regular Python classes. The important thing to understand is that a Python VM manages a world of objects somewhat like a Smalltalk image, lumping all the objects from all the loaded modules into the same objectspace. You can just reference some of them (typically classes) by the way the current hierarchical Python namespace is set up. When you reload a module, it creates completely new class objects which just happen to have the same name as the old one and the namespace now points to those new objects (sort of like if every time you changed a method in Smalltalk the SystemDictionary Smalltalk was made to point to entirely new classes, leaving the old ones floating, which would obviously be a silly thing to do). So any existing instances of original classes point still to the old classes (which are no longer referenceable from the namespace). The various approaches try to get around this in various ways, such as in my approach above by loading a changed module into a temporary namespace and then copying the methods from it on top of the old classes. Also, things get a little more complicate than that, since Python often uses method pointers, equivalent in some way to Smalltalk blocks, to hook up GUI systems. So even if you keep the class the same and replace the methods, your running GUI window typically has bindings by reference to the old overwritten functions. So, as this approach I outlines does, you need to also keep the shell of the old functions and just overwrite what they actually do -- Kind of like if you kept CompiledMethods around and just overwrote their code. It's an awkward way to do things. Now, obviously, Smalltalk sometimes fails at updating GUIs when you tie actions to blocks, where recompiling methods produces new blocks (a big problem in older VisualWorks apps, not sure of the current status of VisualWorks or Squeak). Of course, in Smalltalk you could make the block call a method using a selector, which as least gives you a point of intervention. Similarly in Python, if you modify code which the bound function references, that newer code will be called by the old function's code as dispatch works differently depending whether you have a function pointer as opposed to an inline reference to another function in a Python function. In Python, I can get around this by wrapping any function pointer I give to a GUI callback using an object (like Java :-) where the callback then calls by name. PataPata did some of that internally, though obviously it adds extra layers of overhead to every GUI call, plus the programmer had to proactively add a wrapper, making code uglier, and so on. In Python, people often build specific hooks into their application to explicitly reload modules, including sometimes closing and reopening only one window inside a large application, to try to make the best of this. There are other attempts to do a more selective reloading like the approach I give above. Consider for example: "[Patches] [ python-Patches-1212921 ] option to allow reload to affect existing instances" http://mail.python.org/pipermail/patches/2005-September/018060.html There is some discussion at that link of the issue. And a reference there to another approach: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 From the first patch link: "Brett and I work at the same company. He made this patch because we were wasting hundreds of hours debugging reload problems or just quitting the application and restarting. I've been using this patch now for 3 months. It has saved our company hundreds of man-hours and we didn't have to change one line of existing code to make it work. If the community doesn't want to use it, I'm fine with that. I just want to make sure that you fully understand what you are giving up." I'm not sure what the disposition of that patch is. But, you can see from the responses there is a bit of a cultural problem here getting the value of the idea across. I did not look at the patch, but from the description I wonder if it still might suffer from the issue of allowing identities for functions to change (and so GUIs with actions previously linked to functions would still use the old behavior). There is also a more subtle issue for module reloading. Smalltalk is essentially a declarative language, while Python is essentially an imperative language (maybe there is a better word?). Essentially, compiling a Smalltalk class has no side effects (other than perhaps defining the class). But a Python module on loading can produce lots of side effects from defining metaclasses to affect future class loading to executing random bits of code (sort of like Forth in this sense). So, only modules without these side effects can be easily reloaded. Consider for example what if your module makes a singleton instance of something -- when you reload it, should it make a new singleton instance and leave the other one to still perhaps be used by old instances, or should the singleton instance get upgraded to the new class? Just one of many issues that happen when you are using Python's textual module reloading paradigm. Ideally, when you change a method in a text file you probably almost always want only that specific method to change with everything else being left alone. But that breaks the Python paradigm. Still, even with this problem, easier class reloading like I outlined above still gets you great value when developing new code, especially if you slightly alter how you layout your new modules with this limitation in mind. But reloading older code may present you with problems -- making maintenance using this technique harder. --Paul Fernhout Andreas Raab wrote:
Paul D. Fernhout wrote:
This "edit and continue" support is much more fine grained and easier to use than reloading Python modules, even with the variants I and others have created.
Where can I find out more about these variants and what they do? We are currently trying to enable a workflow that centers around a life environment and these issues become important for us. Are there any other life environments (besides Patapata which I have looked at) that are worth studying?
participants (8)
-
Andre Roberge -
Andreas Raab -
francois schnell -
Guido van Rossum -
kirby urner -
Michael Spencer -
Paul D. Fernhout -
Vern Ceder