[Patches] [ python-Patches-1212921 ] option to allow reload to affect existing instances

[SourceForge.net]

Patches item #1212921, was opened at 2005-06-01 14:18
Message generated for change (Comment added) made by tczotter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1212921&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Brett Rosen (bdrosen)
Assigned to: Nobody/Anonymous (nobody)
Summary: option to allow reload to affect existing instances

Initial Comment:
One of the commonly asked questions (especially when
developing
GUI apps) is how do I reload a modified python file and
have
the existing instances automatically get the changed
behavior.
(this greatly speeds up the development cycle.)
 
Unfortunately python doesn't support this without having
to do the work yourself. To rectify this, I've implemented
smart reloading as detailed below:

Smart reloading is defined as applying modified code to
your
already instantiated objects on reload.

Supporting smart reloading for python:

Make sure that you execute python with the -r flag if
you want smart reloading to work by default. The semantics
of reload work as normal

The normal model has been expanded by adding two special
attributes to classes:

1 __pqname__ : This is used to give a hint as to our
context if we are a nested
class (top level classes don't need to use it) It
specifies the parent
hierarchy, excluding the module name. ie:

    class outer(object):
        class middle(object):
            __pqname__ = "outer"
            class inner(object):
                __pqname__ = "outer.middle"

excluding this for inner classes can lead to confusion
if you have multiple inner classes
with the same name.

2 __reloadMode__ : This can have one of the 3 values -
"overlay", "clear" and "disable"

 a) clear is the default if you are in reload mode (-r)
and you don't specify anything.
    The behavior here is to clear the class dictionary
before doing the reload of the class.
 b) overlay works like clear except that it doesn't
clear the class dictionary first.
 c) disable gives you the classic python behavior and
is the default if you are not in
    reload mode (no -r specified)
 

When using this you almost never want to reuse class
names in a module file. 
For example:

class foo(object):
...


class foo(object):
...

would be bad. The second class foo replaces the first
one, although
with overlay it could be used as syntactic sugar for
adding new methods
to an existing class.






----------------------------------------------------------------------

Comment By: Ted Czotter (tczotter)
Date: 2005-09-02 17:35

Message:
Logged In: YES 
user_id=1339137

Ahh - finally someone is interested in this one.

Michael's recipe does address *some* of the same problems 
solved by this patch, but not all.

Before I get into more detail, let's be clear about something. 
The reload wart is a
problem of inconvenience, not the inability to do something. 
For ordinary development, when
I make a change in one module, I save time and avoid 
mistakes if I don't have to:
		* reload other modules
		* restart all or part of my application 
and thereby have to recover my state
Neither of these stop me from developing, they just slow me 
down.

So a good solution is going to be measured on three things:
	1) How well it improves my efficiency as a 
developer.
	2) The level of effort required to use the solution.


Ok.

So the core of the wart is that when you re-compile a class, 
a *new* class is created,
instead of altering the definitions bound to the existing class. 
The symptoms are:
* Existing instances (like open dialogs or sockets) continue 
to use the old class
and don't inherit new behavior.
* Existing subclasses refer to the old base class and any 
instances created in the future
are still inheriting old behavior.

Michael's recipe addresses these symptoms.

Are there other important symptoms? Unfortunately, yes.
* OTHER REFERENCES TO THE OLD CLASS EXIST
Where? All over the place. Consider this:
	
from mycompany.corporateframework.gui.widgets import 
Button, CheckBox
# This *module* now points to the old Button and CheckBox.
# Many calls to Button() and CheckBox() follow in methods 
after this.

The recipe can't find and fix these references. If I was using 
the recipe, I'd have to
reload all these modules. But then I compound the problem. 
Reloading these other modules
creates *still more new classes*, causing dozens of *other* 
modules to need reloading. In the
end, I'm better off just restarting the whole darn application.

I understand that I could take advantage of the fact that 
modules retain their identity and
so if I use fully qualified names everywhere, I won't have this 
problem because I'd always
be looking up the latest class. But realistically, who is going 
to type
	b = 
mycompany.corporateframework.gui.widgets.Button("OK")
all the time instead of just:
	b = Button("OK")
Making people use python "a certain way" that is not 
idiomatic (i.e. automatic, normal,
how we do things every day) is a problem and lowers the 
usefulness of the solution. And I'm
certainly not about to suggest that anybody go through our 
30,000 lines of python code to
make such idiomatic changes so the recipe can work.

Anyway, even if I did accept that tactic (and I don't) I'm still 
not out of the woods.

Consider warehouses and factories. Here you are storing 
pointers to classes in containers
for later invocations. For example, In wx, the grid class 
requires that you register classes
to be used later for in-place editing in a grid. All these 
registrations are pointing to 
old class definitions. To fix the problem, we have to re-
register. Which usually means
reloading some modules. Which means reloading some more 
modules - you get it by now.

So as a result, my first measures of success are not met 
very well by the recipe. 
Yes, I can type foo.DoLatestThing() at the console, but in 
real life I'm still 
going to have to restart my application a lot.

On the other hand, the patch addresses the *cause* of the 
problem instead of some 
of the symptoms.

****** The true cause is that re-compiled classes change 
their identity. ******

With this patch, they do not. They retain their identity just 
like modules do.

All the symptoms go away.

Now let's talk about the harm the recipe does. What harm 
you ask?

Well to make it work you have to make all your most-base 
classes use Michael's metaclass.
Unfortunately, we already use metaclasses for a lot of things. 
And we use multiple inheritance
a lot (we like mixins). How fast can you say "metaclass 
conflict"?

I like what metaclasses do and I'm happy that Python let's 
me extend "internal" behavior
so neatly. But the metaclass conflict is another serious wart. 
If we can avoid it, good.

And I'm not happy that I seriously have to edit a few hundred 
class definitions in existing
code to get the benefit of the recipe. 
**** The patch approach requires ZERO changes to existing 
applications. Z E R O ****

Brett and I work at the same company. He made this patch 
because we were wasting hundreds
of hours debugging reload problems or just quitting the 
application and restarting. I've
been using this patch now for 3 months. It has saved our 
company hundreds of man-hours
and we didn't have to change one line of existing code to 
make it work.

If the community doesn't want to use it, I'm fine with that. I 
just want to make
sure that you fully understand what you are giving up.

Ted Czotter
CTO
Lavastorm Technologies


----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2005-09-01 05:38

Message:
Logged In: YES 
user_id=6656

My first thought is to plug my recipe:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

In fact, this seems pretty similar to what the patch does, on a quick skim.

Given that the bulk of the functionality can be implemented in Python, I'm 
not sure *I* support the further complication of type_new for this.

----------------------------------------------------------------------

Comment By: Reinhold Birkenfeld (birkenfeld)
Date: 2005-08-31 18:29

Message:
Logged In: YES 
user_id=1188172

Does anyone care to give a quick, obliterative judgement
about this?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1212921&group_id=5470