Out-of-process Execution (was:Re: constructive critisism on pythonwin IDE)
Mike C. Fletcher
mcfletch at rogers.com
Thu Jun 6 09:23:41 CEST 2002
More verbosely, the "stale modules" problem is an issue related to the
"easy way" to implement IDEs for Python, that is, providing an
interpreter window. The problem is that when a Python modules changes
you need to reload each and every loaded module and figure out all sorts
of tricky corner cases with inter-module dependencies and imports.
PythonWin uses the interpreter-based approach, and punts on the reload
requirement (likely because it's a messy problem).
In an interpreter window, you generally _want_ to retain previous results
of calculations. Some issues:
You want all objects to update their class pointers to the newly loaded
classes in the new modules (requires finding every object in the
system). You similarly need each import in the system to be re-done, in
the same order they originally occured (and watching out for weird
corner cases where the same names in different packages confuse the
There is code lying around in various usenet posts to address this issue
by walking the object hierarchy and updating each encountered object.
It's pretty ugly, but it's often possible to get it working right 80%
(or whatever) of the time. The rest of the time, something or other
hasn't been updated (worst being exceptions and dynamically-generated
constants). You spend a lot of time tracking down subtle errors (the
exception is thrown, you catch it, but it doesn't get caught,
etceteras). This approach needs a considerable amount of work to make
it standard, stable and reliable.
One approach to simplifying the problems above would be to re-run the
interactive session on reload/change of a module. But interpreters are
often used for one-off calculations. You don't want to have to re-run a
30 minute calculation just to reload a particular module, but maybe you
want all the other objects to be updated save your particular objects.
If you're going to do this, you need to provide some way of marking a
particular item static across a reload boundary. Doable (store a
reference in some dictionary for later usage), ugly though. Arguably,
you could even store all the objects, load the modules "clean" and if
the user asks for a variable from the pre-reload session, scan it with
algo above before giving it to them.
But, to my mind, though the approaches above might work, they miss the
best solution, and the one you see in most (all?) of the newer IDEs for
Python, namely out-of-process execution IN ADDITION to an interpreter
(which is still used for "playing around" with what-ifs and interactive
calculation sessions). What we could really use is a robust
(BSD-licensed) library for the following:
running a python script in a seperate process (cross-platform, with
ability to specify particular executable, and command-line flags
(including -i for dropping into gui mode)) in either "standard" or
support stdout, stderr stdin suitable for using as hooks with GUI
libraries (i.e. ability to drop into an "interpreter" window when the
script completes, type responses to console-mode programmes, and
generally interact with the script as if it were being run from the
console) GUI will be able to keep this window around after execution if
it just wants to show user results from completed script.
allow debugging a client script (with inspection support (i.e. be able to
run a GUI inspector across the pipe/process barrier)
[ you could do this with a wrapped set of stdout, stderr, stdin pipes
(differentiating between system and "data" messages) or an out-of-band
client-server comm mechanism (COM, CORBA, RPC) ]
kill child process
And, add the following while you're at it ;) :
provide for "interupt-to-debug" support (press a key, move the client
process into debug mode) for debugging situations where code falls into
weird states unpredictably
"slow execution" modes for use with watch variables
watch-variables (same interface as inspector above), basically the value
is passed back on every single step of the programme being debugged.
I can't see adding this to PythonWin myself (though I use PythonWin as
my IDE and find it the most comfortable by far, I can't see spending the
time figuring out the raw win32api calls to create the various dialogs
and support you'd need in the application when I just use the console
for 98% of this kind of work), but if sameer (the original poster) feels
like putting together a robust, reliable and complete module for
out-of-process execution, he'd probably find lots of people willing to
work on it with him, and one of them might be willing to do the Win32API
coding. Such a library would also make writing new Python IDEs
Just a thought,
F. GEIGER wrote:
> This a Python issue, not a PythonWin IDE issue.
> "sameer" <sameer_ at email.com> schrieb im Newsbeitrag
> news:6bd9f01b.0206050736.f469082 at posting.google.com...
>>First of all, let me start of by saying that Pythonwin is a great IDE.
>> Now for the constructive critisism. The thing I hate about PythonWin
>>is, that despite the fact that it's a development environment, there
>>is a lot of caching done of modules, and the cached copy is not
>>checked against a potentialy modified copy. Whenever I change the
>>contents of a file and try to run a module that depends on that file,
>>I always end up with the old copy. I have to then import and then
>>reload the changed module in the interactive window for it to refresh.
>> It would greatly increase my productivity, if I can specify which
>>modules can be cached. I understand that there is a speed and
>>development time issue, but I would like a little more fine grained
>>control on how caching works on PythonWin, as the way it's currently
>>implemented now is not acceptable.
Mike C. Fletcher
More information about the Python-list