[Python-ideas] Python probe: execute code in isolation (subinterpreter?) and get results

[M.-A. Lemburg]

anatoly techtonik wrote:
> Hi,
> 
> Is there a standard way to execute a Python code and inspect the
> results without spawning an external Python process? If there is no
> such way, I'd like to propose the feature, and there are two user
> stories. Both are about application environment probing.
> 
> 
> Story #1: Choosing the best library in a safe manner
> 
> Probing environment is required for Python applications to make
> component selection logic explicit and less error-prone. I can tell
> from my experience with Spyder IDE that startup procedure is the most
> fragile part for this cross-platform application, which makes use of
> optionally installed components on user system. Implicit import nature
> and inability to revert import operation makes situation complicated.
> Below is an example. Take a note that this is not about packaging.
> 
> Spyder IDE is a Qt application that optionally embeds IPython console.
> Qt has two bindings - PyQt4 and PySide. PyQt4 binding has two APIs -
> #1 and #2. If PyQt4 is used and version of installed IPython >= 0.11,
> the API #2 must be chosen. So, the IPython version probing should come
> first. A standard way to detect IPython version is to import IPython
> before the rest of the application, but IPython may detect PyQt4
> itself and import it too for probing version. And if Spyder uses
> PySide we now have a conflict with Qt libraries loaded. If there was a
> way to execute Python script in subinterpreter to probe all installed
> component versions and return results, the selection logic would be
> much more readable and sane.

Given that you are also loading external shared libraries, I don't
see how you could do this within the same process. Unloading shared
libs is possible (even if fragile), but if you don't even know
which libs have been loaded, likely impossible to do in a cross-
platform way.

> Story #2: Get settings from user script
> 
> Blender uses SCons to automate builds. SCons script is written in
> Python and it uses execfile(filename, globals, locals) to read
> platform specific user script with settings. Unfortunately, execfile()
> is a hack that doesn't treat Python scripts the same way the
> interpreter treats them - for example, globals access will throw
> exceptions - http://bugs.python.org/issue14049  More important that
> users won't be able to troubleshoot the exceptions, because standalone
> script works as expected.

You're not using execfile() correctly: if you want a script to be
run in the same way as a module, then the local and global namespace
dictionaries have to be the same. So the second story already works
in vanilla Python :-)

Lots of Python applications read configuration data from user
supplied (Python) config files. It's less secure than e.g. INI
files, but gives you a lot of flexibility in defining what
you need.

> Executing user script code in a subprocess will most likely negatively
> affect performance, which is rather critical for a build tool.
> Pickling and unpickling objects with global state through
> communication pipe may also be the source of bugs. So, having a cheap
> way to communicate with Python subinterpreter and get a simple dict in
> result will make Python more snappy.

I don't see how you could get story #1 working without a subprocess.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 04 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2012-04-03: Python Meeting Duesseldorf                             today

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/