Deploying embedded Python
![](https://secure.gravatar.com/avatar/01c302bbedf1980c8f31a90ad03783ac.jpg?s=120&d=mm&r=g)
Hi -
I'm currently looking into a few deployment issues with our embedded Python interpreter and I'm looking for any information about deploying embedded Python that people may have. Specifically, I'm looking for the following information:
How to define a useful subset of the stdlib that can serve as an initial basis for the installation but later allows upgrade to the "full" library if desirable. In other words, I'd like to deploy a small subset of the stdlib to begin with (simply because of size constraints) which may later be extended to a full stdlib if this is desirable. Has someone done this before? I'd love to have a small "Python.zip" cross-platform stdlib surrogate that just gets shipped with the product. If not, what is the right starting point for analyzing the dependencies inside the stdlib?
How to isolate the embedded interpreter from environmental effects. I have found that on occasion, the interpreter would pick up "stray" installations which can cause weird problems. Which environmental settings affect the startup of an embedded Python interpreter? How does one work around/remove those dependencies? Is there any information available about how exactly the startup works? What is being read/loaded in which order etc?
General advice about deploying embedded Python. Pointers to web sites, general experience (good or bad) etc. are all very welcome.
Thanks,
- Andreas
![](https://secure.gravatar.com/avatar/5de96c00bffb79de077d9172123cfe7a.jpg?s=120&d=mm&r=g)
Hi,
--- Andreas Raab <andreas.raab@gmx.de> wrote:
- How to define a useful subset of the stdlib that can serve as an initial basis for the installation but later allows upgrade to the "full" library if desirable. In other words, I'd like to deploy a small subset of the stdlib to begin with (simply because of size constraints) which may later be extended to a full stdlib if this is desirable. Has someone done this before? I'd love to have a small "Python.zip" cross-platform stdlib surrogate that just gets shipped with the product. If not, what is the right starting point for analyzing the dependencies inside the stdlib?
We do that for windows version of Blender ( http://www.blender.org/ ) (other OSes are assumed to have a full python distro already).
What we did to select the different modules included was very arbitrary. We picked the minimal set of modules that was required by scripts we wanted to ship with Blender at this point. With passing releases, we added some more, trying to keep size low (one of the reasons we don't include the xml parsing modules, IIRC).
Currently, this includes os (and its dependencies), random, webbrowser, threading, struct, tokenize, ...
I really wouldn't recommend the method we used to get dependency, which was (if memory serves right) a mix of looking at the sources and trial/error on a test machine, adding missing dependencies until it worked.
- How to isolate the embedded interpreter from environmental effects. I have found that on occasion, the interpreter would pick up "stray" installations which can cause weird problems. Which environmental settings affect the startup of an embedded Python interpreter? How does one work around/remove those dependencies? Is there any information available about how exactly the startup works? What is being read/loaded in which order etc?
That's an excellent question. I'm not sure if we came up with a viable solution for that problem either, so if anybody else would like to pitch in an answer, we'd appreciate too.
This particular problem causes much pain when people have binary modules from older versions of Python in their import paths.
- General advice about deploying embedded Python. Pointers to web sites, general experience (good or bad) etc. are all very welcome.
From experience, unless you want to limit python input to things included with the distribution of your software, I'd say limit the included modules to the bare minimum, otherwise it's easy to get to a point where you're pretty much including the whole Python distro anyway, so might as well ask your users to install that directly.
I hope that was a bit helpful.
Martin PS: Apologies to Andreas for the double post, I sent my original email to you directly instead of the list, as I intended.
____________________________________________________________________________________
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
![](https://secure.gravatar.com/avatar/8e85924cae8c57651b5eba84302edb29.jpg?s=120&d=mm&r=g)
Andreas Raab wrote:
- How to define a useful subset of the stdlib that can serve as an initial basis for the installation but later allows upgrade to the "full" library if desirable.
There is no formal way of doing this, although you could at least start with the full library and remove the really exotic stuff, like that for other platforms and audio/video/crypto packages. One man's worthless package is another's absolute requirement though.
- How to isolate the embedded interpreter from environmental effects. I have found that on occasion, the interpreter would pick up "stray" installations which can cause weird problems. Which environmental settings affect the startup of an embedded Python interpreter? How does one work around/remove those dependencies? Is there any information available about how exactly the startup works? What is being read/loaded in which order etc?
If you are picking up stray installations it is probably through the PYTHONPATH environment variable. For a brief understanding of these, run the command "python --help". For your embedded world, you ought to change your distributed code to use a different environment variable, maybe a prefix or suffix.
A useful tool in understanding the startup sequence of Python is the "-v" option, which will display the various imports that occur. It stays in effect, so it is also useful to watch the secondary imports that occur as your program executes.
There have been several efforts in the Python community to isolate specific interpreter instances from each other, although not in the sense of embedded work. Two different approaches have been taken, that of SetupTools (the base of Python Eggs) and that of zc.buildout which is a way of bringing together specific versions of packages into a runtime environment. The VirtualEnv program for Python is also good for ideas on how to do this, and is really cool too.
The SetupTools approach is to sprinkle .pth files in the lib/python/site-packages/ directory, thereby enabling specific packages globally across an installation.
The zc.buildout approach is less global but gives more rigorous control, by letting individual scripts insert specific package references during startup. So you end up with startup scripts like:
import sys
sys.path[0:0] = [ '/home/jeff/Clients/Johns/buildout/parts/zope2/lib/python', '/home/jeff/Clients/Johns/buildout/parts/thirdparty-checkouts', ]
_interactive = True if len(sys.argv) > 1: import getopt _options, _args = getopt.getopt(sys.argv[1:], 'ic:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec _val
if _args:
sys.argv[:] = _args
execfile(sys.argv[0])
if _interactive: import code code.interact(banner="", local=globals())
that customize the environment and then run the interpreter on your choice of Python programs. Note that zc.buildout leaves the existing search path intact, so that you still have the problem of things leaking in from PYTHONPATH though.
Hope this gives you some ideas,
-Jeff
![](https://secure.gravatar.com/avatar/08fb585b8f06c4c6529a8866a1921d86.jpg?s=120&d=mm&r=g)
On Fri, Dec 21, 2007 at 01:30:35AM -0600, Jeff Rush wrote:
- How to isolate the embedded interpreter from environmental effects. I have found that on occasion, the interpreter would pick up "stray" installations which can cause weird problems. Which environmental settings affect the startup of an embedded Python interpreter? How does one work around/remove those dependencies? Is there any information available about how exactly the startup works? What is being read/loaded in which order etc?
If you are picking up stray installations it is probably through the PYTHONPATH environment variable. For a brief understanding of these, run the command "python --help". For your embedded world, you ought to change your distributed code to use a different environment variable, maybe a prefix or suffix.
You can also set the Py_IgnoreEnvironmentFlag to 0 before calling Py_Initialize(). This is the equivalent of running python with the -E command line option:
-E : ignore environment variables (such as PYTHONPATH)
-- Jon Parise (jon of csh.rit.edu) :: "Scientia potentia est"
participants (4)
-
Andreas Raab
-
Jeff Rush
-
Jon Parise
-
Martin Poirier