On 15 Sep 2007, at 18:09 , Ian Bicking wrote:

Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http:// pypi.python.org/pypi/virtualenv/).

So the basic technique here is to copy the executable into /ENV/bin/ python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/ opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.)

Since I'm the one who's experiencing this, here's some more info: * Both the Python 2.4 from MacPorts as well as the binary MacPython package fail to create a virtualenv because sys.prefix of the copied executable still points to the old location. * Self-compiled Python 2.4 and 2.5 doesn't have this problem, nor does the Python 2.5 from MacPorts or the MacPython 2.5 binary (!!!) So this seems to be constrained to Python 2.4 from MacPorts and MacPython.

--On 15. September 2007 11:09:44 -0500 Ian Bicking <ianb@colorstudy.com> wrote:

Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/).

I tried virtualenv - however the python in the new environment contains all eggs in sys.path that are installed for the original Python interpreter. Bug or feature? Andreas -------------- jung@galactica:/tmp> /opt/python-2.4.3/bin/virtualenv envtest Creating envtest/lib/python2.4 Creating envtest/lib/python2.4/site-packages Creating envtest/bin Copying /opt/python-2.4.3/bin/python to envtest/bin Installing setuptools Processing setuptools-0.6c7-py2.4.egg creating /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg Extracting setuptools-0.6c7-py2.4.egg to /tmp/envtest/lib/python2.4/site-packages Adding setuptools 0.6c7 to easy-install.pth file Installing easy_install script to /tmp/envtest/bin Installing easy_install-2.4 script to /tmp/envtest/bin Installed /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg Processing dependencies for setuptools==0.6c7 Finished processing dependencies for setuptools==0.6c7 Processing setuptools-0.6c7-py2.4.egg removing '/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg' (and everything under it) creating /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg Extracting setuptools-0.6c7-py2.4.egg to /tmp/envtest/lib/python2.4/site-packages setuptools 0.6c7 is already the active version in easy-install.pth Installing easy_install script to /tmp/envtest/bin Installing easy_install-2.4 script to /tmp/envtest/bin Installed /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg Processing dependencies for setuptools==0.6c7 Finished processing dependencies for setuptools==0.6c7 ajung@galactica:/tmp> cd envtest/ ajung@galactica:/tmp/envtest> bin/python -c "import sys; print sys.path" ['', '/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/psycopg2-2.0.5.1-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Slimp-0.3-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/chardet-1.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/zc.catalog-1.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/textindexng-1.0_trunk-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/epydoc-3.0beta1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pytz-2007c-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/vobject-0.4.8-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/AOPython-1.0.3-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/MyghtyUtils-0.52-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/docutils-0.4-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Elixir-0.3.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Cheesecake-0.6.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/migrate-0.2.2-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/enfold.xapian-0.9.14-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/lemur.xapian-0.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/simplejson-1.7.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/setuptools-0.6c6-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/zc.buildout-1.0.0b25-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/MySQL_python-1.2.2-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/igraph-0.4.2-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pyflakes-0.2.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pylint-0.13.2-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pudge-0.1.3-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/python_dateutil-1.2-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Mako-0.1.8-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Paste-1.4-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/PasteDeploy-1.3.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/PasteScript-1.3.5-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/i18ndude-3.0b2-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/workingenv.py-0.6.5-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Beaker-0.7.4-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/grok-0.9dev_r77767-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/martian-0.8-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/ZConfig-2.3.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/bebop.recipe.svncheckout-0.0.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/TextIndexNGExtensions-4.0.0-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/python_Levenshtein-0.10-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/grokproject-0.5.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/yolk-0.3.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/SQLAlchemy-0.3.10-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pyparsing-1.4.7-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/kid-0.9.6-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/pysqlite-2.3.5-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/zopeproject-0.3.2-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/ctypes-1.0.2-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/buildbot-0.7.5-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/BeautifulSoup-3.0.4-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/z3c.sqlalchemy-1.0.11-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/feedparser-4.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/zopyx.textindexng3-4.0.0-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/zopyx.slimp-0.2.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/haufe.testrunner-0.2.1-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/coverage-2.77-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/genesis.wsl-0.2dev_r48613-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/antlr_python_runtime-3.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/ZopeSkel-0.8-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/Cheetah-1.0-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/haufe.python_repos_handler-0.1.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/PythonEggTools-1.0a0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/haufe.selenium-0.1.0-py2.4.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/python_ldap-2.3.1-py2.4-linux-x86_64.egg', '/opt/python-2.4.3/lib/python2.4/site-packages/virtualenv-0.8.1-py2.4.egg', '/tmp/envtest/lib/python24.zip', '/opt/python-2.4.3/lib/python2.4', '/tmp/envtest/lib/python2.4', '/tmp/envtest/lib/python2.4/plat-linux2', '/tmp/envtest/lib/python2.4/lib-tk', '/opt/python-2.4.3/lib/python2.4/lib-dynload', '/tmp/envtest/lib/python2.4/site-packages', '/opt/python-2.4.3/lib/python2.4/site-packages', '/opt/python-2.4.3/lib/python2.4/site-packages/PIL']

On 15 Sep, 2007, at 18:09, Ian Bicking wrote:

Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/).

So the basic technique here is to copy the executable into /ENV/bin/python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.)

All framework builds behave as you describe, Modules/getpath.c special- cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where the executable is).

Does anyone know a way to get the normal effect on this Mac Python?

You could copy the framework into the virtual python environment and rewrite linker commands in the python command-line executable to use that version of the framework. Ronald

-- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 16 Sep, 2007, at 21:44, Ian Bicking wrote:

Ronald Oussoren wrote:

...

On 15 Sep, 2007, at 18:09, Ian Bicking wrote:

...

Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/).

So the basic technique here is to copy the executable into /ENV/bin/python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.) All framework builds behave as you describe, Modules/getpath.c special-cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where the executable is).

Is there any way to effect that calculation? I.e., in a normal build that calculation is based on the location of the executable, so virtualenv moves the executable to effect that.

Move the framework.

...

...

Does anyone know a way to get the normal effect on this Mac Python? You could copy the framework into the virtual python environment and rewrite linker commands in the python command-line executable to use that version of the framework.

That sounds like the similar situation to the standard library on other posix systems, which I've avoided copying. To do that, I put a text file in that points to the original stdlib location, and in site.py I add that in (I only copy the portions of the stdlib that are needed when site.py is loaded -- os, re, etc). So the same thing should be possible with the framework, and that's not too much of a problem.

I don't have any idea how to figure out the linker commands to change the Python command-line executable...? I don't even have the vocabulary to begin figuring that out, I'm afraid. Can you give an example?

macholib (which is on PyPI) can do that. I'm not aware of a command- line tool that can do the same thing.

-- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

On 16 Sep, 2007, at 21:55, Ian Bicking wrote:

Ronald Oussoren wrote:

...

On 16 Sep, 2007, at 21:44, Ian Bicking wrote:

...

Ronald Oussoren wrote:

...

...

Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/).

So the basic technique here is to copy the executable into /ENV/bin/python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.) All framework builds behave as you describe, Modules/getpath.c special-cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where

On 15 Sep, 2007, at 18:09, Ian Bicking wrote: the executable is).

Is there any way to effect that calculation? I.e., in a normal build that calculation is based on the location of the executable, so virtualenv moves the executable to effect that. Move the framework.

I don't really know what that means...? What exactly is the framework?

The python framework, that is /Library/Frameworks/Python.framework (or /System/... if you use Apple's build of Python). getpath.c uses some API calls to determine the absolute path of the python framework linked into the current executable and sets sys.prefix to a directory inside that framework. As background info for anyone that's not used to the mac way: OSX supports the usual unix organisation of shared libraries but also has a different method: frameworks. A framework is basicly a directory containing the shared library, header files and resources (the last two are optional) and also supports versioning, although Apple's development tools don't offer full support for that. I should be possible to coax a framework install into support virtual- python by creating a directory structure for a python.framework inside the virtual-python environment and using a simular mechanism as you have now for adding the real stdlib to sys.path. You will have to modify the copied python executable to load this mini-framework because the OSX linker adds absolute paths to shared libraries and frameworks to the executable. The macholib library can be used to do this last task, it is used by py2app for rewriting the linker commands in shared libraries that are used in application bundles. I don't have an example for that handy, but it should be easy enough to extract code from macho_standalone. Ronald

-- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

On 19 Sep, 2007, at 6:25, Ian Bicking wrote:

Ronald Oussoren wrote:

...

...

Ronald Oussoren wrote:

...

On 16 Sep, 2007, at 21:44, Ian Bicking wrote:

...

Ronald Oussoren wrote:

...

On 15 Sep, 2007, at 18:09, Ian Bicking wrote: > Hi all. I'm kind of giving up on workingenv, and have started > working > from virtual-python as a basis instead > (http://pypi.python.org/pypi/virtualenv/). > > So the basic technique here is to copy the executable into > /ENV/bin/python, and then sys.prefix will be '/ENV'. The > standard > Python installed on a Mac doesn't seem to do this -- the > prefix remains > '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' > regardless. (Custom built Python's on Mac work like normal.) All framework builds behave as you describe, Modules/getpath.c special-cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where the executable is).

Is there any way to effect that calculation? I.e., in a normal build that calculation is based on the location of the executable, so virtualenv moves the executable to effect that. Move the framework.

I don't really know what that means...? What exactly is the framework? The python framework, that is /Library/Frameworks/Python.framework (or /System/... if you use Apple's build of Python). getpath.c uses some API calls to determine the absolute path of the python

On 16 Sep, 2007, at 21:55, Ian Bicking wrote: framework linked into the current executable and sets sys.prefix to a directory inside that framework.

Do you have a reference to the getpath.c that it uses? Windows seems to have something kind of hardcoded, but also a detection scheme, and maybe similarly on Mac there's something I can do to avoid the hardcoded portion.

It is in the python.org source tree: Modules/getpath.c

...

As background info for anyone that's not used to the mac way: OSX supports the usual unix organisation of shared libraries but also has a different method: frameworks. A framework is basicly a directory containing the shared library, header files and resources (the last two are optional) and also supports versioning, although Apple's development tools don't offer full support for that. I should be possible to coax a framework install into support virtual-python by creating a directory structure for a python.framework inside the virtual-python environment and using a simular mechanism as you have now for adding the real stdlib to sys.path. You will have to modify the copied python executable to load this mini-framework because the OSX linker adds absolute paths to shared libraries and frameworks to the executable. The macholib library can be used to do this last task, it is used by py2app for rewriting the linker commands in shared libraries that are used in application bundles. I don't have an example for that handy, but it should be easy enough to extract code from macho_standalone.

I noticed in p2app it has a file main.c in it, which I think is the Python interpreter code... maybe it recompiles the interpreter?

No, py2app's main.c is something completely different. It is the main executable for an application bundle and is very much tuned for that. Ronald

On Wednesday, September 19, 2007, at 07:26AM, "Ian Bicking" <ianb@colorstudy.com> wrote:

...

It is in the python.org source tree: Modules/getpath.c

That's the file used in the Framework build of Python? I only see some small references to __APPLE__, none of which seem related to Frameworks. Also, if you build Python from source it works fine -- it's only the python that Apple ships that has the problem. They must use some patched version of this file that they use...?

Start reading at line 694 in Modules/getpath.c, that's in the head of the release25-maint branch. I'm pretty sure that you get the same problem when you build from source and give the --enable-framework option to configure. Python.framework is not an Apple invention. Ronald

On 19 Sep, 2007, at 19:23, Ian Bicking wrote:

Ronald Oussoren wrote:

...

On Wednesday, September 19, 2007, at 07:26AM, "Ian Bicking" <ianb@colorstudy.com

...

wrote:

...

It is in the python.org source tree: Modules/getpath.c That's the file used in the Framework build of Python? I only see some small references to __APPLE__, none of which seem related to Frameworks. Also, if you build Python from source it works fine -- it's only the python that Apple ships that has the problem. They must use some patched version of this file that they use...? Start reading at line 694 in Modules/getpath.c, that's in the head of the release25-maint branch.

It's only 695 lines long...?

Oops, I've only been using vi for a decade or so... Luckily you found the right bit anyway ;-)

At line 458 it seems like it is looking for lib/python2.5/os.py, in some Framework-specific code. It will look relative to the binary, but only after looking in the Framework first. Simply breaking the Framework lookup somehow would do it.

Here's the code to find the Framework locations:

pythonModule = NSModuleForSymbol(NSLookupAndBindSymbol("_Py_Initialize")); /* Use dylib functions to find out where the framework was loaded from */ buf = (char *)NSLibraryNameForModule(pythonModule);

I'm not sure how to do anything to that though.

I don't think you can. What you can do create the directory structure for a framework inside the virtual python environment, including a copy of the actual shared library, then copy a minimal subset of the stdlibrary into that (anything you need for your custom site.py) and have that site.py build the correct sys.path. You must then copy sys.executable into the virtual python environment as well and use macholib to rewrite the link command for the python framework from /Library/Frameworks/Python.framework/... to @executable_path/../Frameworks/Python.framework/.... This should ensure that the right shared library is loaded and your environment works as you want. You could also ship with the sources to the command-line interpreter and build that with the right LDFLAGS during installation of the virtual-python package. Ronald

-- Ian Bicking : ianb@colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers