I'm trying to add a lightweight "virtualenv" to a local build system using PYTHONUSERBASE. Unfortunately the package maintainers for distutils and setuptools broke it on my Ubuntu laptop. I have incontrovertable proof! PYTHONUSERBASE doesn't scan the exact directory you set it to; you're pointing it at a "prefix" directory, and it looks in directories under there. Well, *directory*, really: PYTHONUSERBASE only causes CPython to scan one additional directory for packages. That directory is: "{prefix}/lib/python2.6/site-packages". This is true whether you use the Debian packaged build or if you build Python yourself from source. AFAICT distutils uses distutils.sysconfig.get_python_lib() to decide where you should install a site package. Depending on the inputs, it will give you one of two directories. If it's a "standard library", it'll give you "{prefix}/lib/python2.6". However! If you call get_python_lib() saying it's *not* a standard library, you get different results. If you build CPython from scratch you'll get "{prefix}/lib/python2.6/site-packages". If you use the Ubuntu Python 2.6 package, you'll get "{prefix}/lib/python2.6/dist-packages". But PYTHONUSERBASE still only looks in the "site-packages" directory. It ignores this directory and therefore doesn't pick up your packages. A more obscure but more widespread bug: consider that PYTHONUSERBASE also ignores the "standard library" directory returned by get_python_lib(). If you wanted to have your own local version of a "standard library", you couldn't put it in "{prefix}/lib/python2.6". This is true whether you build CPython from scratch or use the Ubuntu packages. setuptools under Ubuntu breaks PYTHONUSERBASE in a similar but different way. If you install the setuptools egg by hand, Setuptools will install packages into "{prefix}/lib/python2.6/site-packages". If you use the Ubuntu package for setuptools, setuptools will install packages into "{prefix}/local/lib/python2.6/dist-packages". Again, this is a directory PYTHONUSERBASE ignores. In my opinion the best way to fix this would be to add a new environment variable and deprecate the old one. I nominate "PYTHONUSERBASES"--note the plural. PYTHONUSERBASES would support a local-path-separator-separated list of directories, all of which would be used as site package directories. For each directory on the path, we would add *the same list of subdirectories* that site.addsitepackages() does. Failing that, PYTHONUSERBASE should be at least be changed so it adds the same directories as site.addsitepackages(). I would be happy to contribute a patch to do either of these, for Python 2.x and 3.x. What follows is my test case where I figured all this out. I started by creating a PYTHONUSERBASE directory, which in my late-night hacking fever I called "/home/larry/pwned". Inside I created subdirectories matching every directory I'd ever seen scanned for site packages, like "lib/python2.6/site-packages". Then for each directory I added a do-nothing module named for the directory it was in, with a "UB" on the front so I knew it came from my PYTHONUSERBASE. For example, one file was called "/home/larry/pwned/local/lib/python2.6/dist-packages/UBlocallibpythondistpackages.py". Next, I added those same files to the equivalent directories in "/usr", only with "SYSTEM" on the front. For instance, "/usr/lib/python2.6/site-packages/SYSTEMlibpythonsitepackages.py" Next, I built Python 2.6 from source, with a prefix directory of "/home/larry/src/python/userbase/release", and installed setuptools from the egg on cheeseshop. I then added these same files one last time, and again with SYSTEM on the front, to my locally-built Python's prefix directory. Finally I wrote a Python script that tried to import every one of these modules, and ran it as follows: % PYTHONUSERBASE=/home/larry/pwned python findcookies.py % PYTHONUSERBASE=/home/larry/pwned ./python findcookies.py Here's the output. First, the results from running my built-from-source Python. Hopefully you're reading this in a fixed-point font on a wide screen: -- % PYTHONUSERBASE=/home/larry/pwned /home/larry/src/python/userbase/release/bin/python /home/larry/findcookies.py ------------------------------------------------------------------------------ Where does distutils say we should install? Calling distutils.sysconfig.get_python_lib() with two different prefixes (sys.prefix and "/home/larry/pwned") and all combinations of its two boolean arguments: du.sc.gpl(True , True , '/home/larry/src/python/userbase/release') = '/home/larry/src/python/userbase/release/lib/python2.6' du.sc.gpl(True , True , '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6' du.sc.gpl(True , False, '/home/larry/src/python/userbase/release') = '/home/larry/src/python/userbase/release/lib/python2.6/site-packages' du.sc.gpl(True , False, '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6/site-packages' du.sc.gpl(False, True , '/home/larry/src/python/userbase/release') = '/home/larry/src/python/userbase/release/lib/python2.6' du.sc.gpl(False, True , '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6' du.sc.gpl(False, False, '/home/larry/src/python/userbase/release') = '/home/larry/src/python/userbase/release/lib/python2.6/site-packages' du.sc.gpl(False, False, '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6/site-packages' ------------------------------------------------------------------------------ Where does setup_tools say we should install? Expanding setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"] with two different prefixes (sys.prefix and "/home/larry/pwned"): /home/larry/src/python/userbase/release/lib/python2.6/site-packages /home/larry/pwned/lib/python2.6/site-packages ------------------------------------------------------------------------------ Finally: trying to load a module from every possible site-packages directory. There are five columns in the output; here's what they mean. +----------- Marked with "*" if distutils.sysconfig.get_python_lib | told us to use this directory. | | +--------- Marked with "EZ" if setuptools.command.easy_install.easy_install.INSTALL_SCHEMES | | told us to use this directory. | | | | +------ Could we load this module? | | | | | | +--- What silly name did I give +--- What directory did I | | | | to this module? | stick this module into? | | | | | v v v v v * True SYSTEMlibpython /home/larry/src/python/userbase/release/lib/python2.6 True SYSTEMlibsitepython /home/larry/src/python/userbase/release/lib/site-python False SYSTEMlibdistpython /home/larry/src/python/userbase/release/lib/dist-python * EZ True SYSTEMlibpythonsitepackages /home/larry/src/python/userbase/release/lib/python2.6/site-packages False SYSTEMlibpythondistpackages /home/larry/src/python/userbase/release/lib/python2.6/dist-packages False SYSTEMlocallibpythondistpackages /home/larry/src/python/userbase/release/local/lib/python2.6/dist-packages * False UBlibpython /home/larry/pwned/lib/python2.6 False UBlibsitepython /home/larry/pwned/lib/site-python False UBlibdistpython /home/larry/pwned/lib/dist-python * EZ True UBlibpythonsitepackages /home/larry/pwned/lib/python2.6/site-packages False UBlibpythondistpackages /home/larry/pwned/lib/python2.6/dist-packages False UBlocallibpythondistpackages /home/larry/pwned/local/lib/python2.6/dist-packages -- Next, the output from the CPython and setuptools from the Ubuntu packages: -- % PYTHONUSERBASE=/home/larry/pwned /usr/bin/python /home/larry/findcookies.py ------------------------------------------------------------------------------ Where does distutils say we should install? Calling distutils.sysconfig.get_python_lib() with two different prefixes (sys.prefix and "/home/larry/pwned") and all combinations of its two boolean arguments: du.sc.gpl(True , True , '/usr') = '/usr/lib/python2.6' du.sc.gpl(True , True , '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6' du.sc.gpl(True , False, '/usr') = '/usr/lib/python2.6/dist-packages' du.sc.gpl(True , False, '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6/dist-packages' du.sc.gpl(False, True , '/usr') = '/usr/lib/python2.6' du.sc.gpl(False, True , '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6' du.sc.gpl(False, False, '/usr') = '/usr/lib/python2.6/dist-packages' du.sc.gpl(False, False, '/home/larry/pwned') = '/home/larry/pwned/lib/python2.6/dist-packages' ------------------------------------------------------------------------------ Where does setup_tools say we should install? Expanding setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"] with two different prefixes (sys.prefix and "/home/larry/pwned"): /usr/local/lib/python2.6/dist-packages /home/larry/pwned/local/lib/python2.6/dist-packages ------------------------------------------------------------------------------ Finally: trying to load a module from every possible site-packages directory. There are five columns in the output; here's what they mean. +----------- Marked with "*" if distutils.sysconfig.get_python_lib | told us to use this directory. | | +--------- Marked with "EZ" if setuptools.command.easy_install.easy_install.INSTALL_SCHEMES | | told us to use this directory. | | | | +------ Could we load this module? | | | | | | +--- What silly name did I give +--- What directory did I | | | | to this module? | stick this module into? | | | | | v v v v v * True SYSTEMlibpython /usr/lib/python2.6 False SYSTEMlibsitepython /usr/lib/site-python True SYSTEMlibdistpython /usr/lib/dist-python False SYSTEMlibpythonsitepackages /usr/lib/python2.6/site-packages * True SYSTEMlibpythondistpackages /usr/lib/python2.6/dist-packages EZ True SYSTEMlocallibpythondistpackages /usr/local/lib/python2.6/dist-packages * False UBlibpython /home/larry/pwned/lib/python2.6 False UBlibsitepython /home/larry/pwned/lib/site-python False UBlibdistpython /home/larry/pwned/lib/dist-python True UBlibpythonsitepackages /home/larry/pwned/lib/python2.6/site-packages * False UBlibpythondistpackages /home/larry/pwned/lib/python2.6/dist-packages EZ False UBlocallibpythondistpackages /home/larry/pwned/local/lib/python2.6/dist-packages -- For what it's worth, here's my horrible hacked-together script: -- import distutils.sysconfig import os import sys paths = {} print """ %% PYTHONUSERBASE=%s %s %s ------------------------------------------------------------------------------ Where does distutils say we should install? Calling distutils.sysconfig.get_python_lib() with two different prefixes (sys.prefix and "/home/larry/pwned") and all combinations of its two boolean arguments: """.strip() % (os.environ["PYTHONUSERBASE"], sys.executable, sys.argv[0]) print for platformSpecific in (True, False): for standardLib in (True, False): for prefix in (sys.prefix, "/home/larry/pwned"): path = distutils.sysconfig.get_python_lib(platformSpecific, standardLib, prefix) print (" du.sc.gpl(" + str(platformSpecific).ljust(5) + ", " + str(standardLib).ljust(5) + ", " + repr(prefix) + ")").ljust(69), "=", repr(path) paths[path] = "* " print print print """ ------------------------------------------------------------------------------ Where does setup_tools say we should install? Expanding setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"] with two different prefixes (sys.prefix and "/home/larry/pwned"): """.strip() print import setuptools.command.easy_install as ei for where, base in (("system", sys.prefix), ("userbase", "/home/larry/pwned")): path = ei.easy_install.INSTALL_SCHEMES[os.name]["install_dir"].replace("$base", base).replace("$py_version_short", "2.6") print " ", path if path in paths: paths[path] = "* EZ" else: paths[path] = " EZ" print print """ ------------------------------------------------------------------------------ Finally: trying to load a module from every possible site-packages directory. There are five columns in the output; here's what they mean. +----------- Marked with "*" if distutils.sysconfig.get_python_lib | told us to use this directory. | | +--------- Marked with "EZ" if setuptools.command.easy_install.easy_install.INSTALL_SCHEMES | | told us to use this directory. | | | | +------ Could we load this module? | | | | | | +--- What silly name did I give +--- What directory did I | | | | to this module? | stick this module into? | | | | | v v v v v """.strip() for prefix, pathprefix, where in (("SYSTEM", os.path.normpath(sys.prefix) + "/", "system "), ("UB", "/home/larry/pwned/", "userbase")): for _name, path in ( ("libpython", "lib/python2.6"), ("libsitepython", "lib/site-python"), ("libdistpython", "lib/dist-python"), ("libpythonsitepackages", "lib/python2.6/site-packages"), ("libpythondistpackages", "lib/python2.6/dist-packages"), # nobody ever told me to use one of these four, so I'm removing 'em. # ("locallibpython", "local/lib/python2.6"), # ("locallibsitepython", "local/lib/site-python"), # ("locallibdistpython", "local/lib/dist-python"), # ("locallibpythonsitepackages", "local/lib/python2.6/site-packages"), ("locallibpythondistpackages", "local/lib/python2.6/dist-packages"), ): name = prefix + _name fullpath = os.path.join(pathprefix, path) filename = os.path.join(fullpath, name + ".py") assert os.path.isfile(filename), "Your test is wrong, %s does not exist!" % repr(filename) s = "import {0}".format(name) try: exec s worked = True except ImportError: worked = False starred = paths.get(fullpath, " ") print " ", starred, str(worked).ljust(6), name.ljust(35), fullpath if 0: #print path if 1: name = name.replace("UB", "SYSTEM") if not os.path.isdir(path): os.makedirs(path) filename = path + "/" + name + ".py" filename = filename.replace("/python/", "/python2.6/") #print filename #continue f = open(filename, "wt") f.write('cookie = "PIXIE UNICORN"\n') f.close() print -- If you read this far, I congratulate you on your powers of concentration. Hope this helps, /larry/