Third-party packages inside a project
Hello, for our Windows-only applications, we use a directory setup (working copy) like this: /lib /a /b /c /src boot.py pkg1/ pkg2/ pkg3/ /lib/{a,b,c} are third-party standard distutils packages downloaded from Internet. We stuff them within the working copy because we don't want any external dependency besides Python itself (basically, if you install Python and do a checkout, you must have everything needed to run the application). /src contains our own application source code. boot.py is basically the entry point, and the code is contained within several sub-packages. Third-party packages are installed into /src. Basically, after compilation, you end up with something like: /src boot.py a/ [installed] b/ [installed] c/ [installed] pkg1/ pkg2/ pkg3/ If you run boot.py from /src, this works fine (and has worked fine for years). In the past few months, we realized that this layout is probably going to break in the future (moving to newer python versions and making the application cross-platform) because: 1) It doesnt't work for third-party packages which use setuptools. Setuptools always creates and installs a .pth file (to the best of my understanding), and .pth files are not process inside /src, because it's not part of PYTHONPATH. I can't find a viable fix for this: * Creating a sitecustomize.py inside /src (which does something like site.addsitedir(os.getcwd()) does not work anymore since Python 2.5. It looks like it never worked under *NIX, and was "fixed" under Windows lately. sitecustomize.py is not imported from the current directory, probably for security reason (code injection). * Pointing PYTHONPATH to /src is a PITA, since developers always move among projects and different branches of the same project (so different working copy), and asking them to remember to change PYTHONPATH everytime is simply a recipe for disaster. * Using a "Custom Installation Locations" as suggested by EasyInstall docs is not a solution, because those are "tricks" to install python packages into an user-specific directory (instead of the system directory). I need a solution for a project-specific directory. 2) In Python 2.4, we were able to directly run a script in a subpackages (eg: a test). Basically, from /src, you could run "python pkg1/test_foo.py", and that would work correctly (= absolute imports like "import pkg2" were working as expected). In Python 2.5, this does not work anymore. I was wondering if someone could suggest a fix, especially for #1: I can't believe there's no way to ask distutils/setuptools to install a project-specific package (I am aware of the chapter "Custom Installation Locations" in EasyInstall docs, but that's more about installing packages at user-level instead of system-level; it doesn't cover the project-level which I need). Sorry for the long mail (I reckon it's better to err on the side of verbosity to make sure everything is clear). -- Giovanni Bajo
At 12:45 PM 11/14/2007 +0100, Giovanni Bajo wrote:
I was wondering if someone could suggest a fix, especially for #1: I can't believe there's no way to ask distutils/setuptools to install a project-specific package (I am aware of the chapter "Custom Installation Locations" in EasyInstall docs, but that's more about installing packages at user-level instead of system-level; it doesn't cover the project-level which I need).
setup.py install --single-version-externally-managed --record=somefile ... The above incantation (along with whatever other options you want) will let you install setuptools-based packages "the old fashioned way". You will be responsible for installing the packages' dependencies in the same way, however. You'll also be responsible for any uninstallation prior to upgrading versions of those packages. Various .egg-info directories will be installed alongside the packages; you will need to leave these alone, until/unless you are uninstalling the corresponding package.
On 11/14/2007 1:19 PM, Phillip J. Eby wrote:
I was wondering if someone could suggest a fix, especially for #1: I can't believe there's no way to ask distutils/setuptools to install a project-specific package (I am aware of the chapter "Custom Installation Locations" in EasyInstall docs, but that's more about installing packages at user-level instead of system-level; it doesn't cover the project-level which I need).
setup.py install --single-version-externally-managed --record=somefile ...
The above incantation (along with whatever other options you want) will let you install setuptools-based packages "the old fashioned way". You will be responsible for installing the packages' dependencies in the same way, however. You'll also be responsible for any uninstallation prior to upgrading versions of those packages.
That's fine since it's exactly how it was working before. I'm happy there is a "backward-compatibility" option, but it's a pity there's no way to fix it in a way that still allows to fully use setuptools. -- Giovanni Bajo
At 02:37 PM 11/14/2007 +0100, Giovanni Bajo wrote:
On 11/14/2007 1:19 PM, Phillip J. Eby wrote:
I was wondering if someone could suggest a fix, especially for #1: I can't believe there's no way to ask distutils/setuptools to install a project-specific package (I am aware of the chapter "Custom Installation Locations" in EasyInstall docs, but that's more about installing packages at user-level instead of system-level; it doesn't cover the project-level which I need). setup.py install --single-version-externally-managed --record=somefile ... The above incantation (along with whatever other options you want) will let you install setuptools-based packages "the old fashioned way". You will be responsible for installing the packages' dependencies in the same way, however. You'll also be responsible for any uninstallation prior to upgrading versions of those packages.
That's fine since it's exactly how it was working before. I'm happy there is a "backward-compatibility" option, but it's a pity there's no way to fix it in a way that still allows to fully use setuptools.
Well, there is such a thing, I just assumed you didn't want it. The -m option to easy_install can install packages without a .pth file, but then to use them, your package must also be using setuptools and declare its dependencies, or explicitly use require() to add the packages to sys.path at runtime.
On 11/14/2007 3:00 PM, Phillip J. Eby wrote:
At 02:37 PM 11/14/2007 +0100, Giovanni Bajo wrote:
On 11/14/2007 1:19 PM, Phillip J. Eby wrote:
I was wondering if someone could suggest a fix, especially for #1: I can't believe there's no way to ask distutils/setuptools to install a project-specific package (I am aware of the chapter "Custom Installation Locations" in EasyInstall docs, but that's more about installing packages at user-level instead of system-level; it doesn't cover the project-level which I need). setup.py install --single-version-externally-managed --record=somefile ... The above incantation (along with whatever other options you want) will let you install setuptools-based packages "the old fashioned way". You will be responsible for installing the packages' dependencies in the same way, however. You'll also be responsible for any uninstallation prior to upgrading versions of those packages.
That's fine since it's exactly how it was working before. I'm happy there is a "backward-compatibility" option, but it's a pity there's no way to fix it in a way that still allows to fully use setuptools.
Well, there is such a thing, I just assumed you didn't want it. The -m option to easy_install can install packages without a .pth file, but then to use them, your package must also be using setuptools and declare its dependencies, or explicitly use require() to add the packages to sys.path at runtime.
You're right. --single-version-externally-managed seems like a better fit for my usecase. After all, I don't have to manage multiple versions of the same package within the same project. Thanks! -- Giovanni Bajo
participants (2)
-
Giovanni Bajo
-
Phillip J. Eby