Using setuptools entry points at Google (and other places...)
Disclaimer: I don't work at Google, just talk to people who do. I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow. Is there a script that could be run manually before applications start that require entry points? Or some way setuptools itself can have some intelligence in it so that it can detect when its not in charge of the layout and read a manually configured file for entry point information instead? Thanks, Ben
At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote:
Disclaimer: I don't work at Google, just talk to people who do.
I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow.
Just make sure that packages' .egg-info directory is installed alongside the code; setuptools will do the rest. See also: http://peak.telecommunity.com/DevCenter/EggFormats#eggs-and-their-formats """The .egg-info format, on the other hand, was created to support backward-compatibility, performance, and ease of installation for system packaging tools that expect to install all projects' code and resources to a single directory (e.g. site-packages). Placing the metadata in that same directory simplifies the installation process, since it isn't necessary to create .pth files or otherwise modify sys.path to include each installed egg."""
Phillip J. Eby wrote:
At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote:
Disclaimer: I don't work at Google, just talk to people who do.
I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow.
Just make sure that packages' .egg-info directory is installed alongside the code; setuptools will do the rest. See also:
Thanks Phillip. What's not clear to me is if there is also some versioning mechanism. There seems to be two options for naming your egg-info. Let's take the package Mako version 0.1.8, as it has a short name. If I do setup.py install with the --root option, it will install the package Mako into the mako directory in my specified location and along side, it will create a Mako-0.1.8-py2.4.egg-info directory. I'm assuming I can have that directory simply named mako.egg-info and remove the version info from the filename. Now, in Mako-0.1.8-py2.4.egg-info/top_level.txt is the line 'mako'. Is this some sort of pointer to the package? from the peak website: "....top_level.txt ... lists all top-level modules and packages in the distribution. This is used by the easy_install command to find possibly-conflicting "unmanaged" packages when installing the distribution." Does this mean that I have a rough version control system where I could have, in addition to this plain 'mako' folder that held version 0.1.8, I could have a mako-0.1.7 distribution directory with a Mako-0.1.7-py2.4.egg-info directory that had the pointer to mako-0.1.7 in top_level.txt and setuptools would use this info to find the right version of mako? In this case, if you did 'import mako' in python you'd get mako-0.1.8 but if you used setuptools to find the package you could pick from 0.1.8 or 0.1.7? thanks. davep -- View this message in context: http://www.nabble.com/Using-setuptools-entry-points-at-Google-%28and-other-p... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
At 11:24 PM 6/25/2007 -0700, primco wrote:
Phillip J. Eby wrote:
At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote:
Disclaimer: I don't work at Google, just talk to people who do.
I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow.
Just make sure that packages' .egg-info directory is installed alongside the code; setuptools will do the rest. See also:
Thanks Phillip. What's not clear to me is if there is also some versioning mechanism. There seems to be two options for naming your egg-info. Let's take the package Mako version 0.1.8, as it has a short name.
If I do setup.py install with the --root option, it will install the package Mako into the mako directory in my specified location and along side, it will create a Mako-0.1.8-py2.4.egg-info directory.
I'm assuming I can have that directory simply named mako.egg-info and remove the version info from the filename.
That's correct, but for performance reasons it is NOT recommended. Without the other information in the directory name, pkg_resources will have to read the PKG-INFO file at runtime to ascertain the package version, and this is then a lot slower than just doing a single listdir to get version info for all the projects in that directory. I therefore strongly recommend leaving setuptools' naming alone for installed packages. (For packages in development, the performance tradeoff is usually acceptable.)
Now, in Mako-0.1.8-py2.4.egg-info/top_level.txt is the line 'mako'. Is this some sort of pointer to the package?
from the peak website: "....top_level.txt ... lists all top-level modules and packages in the distribution. This is used by the easy_install command to find possibly-conflicting "unmanaged" packages when installing the distribution."
Does this mean that I have a rough version control system where I could have, in addition to this plain 'mako' folder that held version 0.1.8, I could have a mako-0.1.7 distribution directory with a Mako-0.1.7-py2.4.egg-info directory that had the pointer to mako-0.1.7 in top_level.txt and setuptools would use this info to find the right version of mako?
In this case, if you did 'import mako' in python you'd get mako-0.1.8 but if you used setuptools to find the package you could pick from 0.1.8 or 0.1.7?
No. top_level.txt has only ever been used to check for conflicts with non-egg packages, and it's not even used for that any more. It doesn't magically make multi-versioning possible; for multi-versioning in a single sys.path directory you have to use .egg format (either the zipfile or directory versions). But if you use the .egg format then you can indeed get your pick by using pkg_resources to request your desired version(s). The .egg-info format trades off multi-versioning and some package management features in order to provide backward compatibility and/or startup performance, which is why it's called "single version, externally managed". You need some sort of package manager to manage it, and you don't get multiple versions.
Phillip J. Eby wrote:
At 11:24 PM 6/25/2007 -0700, primco wrote:
Phillip J. Eby wrote:
At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote:
Disclaimer: I don't work at Google, just talk to people who do.
I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow.
Just make sure that packages' .egg-info directory is installed alongside the code; setuptools will do the rest. See also:
Thanks for the answers. That's what my messing around told me but I wanted to check. On a related note, and to finish of the dismantling of setuptools package management, how do I give the setuptools package itself this treatment? Some of the things I'd like to do: not use any .pth files for any packages not change site.py or sys.path (PYTHONPATH is ok) at runtime Avoid the included site.py that is generated by installing setuptools itself as single version externally managed. It has a "__boot()" function that does some path management. What I've done is "setup.py install --root..." setuptools from the source. This made a site.py for me which I just left out of site-packages and it all seems to work. I was able to run Pylons (a pretty good setuptools workout I think) with all dependencies installed in the same unmanaged manner. Did I just get lucky or should I assume that I've safely disabled the package management features of setuptools while still keeping some dynamic things like entrypoints and find_packages working? davep -- View this message in context: http://www.nabble.com/Using-setuptools-entry-points-at-Google-%28and-other-p... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
At 10:06 PM 6/26/2007 -0700, primco wrote:
On a related note, and to finish of the dismantling of setuptools package management, how do I give the setuptools package itself this treatment?
Some of the things I'd like to do: not use any .pth files for any packages not change site.py or sys.path (PYTHONPATH is ok) at runtime Avoid the included site.py that is generated by installing setuptools itself as single version externally managed. It has a "__boot()" function that does some path management.
What I've done is "setup.py install --root..." setuptools from the source. This made a site.py for me which I just left out of site-packages and it all seems to work. I was able to run Pylons (a pretty good setuptools workout I think) with all dependencies installed in the same unmanaged manner. Did I just get lucky or should I assume that I've safely disabled the package management features of setuptools while still keeping some dynamic things like entrypoints and find_packages working?
Sounds correct to me. The site.py in site-packages wouldn't harm anything, btw, as it would never get imported. I suspect removing it breaks any attempt to use easy_install to install eggs to $PYTHONPATH (probably with a traceback and no sane error message) but perhaps that's what you intend.
participants (3)
-
Ben Bangert -
Phillip J. Eby -
primco