EasyInstall: bootstrapping
I added a subcommand to Paste "paster install X", which is basically just easy_install with some hooks into Paste's configuration. But anyway, now I'm wondering how to get the whole thing going. You can't run easy_install without setuptools installed. I'm inclined to install easy_install with easy_install, but obviously that won't work ;) Of course, I can download the egg and put that into the path; that's what Paste's setup.py does currently (which also gives me package_data on older Pythons as well). But where should I put the egg? For setup.py it doesn't matter that much, but it matters more when I want to keep it around for future commands. Should I download it, put it on the path, then use it to install itself? I dunno, I'm just feeling a little mixed up. I'm also thinking I could start installing things globally now that there's essentially versioned imports... except that I probably have to hardcode the versions when deploying, so I don't accidentally upgrade apps without meaning to. Or I can stick with app-packages to be extra safe. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 01:51 PM 6/3/2005 -0500, Ian Bicking wrote:
I added a subcommand to Paste "paster install X", which is basically just easy_install with some hooks into Paste's configuration.
But anyway, now I'm wondering how to get the whole thing going. You can't run easy_install without setuptools installed. I'm inclined to install easy_install with easy_install, but obviously that won't work ;) Of course, I can download the egg and put that into the path; that's what Paste's setup.py does currently (which also gives me package_data on older Pythons as well). But where should I put the egg? For setup.py it doesn't matter that much, but it matters more when I want to keep it around for future commands. Should I download it, put it on the path, then use it to install itself?
I dunno, I'm just feeling a little mixed up. I'm also thinking I could start installing things globally now that there's essentially versioned imports... except that I probably have to hardcode the versions when deploying, so I don't accidentally upgrade apps without meaning to. Or I can stick with app-packages to be extra safe.
Hm. I would suggest that the safest tack to take is try importing pkg_resources and do a 'require()' for the setuptools version that you need. If the import fails or the version fails, bail out of setup.py, asking for the user to install the correct setuptools. I think that trying to invoke easy_install recursively or guess where the egg should be installed is a bad idea. (E.g., guessing about the egg location will cause easy_install's sandboxing to shut you down anyway) However, it would certainly be nice to have a way for setuptools-using packages to bootstrap the setuptools install -- and that includes setuptools itself. (My head is starting to spin now.) Hm. Unfortunately, it seems that the only other sane way to do this is to distribute Paste as an egg, and require people to use easy_install to install it, because then any dependency back to setuptools can be resolved properly once easy_install's automatic dependency installation is in place. That doesn't help you much *now*, I realize. The big problem is that 'pkg_resources' really needs to be a singleton; there can't be multiple versions of it floating around at the same time, due to it containing global registries. Otherwise I'd tell you to just stick pkg_resources and easy_install in a package of your own and not worry about it. Anyway, this is definitely a case for putting at least pkg_resources into the standard library as soon as it stabilizes sufficiently.
Phillip J. Eby wrote:
Hm. I would suggest that the safest tack to take is try importing pkg_resources and do a 'require()' for the setuptools version that you need. If the import fails or the version fails, bail out of setup.py, asking for the user to install the correct setuptools. I think that trying to invoke easy_install recursively or guess where the egg should be installed is a bad idea. (E.g., guessing about the egg location will cause easy_install's sandboxing to shut you down anyway)
However, it would certainly be nice to have a way for setuptools-using packages to bootstrap the setuptools install -- and that includes setuptools itself. (My head is starting to spin now.)
Hm. Unfortunately, it seems that the only other sane way to do this is to distribute Paste as an egg, and require people to use easy_install to install it, because then any dependency back to setuptools can be resolved properly once easy_install's automatic dependency installation is in place.
I wouldn't mind so much requiring easy_install to be installed -- if that's the only firm prerequesite, that's okay. But a Paste checkout gives you docs and examples and unit tests, which aren't part of an egg, so Paste as installed by easy_install isn't complete. One option would be if there was an option to install something the same way you recommend using a development checkout -- don't create an egg, just make sure the PackageName.egg-info directory is in place. Incidentally, should there be a egg_setup command or something short of creating an egg you won't use to create that special directory?) Also, if I was just going to get rid of paster install (which is fine, no big investment there), it does have the plugin-enabling code (where plugins can register themselves with Paste). I'm not sure where that would go; you could put the code into plugins themselves -- very reasonable -- except how do they find the appropriate Paste installation? I hate PYTHONPATH, and that's where this would lead.
That doesn't help you much *now*, I realize. The big problem is that 'pkg_resources' really needs to be a singleton; there can't be multiple versions of it floating around at the same time, due to it containing global registries. Otherwise I'd tell you to just stick pkg_resources and easy_install in a package of your own and not worry about it.
Well, I'm wondering if it's okay to just download the setuptools egg and put it on sys.path manually (obviously not using require() or anything in setuptools). That's pretty easy to do. The only problem is when they have a version of setuptools that is older than required -- if I try to import it (not using require()) and it's too old, then I have to delete it from sys.modules, which starts getting kind of long. Especially since I can't write a library to do this ;) I have thought in the past about writing a setup.py-builder, which might make this more reasonable.
Anyway, this is definitely a case for putting at least pkg_resources into the standard library as soon as it stabilizes sufficiently.
That might help. But only maybe. Having distutils in the stdlib isn't making things any less complicated right now. I almost think it makes sense to just have a hard line for packages that use this stuff: if setuptools isn't installed (or is too old), then go install it and start again. That way, assuming it is popular, it can become kind of standard regardless of your Python version. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 04:05 PM 6/4/2005 -0500, Ian Bicking wrote:
Phillip J. Eby wrote:
Hm. I would suggest that the safest tack to take is try importing pkg_resources and do a 'require()' for the setuptools version that you need. If the import fails or the version fails, bail out of setup.py, asking for the user to install the correct setuptools. I think that trying to invoke easy_install recursively or guess where the egg should be installed is a bad idea. (E.g., guessing about the egg location will cause easy_install's sandboxing to shut you down anyway) However, it would certainly be nice to have a way for setuptools-using packages to bootstrap the setuptools install -- and that includes setuptools itself. (My head is starting to spin now.) Hm. Unfortunately, it seems that the only other sane way to do this is to distribute Paste as an egg, and require people to use easy_install to install it, because then any dependency back to setuptools can be resolved properly once easy_install's automatic dependency installation is in place.
I wouldn't mind so much requiring easy_install to be installed -- if that's the only firm prerequesite, that's okay. But a Paste checkout gives you docs and examples and unit tests, which aren't part of an egg, so Paste as installed by easy_install isn't complete.
Well, there's always -b, as of 0.3a3, which keeps the unpacked source distribution after installation. Does that help?
One option would be if there was an option to install something the same way you recommend using a development checkout -- don't create an egg, just make sure the PackageName.egg-info directory is in place.
Ugh. I don't want to do that for stuff that isn't either a development checkout or part of the standard library. Anywhere in between and you start to mess up version management. Note that when an .egg-info directory is in a directory on sys.path, it means the egg in question is *already* on sys.path, which means you can't require() a version that conflicts with it. So, .egg-info is really a trick for development. EasyInstall already takes care of everything you need for installing as an egg directory, so installing .egg-info doesn't really provide any benefit that I see. Now, if somebody wants to manually set a .pth file (or PYTHONPATH et al) to point to a Paste checkout and its included .egg-info, that's fine: just add the .egg-info directory to subversion after you've generated it, and you're all set. This is in fact recommended for development scenarios. You just don't want to try and do this by actually *installing* the package and .egg-info to site-packages, because then you can't get it back out again.
Incidentally, should there be a egg_setup command or something short of creating an egg you won't use to create that special directory?)
I guess I could factor it out into an 'egg_info' command, and call it from 'bdist_egg'. It might take some work to prevent duplication, though, and a big problem is that it wants to build a 'native_libs.txt' in the info directory, which it has to get the information for from the install_lib command. But I could maybe not generate that file in the 'egg_info' command, but then there's duplication of process in bdist_egg to find the egg-info directory. No, it doesn't make any sense to do all that extra work. The idea that the egg is something "you won't use" is an illusion in the long run, and meanwhile the build process helps validates the correctness of your ``setup.py`` and much of the rest of your package. The disk space and CPU to build the egg should be minimal, and you only need to do it again if you change your package's metadata (e.g. its version). The rest of the time you can use it as-is from subversion or CVS. So just use bdist_egg.
Also, if I was just going to get rid of paster install (which is fine, no big investment there), it does have the plugin-enabling code (where plugins can register themselves with Paste). I'm not sure where that would go; you could put the code into plugins themselves -- very reasonable -- except how do they find the appropriate Paste installation? I hate PYTHONPATH, and that's where this would lead.
I don't understand what you're trying to do here. Why do plugins need to find Paste? Shouldn't Paste find the plugins? My general approach to application plugin installation is that applications have a plugin directory where the plugins' eggs get dropped (either via EasyInstall, or by the app itself). The application then just uses ``find_distributions(directory)`` to iterate over the distributions and check for plugin metadata, and do any registering of services provided by the plugins. (Or, you can use the ``AvailableDistributions`` class to automatically filter, index, and sort the eggs found.) For larger application frameworks comparable to Eclipse in the Java world, those apps would likely keep a registry of already-installed plugins and their registered services, and thus would only scan the plugin directory to locate newly- installed plugins. Some applications might also have an explicit installation process of their own. Anyway, as you can see, there are many architectures you can use to manage application plugins using Python Eggs. I should probably expand on the above a bit and add it to the docs, but I'd really like the rest of EasyInstall to be done, so that it can use EasyInstall as an example of downloading packages and resolving dependencies automatically. (Although in truth most application plugin systems will probably want to just download eggs directly rather than building them, and have some application-centric directory of download URLs.)
I almost think it makes sense to just have a hard line for packages that use this stuff: if setuptools isn't installed (or is too old), then go install it and start again. That way, assuming it is popular, it can become kind of standard regardless of your Python version.
That's my inclination as well. I suspect that web application deployment is a killer use case for eggs, too. Certainly it was the use case (besides Chandler) that pushed me to take action on designing and implementing Python Eggs.
participants (2)
-
Ian Bicking -
Phillip J. Eby