-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote:

At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:

...

Just to illustrate what I'm trying to achieve. I've updated the Fedora Packaging Guidelines[1]_ to allow two versions of a package to coexist. I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and the output of the test-sql.sh script using this procedure. The end result is what we want but the build step to get there seem a tad fragile and kludgey. Since these are going to become guidelines for all of our python packages, I'd like to know if either: 1) there's a better way to do this or 2) the results I'm achieving are not expected and could disappear with a random upgrade of setuptools.

.. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs

Here's the thing: if you want multiple installed versions of a package, *and* you want there to be a default version, then you *have* to have easy_install (or zc.buildout, or some similar tool) generate the startup scripts for anything that wants to use a non-default version.

You'll have to explain why that is because all my experiments show that to be false. Installing one package via easy_install -m and another via - --single-version-externally-managed gets us 75% of the way to this working. Using easy_install -m for both and then copying/symlinking gets us 100% of the way.

This is true irrespective of the formats of the versions involved, whether they're zipfiles, directories, single-version, or whatever. It's just the nature of the beast, due to the fact that there is a global 'working_set' that lists the projects that are currently on sys.path, and it is initialized when pkg_resources is imported. (And currently, there is no way to *remove* a distribution from the working set.)

Since the working_set doesn't explicitly list eggs unless they are specified in a project's requires.txt it seems like pkg_resources.require() can override a module which is the default because it is installed in site-packages. In fact, I understood this to be a feature of setuptools: allowing someone to override the vendor installed packages in site-packages with your own eggs.

Thus, if you want multiple versions *and* want to be able to select a version after pkg_resources has been imported, you *cannot* have a default version. In other words, the egg must not be on sys.path when pkg_resources is imported. Then, pkg_resources can locate the desired version and add it.

Not quite. In the single-version case, the egg is on sys.path because the module directory is in site-packages. Therefore pkg_resources make a different version importable by placing a different egg's path before site-packages in sys.path. The goal is to have all of these things work: 1) import MODULE should import whatever the vendor decided should be the default. 2) requires.txt in a project's egginfo should work to select a specific version from the installed eggs. 3) In a simple script (without requires.txt), pkg_resources.requires() should work to select a specific version from the installed eggs. I think the basic way to enable this is: For 1) Either install a module that is not an egg into site-packages or install an egg as single-version-externally-managed into site-packages. When the user does import MODULE, python looks in site-packages, finds the MODULE, and imports it without touching any setuptools code. For 2) All installed modules that we want to be selectable must be eggs. This needs to work with both single-version and multiple-version eggs when determining the best version match then, if necessary, modify sys.path to place the best match in front of the other eggs. For 3) No eggs for this module can be on sys.path before the pkg_resources.require() call. This does not count modules brought in via site-packages as those are going to be overridden when we place egg paths before site-packages. This seems to mostly work right now with the procedure I outlined in my previous message. I have to fix the bug found in my other message to see if I can get it to work all the time (and perhaps eliminate the kludginess in my original procedures.) You seem to think there's something wrong with this so there's obviously something you're seeing that I don't. Can you give an example of where this will fail? - -Toshio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG2dUKX6yAic2E7kgRAiMnAJ40guBFKrh0V3Yl+kzJqRlZ6V5TJgCgrA/j fnHPHB9Q6533xSAzELAV6Dc= =swaR -----END PGP SIGNATURE-----

At 02:09 PM 9/1/2007 -0700, Toshio Kuratomi wrote:

Phillip J. Eby wrote:

...

At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:

...

Just to illustrate what I'm trying to achieve. I've updated the Fedora Packaging Guidelines[1]_ to allow two versions of a package to coexist. I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and the output of the test-sql.sh script using this procedure. The end result is what we want but the build step to get there seem a tad fragile and kludgey. Since these are going to become guidelines for all of our python packages, I'd like to know if either: 1) there's a better way to do this or 2) the results I'm achieving are not expected and could disappear with a random upgrade of setuptools.

.. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs

Here's the thing: if you want multiple installed versions of a package, *and* you want there to be a default version, then you *have* to have easy_install (or zc.buildout, or some similar tool) generate the startup scripts for anything that wants to use a non-default version.

You'll have to explain why that is because all my experiments show that to be false. Installing one package via easy_install -m and another via - --single-version-externally-managed gets us 75% of the way to this working. Using easy_install -m for both and then copying/symlinking gets us 100% of the way.

I don't understand what you're saying. If you have multiple versions installed, and one of them is active by default (i.e., listed in a .pth, or a single-version in site-packages), then the only simple way to access the non-default version is via an easy_install-generated script.

Since the working_set doesn't explicitly list eggs unless they are specified in a project's requires.txt it seems like pkg_resources.require() can override a module which is the default because it is installed in site-packages. In fact, I understood this to be a feature of setuptools: allowing someone to override the vendor installed packages in site-packages with your own eggs.

It is -- but as I said, it only works for scripts generated by easy_install, if you want to import the non-default version.

...

Thus, if you want multiple versions *and* want to be able to select a version after pkg_resources has been imported, you *cannot* have a default version. In other words, the egg must not be on sys.path when pkg_resources is imported. Then, pkg_resources can locate the desired version and add it.

Not quite. In the single-version case, the egg is on sys.path because the module directory is in site-packages. Therefore pkg_resources make a different version importable by placing a different egg's path before site-packages in sys.path.

Again, that's *only* if you don't have a conflicting egg that's already the default.

The goal is to have all of these things work: 1) import MODULE should import whatever the vendor decided should be the default. 2) requires.txt in a project's egginfo should work to select a specific version from the installed eggs. 3) In a simple script (without requires.txt), pkg_resources.requires() should work to select a specific version from the installed eggs.

#1 and #2 are easy. #3 is possible only if you use the hack that easy_install-generated scripts use.

For 3) No eggs for this module can be on sys.path before the pkg_resources.require() call. This does not count modules brought in via site-packages as those are going to be overridden when we place egg paths before site-packages.

Yes it *does too* count, which is why it doesn't work as you expect.

You seem to think there's something wrong with this so there's obviously something you're seeing that I don't. Can you give an example of where this will fail?

Any time you expect to be able to import the non-default version of a package, and you either run the interpreter with no script, or run a script that wasn't generated by easy_install or otherwise hacked to work around the conflict issue.

At 02:54 PM 9/1/2007 -0700, Toshio Kuratomi wrote:

Toshio Kuratomi wrote:

...

This code creates an unordered list because you might have already had to place something after one of the duplicate locations that this code is removing. We need to do a sort of the entire list after each add + duplicate removal run.

Here's a quick and dirty patch. It's correct but it's inefficient. A better sort algorithm would help some. Refactoring code so we call a sort method after we finish adding entries to the path would be even better.

I'll test how this affects packaging tonight.

Please, please stop and back up a minute. appreciate your eagerness to help, really, but right now I can't trust the correctness of the initial system against which you are performing your tests. You've got crazy stuff on sys.path, you appear to be calling insert_on() (which isn't a documented API), and you're manually creating .pth files, among other things. So before you do anything else, please restore your system to something which consists *only* of files generated by setuptools or easy_install without *any* added hacks, workarounds, or manually edited files. That also means NO scripts that were not generated by easy_install. If you *must* have other scripts, there is an undocumented internal feature that you can use to specify a script's requirements such that they override the default package versions. What you have to do is add a __requires__ definition to the script, e.g.: __requires__= 'TurboGears>1.0', 'FibbledyDee<27.2' This definition must be in the actual script run by Python. When pkg_resources is initially imported, any __requires__ requirements are given higher precedence than the default versions. You must, however, still import pkg_resources, and it must be imported *after* setting __requires__, not before. Assuming I understand your requirements, you should be able to accomplish everything you want using only this one feature, plus single-version eggs for system default packages, and multi-version eggs (i.e. *no* .pth files) for everything else. It should not be necessary for you to generate or edit any files, use other undocumented APIs, or anything else.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote:

At 02:54 PM 9/1/2007 -0700, Toshio Kuratomi wrote:

...

Toshio Kuratomi wrote:

...

This code creates an unordered list because you might have already had to place something after one of the duplicate locations that this code is removing. We need to do a sort of the entire list after each add + duplicate removal run.

Here's a quick and dirty patch. It's correct but it's inefficient. A better sort algorithm would help some. Refactoring code so we call a sort method after we finish adding entries to the path would be even better.

I'll test how this affects packaging tonight.

Please, please stop and back up a minute. appreciate your eagerness to help, really, but right now I can't trust the correctness of the initial system against which you are performing your tests. You've got crazy stuff on sys.path, you appear to be calling insert_on() (which isn't a documented API), and you're manually creating .pth files, among other things.

Well, *I'm* not calling insert_on(). setuptools is calling insert_on() when it initializes via __requires__ = "TurboGears"; import pkg_resources (I did not write that script, it's tg-admin from TurboGears.) I just instrumented the pkg_resources() code to figure out what's going on and found sys.path being updated incorrectly. As for manually created .pth files, all the ones I created were removed right after I tested that they didn't work. None of the results I've posted involved them. However, there are .pth files that were generated by other people's packaging. I'll go through and remove or fix all of those packages on my system.

So before you do anything else, please restore your system to something which consists *only* of files generated by setuptools or easy_install without *any* added hacks, workarounds, or manually edited files. That also means NO scripts that were not generated by easy_install.

Uhm... I can get rid of the two packages that I changed and take care of the .pth files but really unless I remove everything but python and setuptools from this system, there's going to be packages that were built with distutils, setuptools, configure scripts, and etc here. Everything is managed via rpm so I could do that but I'd rather know what in particular I need to get rid of. (Like: Remove anything that installs a .pth file.) If you say remove everything that installs an egg that I haven't audited how it builds I can do that pretty easily.

If you *must* have other scripts, there is an undocumented internal feature that you can use to specify a script's requirements such that they override the default package versions. What you have to do is add a __requires__ definition to the script, e.g.:

__requires__= 'TurboGears>1.0', 'FibbledyDee<27.2'

This definition must be in the actual script run by Python. When pkg_resources is initially imported, any __requires__ requirements are given higher precedence than the default versions. You must, however, still import pkg_resources, and it must be imported *after* setting __requires__, not before.

Assuming I understand your requirements, you should be able to accomplish everything you want using only this one feature, plus single-version eggs for system default packages, and multi-version eggs (i.e. *no* .pth files) for everything else.

Does this work from the interpreter shell?

...

...

__requires__ = 'SQLAlchemy>=0.3,<0.4beta1' import pkg_resources import sqlalchemy

That might do the trick but it's not ideal for us to drop support for a documented interface in favor of an undocumented one. I'm writing guidelines for packagers, not programmers. By and large, we're not writing scripts, we're packaging python modules so that people who are writing scripts can get their work done. It's important that the documented way of doing things works. pkg_resource.requires() is documented on the web page, in pkg_resource.txt, and when you python setup.py install - --single-version-externally-managed. __requires__ is documented... nowhere. If you're willing to change documentation that says to use pkg_resources.require() to use __requires__ instead then this will work out perfectly. If not, we'll need to find a way to make pkg_resources.require() work so that we aren't constantly explaining to people that it doesn't work because upstream tells us it doesn't work and we're sorry that all the official upstream documentation says otherwise.

It should not be necessary for you to generate or edit any files, use other undocumented APIs, or anything else.

That's the unstated goal that goes along with the other three :-) Do you IRC? I'll fix and remove packages and then tell you what still works and doesn't work. - -Toshio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG2iPCX6yAic2E7kgRAkabAJoCbxLbgaiwtmjz19pyEESBTi4DZgCeO4W1 qok/7j9Jl2NW4UFQCpW12lw= =ye7N -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote:

At 07:45 PM 9/1/2007 -0700, Toshio Kuratomi wrote:

...

pkg_resource.requires() is documented on the web page, in pkg_resource.txt, and when you python setup.py install - --single-version-externally-managed. __requires__ is documented... nowhere. If you're willing to change documentation that says to use pkg_resources.require() to use __requires__ instead then this will work out perfectly. If not, we'll need to find a way to make pkg_resources.require() work so that we aren't constantly explaining to people that it doesn't work because upstream tells us it doesn't work and we're sorry that all the official upstream documentation says otherwise.

As I've explained repeatedly before, your choices are to either have a default version, or not to have one. If you do not have one, then everything works as you wish, except for the fact that you must always explicitly require() something to use it (because there's no default).

If do you have a default version, then the only way to get something *other* than the default version is to use __requires__ or a setuptools-generated script (which automatically includes __requires__).

Yes. And I'm repeating that the problem is the documentation doesn't match the behaviour. If using __requires__ works, then the documentation needs to mention it. Preferably, it should substitute for all the places where pkg_resources.require() is currently highlighted as the right way to do things. For instance, this output from easy_install:: ''' Because this distribution was installed --multi-version, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("SQLAlchemy") # latest installed version pkg_resources.require("SQLAlchemy==0.3.10") # this exact version pkg_resources.require("SQLAlchemy>=0.3.10") # this version or higher ''' I realize that taken in the vacuum of that single easy_install run, require() works. But the instructions are neglecting to tell the user that things are more complex than that. That depending on how the other versions of the module are installed, pkg_resources may not work at all. Since __require__ works for this instance and for a mixture of -m and - -s isn't it best to give users instructions that they can use everywhere? - -Toshio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG2lokX6yAic2E7kgRAjtcAKCw1qg4iRrQAQxeyyX8sE9WvMwMDwCbB6ZY ZrjwE12dAxBKrEoGra2b19s= =qjbi -----END PGP SIGNATURE-----

At 11:37 PM 9/1/2007 -0700, Toshio Kuratomi wrote:

I realize that taken in the vacuum of that single easy_install run, require() works. But the instructions are neglecting to tell the user that things are more complex than that. That depending on how the other versions of the module are installed, pkg_resources may not work at all.

You're taking this out of context. When you run "easy_install -m", the normal result is that *there is no default package*. So require() is both necessary and sufficient in that case.

Since __require__ works for this instance and for a mixture of -m and - -s isn't it best to give users instructions that they can use everywhere?

The problem is that you are adding a new situation to "everywhere", that previously didn't exist -- mixing single-version and multi-version at the system packaging level. Either have a default version and deal with conflicts, or no default version and be explicit. __requires__ is a workaround to avoid conflicts in specific cases. It is intended only as a workaround, and really it's only for tools like easy_install and zc.buildout that generate scripts from a project description. I do not intend to support it for any other usage. If you, as a packager, wish to package scripts that use __requires__, I don't see a problem with that. It is emphatically *not* intended for general use. There are already tools in place to do what you want; you're just trying to get away with not using them. To put it another way, if you want to support multiple versions with a default, then you need to support the *end-user being able to choose* what version is the default. If the user does an easy_install of some specific version, then *that* version needs to become the default. And the only way to support that, is for you to generate your scripts with easy_install or zc.buildout, or to make your own tool that generates scripts using __requires__. If you don't, then your system-level Python scripts will be hosed if the user changes the default version of a package they use. Now, it could be argued that you have packages that don't declare their dependencies currently (i.e., they don't use setuptools), and would also be hosed by a change in the default version. But if that's true, then you're not fully supporting multiple versions. So, my point here is that this issue is inherent to the nature of multiple versions. Either you have a default, or you don't. If you don't, you can't have conflicts but have to be explicit. If you do have a default, then you need to *let the user change it*, and make everything you supply be explicit (so that it will automatically resolve conflicts). I don't plan to support __requires__ for end-user scripts, but I'm happy to co-ordinate with system-level packagers and the authors of packaging tools to make sure you don't get messed up when it (eventually) goes away and is replaced by something better.

At 01:12 PM 9/2/2007 -0700, Toshio Kuratomi wrote:

What is the real way to allow people to do quick and dirty scripting and experimentation from the interpreter shell in this environment?

Either: 1. Select the desired default version using "easy_install package==version", or 2. Have *no* default version, and always use require() (or automatically-managed dependencies declared via a setup.py script).

And once again, we are *not* creating and packaging scripts that use __require__. We are packaging multiple versions of modules with a default.

And once again, this won't work. Either you must not have a default version, or you must deal with the potential conflict in your scripts. The supported way of doing that is to generate the scripts using easy_install. If you are packaging something that doesn't use setuptools explicitly, you'll just have to patch it to declare its dependencies. You're already patching the .spec file; heck, if it would make it more palatable, perhaps we could add a way to specify the requirements via the command line. (Alternately, you could just create a requires.txt in the .egg-info directory.)

We need to have instructions for end-users who want to do quick and dirty scripting or experiment in the interpreter shell how to select the version that they wish.

easy_install package==version will change the default version, and require() will work if there is no default version. Those are the supported ways of doing this.

Once again, *we are not creating any system level python scripts that make use of a specific, non-default version*.

I'm saying that even your programs that use the default version, should explicitly specify what version they use. If you're going to support multiple versions, then declare *all* the dependencies, or at least the ones that would otherwise have conflicts.

B) For projects that have not been ported to a new version of the module. We have no control over how those upstream scripts are written -- whether they use easy_install, hand code __requires__, or munge sys.path themselves.

The correct instructions are for them to use setuptools and declare their dependencies, or live with whatever the status quo is.

This is true. But it's not going to change until all upstreams have moved to using setuptools. We provide a set of packages that work together. And we provide a means for the user to shoot themselves in the foot. But this is nothing new, they've had access to PYTHONPATH, su, rm -rf, etc in the past.

What we can do is make as much of what they expect to work work as it used to. And where it differs, we offer a standard, logical, and simple method to get the behaviour they want.

Right - and the single simplest solution for all of this is to simply use setuptools as it was designed: select a default version using easy_install, and declare dependencies for everything else. Alternatively, don't install *anything* as the default version, and declare *all* dependencies. If your goal is to have simple and consistent behavior, the second is the best choice. If you want the greatest backward-compatibility, the first is the best choice. Note that it isn't really possible for users *now* to choose a different package version at runtime, so requiring the use of easy_install to switch default versions doesn't seem like a big deal. They already have to make this choice by what RPMs they install, and it's just as global a choice -- except that this way, they can do it without breaking anything.

If you feel that __requires__ is a private method that shouldn't be used then we'd respect that. But between that and your teelling us that pkg_resources.require() isn't meant to work like that, it means that there's simply no method for making setuptools do what we need it to and we'll have to find some other method to accomplish this.

There are *two* ways to accomplish it: have a default (and use easy_install to change it when desired), or don't have a default. Either way, you are going to ultimately end up at a place where everything needs to declare its dependencies, preferably in a setup.py. Having a default version may be useful for the transition period, but the best way to encourage people to actually transition is to have some *benefit* for declaring their dependencies -- like being able to use the non-default versions. This is why I even discourage direct use of require(), as it says in the docs: """In general, it should not be necessary for you to call this method directly. It's intended more for use in quick-and-dirty scripting and interactive interpreter hacking than for production use. If you're creating an actual library or application, it's strongly recommended that you create a "setup.py" script using setuptools, and declare all your requirements there. That way, tools like EasyInstall can automatically detect what requirements your package has, and deal with them accordingly."""

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phillip J. Eby wrote:

At 12:49 AM 9/2/2007 -0700, Toshio Kuratomi wrote:

...

Let me know what else you need.

What are the current contents of easy-install.pth?

import sys; sys.__plen = len(sys.path) import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) Do you want me to get rid of it and try again? - -Toshio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFG247uX6yAic2E7kgRAnHsAJ9KO93oSNgVH3xhuQwcB2pbd1EoaQCfftvj VLnmUUNYGn0kV9qAYogS5FI= =+rgQ -----END PGP SIGNATURE-----

On 02/09/07, Phillip J. Eby <pje@telecommunity.com> wrote:

At 01:12 PM 9/2/2007 -0700, Toshio Kuratomi wrote:

...

What is the real way to allow people to do quick and dirty scripting and experimentation from the interpreter shell in this environment?

Either:

1. Select the desired default version using "easy_install package==version", or 2. Have *no* default version, and always use require() (or automatically-managed dependencies declared via a setup.py script).

Is it really setuptools job to attempt to address "quick & dirty scripting" or "interactive experimentation" ? I mean, as I understand it, easy_install is specifically about giving you a clean painless install of a piece of python software without having to care about the details and without needing to understand anything about python. But the expectations and needs of the user in Toshio's scenario are completely at odds with this. This is a person who is clearly interested in, or needs to understand, the details of some piece of python software but who does not want to be concerned with all the horrors of installation / distribution management. Or perhaps it's simply a person who is *considering* system wide installation of some python package and wishes to "try before they buy". I frequently want to do exactly this sort of thing - often in circumstances where I can not count on having a consistent setuptools environment. Or, where I want to use a mix of installed and non installed python packages. I almost always need to deal with a mix of distutils installed, egg installed and plain old python. Occasionally I'm in the situation where I have a setuptools egg on disc, I'd like to use it, but I do not want to or simply can't install it. Almost by definition anything that needs to be "installed" is not going to help; Bootstrapping issues abound. But I *know* roughly where the python code is that I want to make use of. And heck & damn it all; There are a great many python packages whose source distributions are perfectly capable of running in place - why should I have to "install" such software before I'm satisfied it makes a useful addition to my system ? In something of a fit of frustration I wrote this ~1000 line python module: http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.py http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.html wget is all the installation it needs. The -i -d and -p modes of operation are the ones I think might address "quick & dirty scripting" and "experimentation" from Toshios scenario. I've not released it on cheeseshop because I wasn't really convinced there would be much demand. And also because I didn't like the thought that it might be confused as "general script stub" - I really would not like to encounter a site installed script that used pyrun.py to shim around installation / packaging issues. I'm not sure I fully appreciate the issues your facing Toshio but your posts struck something of a chord with me. If you think pyrun.py can be made to help please let me know. Cheers, Robin