On Aug 25, 2013, at 3:06 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 25 August 2013 16:41, Donald Stufft <donald@stufft.io> wrote:

...

On Aug 25, 2013, at 1:57 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

[snip]

I'll look at this closer, my off the cuff response isn't good but before I commit to a side I want to dig into how it actually works currently.

The clumsiness of the __main__.__requires__ workaround aside, the main advantage this offers is that it *should* result in a relatively straightforward addition to pkg_resources to make it work with wheel files as well as eggs. That's important, because anyone that is currently doing side-by-side multi-versioning in Python is using the pkg_resources API to do it, since that's the only option currently available.

If Fedora is going to switch completely to a wheel based build process, we need to be able to do it in a way that allows side-by-side imports through pkg_resources to continue to work.

I'd previously considered writing a pkg_resources replacement that worked with wheels instead of eggs, but I've now realised that would be repeating one of the mistakes made with the distutils2 effort: just as we needed to account for the fact that a lot of people are currently building their projects with setuptools, we also need to account for the fact that anyone doing side-by-side imports is using pkg_resources.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yea I understand the motivations for it. The main thing i'm worried about is codifying a solution that ends up being a misfeature instead of a feature. This may be perfectly fine (hence why I want to dig into further having never used it greatly before) but in general I think we should be conservative in copying things over from the old way into the new way. The new tools in many ways are removing features that are less than optimal or overall a bad idea and that's good. We don't need to copy over every single feature if someone really depends on a feature we don't bring over they can continue to use the existing tooling. That being said I don't know for sure that this falls into that category which is why I want to dig into it more. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 25 August 2013 16:41, Donald Stufft <donald@stufft.io> wrote:

...

On Aug 25, 2013, at 1:57 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

[snip]

I'll look at this closer, my off the cuff response isn't good but before I commit to a side I want to dig into how it actually works currently.

The clumsiness of the __main__.__requires__ workaround aside, the main advantage this offers is that it *should* result in a relatively straightforward addition to pkg_resources to make it work with wheel files as well as eggs. That's important, because anyone that is currently doing side-by-side multi-versioning in Python is using the pkg_resources API to do it, since that's the only option currently available.

No. It isn't. Buildout doesn't use pks_resources to do it. (Buildout used pkg_resources at build time to manage package meta data, but I think that's orthogonal to what you're talking about.) I'd also hazard to guess that most of the folks with multi-version installs are using buildout to do it, as buildout does have a fair number of users. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On 25 August 2013 20:53, PJ Eby <pje@telecommunity.com> wrote:

FWIW, I would also note that if you use easy_install to install anything, you are quite possibly using multi-version installs without realizing it. (The __main__.__requires__ API is used in easy_install-generated script wrappers, so there isn't any way you'd know about it without paying specific attention.)

Unless I'm missing something, I suspect that this over-counts the number of people using multi-version, in the sense that many (the majority?) of wrapper scripts using multi-version do not actually need to,because the users never install more than one version. And quite likely don't even know that they could. Paul.

On 25 August 2013 23:14, PJ Eby <pje@telecommunity.com> wrote:

...

Unless I'm missing something, I suspect that this over-counts the number of people using multi-version, in the sense that many (the majority?) of wrapper scripts using multi-version do not actually need to,because the users never install more than one version. And quite likely don't even know that they could.

That's just it: if you install two programs, one of which needs CherryPy 2 and the other CherryPy 3, then with easy_install this just works, without you having any idea that you even have more than one version installed, unless you for some reason choose to look into it.

Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around.

OK, I see. But I'm not sure if we're agreeing or disagreeing over the result. To me, this is a bad thing on the principle that there is a cost to multiversion support (it's not part of core Python, so you have to do *something* to make it work) and so having people inadvertently pay that cost to use a feature that they don't actually *need* is wrong. An opt-in solution is fine, as in that case the user has to choose to use multiversion, and if they don't want to they can choose an alternative approach that they accept the cost of (for example, running their one CherryPy 2 using application in a virualenv). One other point, just as a matter of curiosity (because it's not relevant to the current discussion): in your explanation above, there doesn't seem to be any step that says the user normally uses CherryPy 3 (so that would be the one they would get automatically at the interactive interpreter). For me, that's really the only use case I'd have for multi-versioning - 99% of the time I use a particular version of a project, but I have one particular application that can't work with the version I prefer. Paul

Daniel Holth wrote:

I would like to see some consideration given to what Ruby and npm do, which is to place what we are calling dists into a special directory that only contains dists /somedir/distname-1.0/... rather than placing them as specially named objects on the normal search path.

What advantage would there be in doing that? I can think of one disadvantage -- we would then need new rules to determine their priority in the search order. -- Greg

PJ Eby <pje <at> telecommunity.com> writes:

That is all easy_install does: add a naming convention for the directories, and automate the sys.path manipulation.

Buildout does the same thing, it just writes the sys.path manipulation into the scripts statically, instead of using pkg_resources at runtime.

So the notion of "cost" doesn't make any sense. Tools like easy_install and buildout *reduce* the management cost, they don't add anything to core Python.

(Now, if you're talking about the .pth files from easy_install, those are something that I added because people complained about having to use require(), and wanted to have a default version available in the interpreter.)

Pre-3.3, there is a non-negligible runtime cost per sys.path entry, because each import tries importing multiple filenames for each sys.path entry. Post-3.3, things should be better (thanks to the directory contents cache), but there's still a computational overhead for each sys.path entry. So, yes, easy_install adding sys.path entries *by default* using .pth files is clearly not costless. pip doesn't have this problem AFAIK (it doesn't create .pth files). Regards Antoine.

Just to add a bit more "FUD" ;-) I do a lot of packaging things up with py2app, py2exe, etc. -- I find I often want to be able to give folks "one thing" that they can install and run, and I'd rather they don't even need to know it's built with python. A while back, when I was doing this with a web app (actually a "Browser Interface, Local Server (BILS) app --i.e, a web server and html interface, but the browser embedded in a wx app so it acted like a desktop app to the user...) setuptools, and pkg_resources really made it hard. I don't remember the details at this point, but it was built on Pylons, which made heavy use of setuptools at the time. And easy_install pylons worked great. But bundling it up was a pain -- we had to put an enormous amount of crap in there to satisfy pkg_resources, and who knows what -- crap that was not used, and should not have been needed at run time. In fact, we needed to put in at least the skeleton of stuff that was a different version than the one used. Note also that the only reliable way to get an egg-installed package to work with py2app was to include the whole egg. My conclusion at the time was that there was a real confusion between what should happen at install time, and what should happen at run time. For instance, making sure that a dependency is there and the right version seems like an install-time problem to me -- and yet, there is was, getting checked every time you started up the app. I'm not clear at this point how much of that was inherent to how setuptools works, and how much was how Pylons chose to use it, but I hope we'll all keep this use-case in mind at this point. i.e. if a user has only needs one version of a given package installed, lets not have much overhead there to support that, and let's not require much run-time support at all. I note that I've been a user of wx.version for wxPython, and it was always easy and painless, including py2app, py2exe use. So it can be done. -Chris On Mon, Aug 26, 2013 at 8:15 AM, Donald Stufft <donald@stufft.io> wrote:

On Aug 26, 2013, at 10:33 AM, PJ Eby <pje@telecommunity.com> wrote:

...

On Mon, Aug 26, 2013 at 5:20 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

On 25 August 2013 23:14, PJ Eby <pje@telecommunity.com> wrote:

...

Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around.

OK, I see. But I'm not sure if we're agreeing or disagreeing over the result. To me, this is a bad thing on the principle that there is a cost to multiversion support (it's not part of core Python, so you have to do *something* to make it work)

Seriously? The basic functionality of using sys.path to have multiple versions *is* part of core Python, and has been since 1.5.2 (16 years ago), and probably longer than that.

In the days before easy_install and virtualenv, if you needed different versions of things, you used "setup.py install" to different directories (assuming distutils was involved, otherwise you just copied files) and either put your scripts in the same directories, or used PYTHONPATH or explicit sys.path manipulation.

That is all easy_install does: add a naming convention for the directories, and automate the sys.path manipulation.

Buildout does the same thing, it just writes the sys.path manipulation into the scripts statically, instead of using pkg_resources at runtime.

So the notion of "cost" doesn't make any sense. Tools like easy_install and buildout *reduce* the management cost, they don't add anything to core Python.

(Now, if you're talking about the .pth files from easy_install, those are something that I added because people complained about having to use require(), and wanted to have a default version available in the interpreter.)

...

and so having people inadvertently pay that cost to use a feature that they don't actually *need* is wrong.

What cost are you talking about here? Given that most people don't even know they *have* multiple versions installed or care, how is a cost being imposed upon them? Are you talking about disk storage?

...

One other point, just as a matter of curiosity (because it's not relevant to the current discussion): in your explanation above, there doesn't seem to be any step that says the user normally uses CherryPy 3 (so that would be the one they would get automatically at the interactive interpreter).

If they easy_install that version, sure, that's what they'll get as a default version.

...

For me, that's really the only use case I'd have for multi-versioning - 99% of the time I use a particular version of a project, but I have one particular application that can't work with the version I prefer.

Yes, and that's the sort of scenario Nick was proposing pip support, that you have an explicit "install me a different version for my other app" capability -- such that that app's script wrapper adds its alternate version to sys.path ahead of the default one. So it would have been opt-in and impose the "cost" of a slightly longer sys.path and increased disk space usage only on those who ask for it.

(Honestly, 90% of this entire thread has sounded like complete FUD to me, i.e. fear based on a lack of understanding that there actually isn't anything magical about multi-version support. As Jim has pointed out, buildout does multi-version support without even using pkg_resources. And before all these tools existed, people just installed things in different directories and used either adjacent scripts, PYTHONPATH, or explicit sys.path manipulation. There is nothing magical whatsoever about having multiple versions of a thing installed on your system; all the tools do is add naming conventions for where stuff is installed... and having such naming conventions is a *good* thing, compared to the old days.)

There is always a cost. In this case mostly in complexity and start up time.

As you mentioned originally the cost to multi version support was the need to use a require() function and when people complained about that you added the .pth files which imposed another sort of cost to people using multi versioned installs.

You claim it is part of core Python but it's really not, if it was it wouldn't require importing pkg_resources of the .pth files to make it work.

I find it ridiculous that you'd call this thread 90% FUD when the vast bulk of the thread has been trying to determine if there were any reasonable concerns with the approach and upon examination determined that the biggest problem with it was attaching it to Wheel and not the multi version support at all. I realize setuptools and easy_install are your baby but the persecution complex doesn't help to win people over to your side of things.

In my experience setuptools has a lot of good ideas but they are wrapped in bad ideas or implementations that obscure the fact that there *are* good ideas there. I do not believe it to be unreasonable for people to want to make sure that we're standardizing around one of the *good* ideas instead of one of the bad ideas.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Mon, Aug 26, 2013 at 11:15 AM, Donald Stufft <donald@stufft.io> wrote:

There is always a cost. In this case mostly in complexity and start up time.

As you mentioned originally the cost to multi version support was the need to use a require() function and when people complained about that you added the .pth files which imposed another sort of cost to people using multi versioned installs.

See, this is exactly what I'm talking about: you've got this 100% backwards: .pth files are for people who *aren't* using multi-version imports. They're for *default* versions, not alternate versions! And they're utterly unnecessary for Nick's proposal.

You claim it is part of core Python but it's really not, if it was it wouldn't require importing pkg_resources of the .pth files to make it work.

As I pointed out in the email you apparently didn't read, along with multiple emails from Jim: pkg_resources isn't necessary for alternate-version support. All that's required for alternate versions is to add them to sys.path, which buildout does just fine *without pkg_resources*.

I find it ridiculous that you'd call this thread 90% FUD when the vast bulk of the thread has been trying to determine if there were any reasonable concerns with the approach and upon examination determined that the biggest problem with it was attaching it to Wheel and not the multi version support at all

What I'm referring to as the FUD is that people have been confusing what Nick proposed with what setuptools does, and getting *both* of them wrong in the details. Nick's proposal was not to mimic setuptools' multi-version support, but rather to provide something else: let's call it "alternate version support", to separate it from what setuptools does. In Nick's AVS proposal, there is *no* overhead for anything that doesn't need a non-default version, and it's 100% opt-in, used only for things that need *non-default* versions. Note, by the way, that since these *non-default* packages aren't on sys.path by default, *there is no overhead and no .pth files are involved*. They are effectively invisible and irrelevant for anything that doesn't use them. The only place where there's overhead is in the script that needs the alternative version(s), and its sys.path is lengthened only by those items that it can't obtain from the default sys.path. And if you use buildout's approach of simply adding: sys.path[0:0] = [path1,...] to the head of a script, then *pkg_resources isn't involved either*. This is bog-standard stock Python. So the FUD part I was referring to is all the "oh no, setuptools is complicated" in response to Nick's perfectly reasonable idea *which doesn't involve any of setuptools' complexity*, because it's doing something completely different.

I realize setuptools and easy_install are your baby but the persecution complex doesn't help to win people over to your side of things.

I think you're confused here. I don't think setuptools is being persecuted, I think *Nick's idea* is being misunderstood, and being construed as almost the exact *opposite* of what it is. All the stuff people bitch about that relates to multi-versions in setuptools are actually issues with setuptools' implementation of *default* versions, not *alternative* versions. So to look at Nick's proposal and think it's going to have the same problems is completely ludicrous - it's 180 degrees opposite of what setuptools does, because for setuptools, *default versions* are the special case -- they're what cause 90% of the complexity in pkg_resources' manipulation of sys.path, and they're the main reason .pth files are ever used. So it's crazy-making to see people thinking Nick's proposal is going to bring all that crap along, when that's the exact *opposite* of the situation.

In my experience setuptools has a lot of good ideas but they are wrapped in bad ideas or implementations that obscure the fact that there *are* good ideas there. I do not believe it to be unreasonable for people to want to make sure that we're standardizing around one of the *good* ideas instead of one of the bad ideas.

It would help if people understood the actual facts, then. AFAICT, Nick's proposal doesn't do any of the things that people are worried about, or at the very least does not *require* them. As Jim and I have pointed out more than once, pkg_resources is not a runtime requirement to allow alternative versions to be importable by code that wants them. It would really be a shame to shoot down Nick's idea based on a vague misunderstanding of it. It's a good proposal, and has far less to do with setuptools than most people in the thread seem to think.

On Aug 26, 2013, at 4:40 PM, PJ Eby <pje@telecommunity.com> wrote:

On Mon, Aug 26, 2013 at 11:15 AM, Donald Stufft <donald@stufft.io> wrote:

...

There is always a cost. In this case mostly in complexity and start up time.

As you mentioned originally the cost to multi version support was the need to use a require() function and when people complained about that you added the .pth files which imposed another sort of cost to people using multi versioned installs.

See, this is exactly what I'm talking about: you've got this 100% backwards:

.pth files are for people who *aren't* using multi-version imports. They're for *default* versions, not alternate versions!

And they're utterly unnecessary for Nick's proposal.

...

You claim it is part of core Python but it's really not, if it was it wouldn't require importing pkg_resources of the .pth files to make it work.

As I pointed out in the email you apparently didn't read, along with multiple emails from Jim: pkg_resources isn't necessary for alternate-version support. All that's required for alternate versions is to add them to sys.path, which buildout does just fine *without pkg_resources*.

...

I find it ridiculous that you'd call this thread 90% FUD when the vast bulk of the thread has been trying to determine if there were any reasonable concerns with the approach and upon examination determined that the biggest problem with it was attaching it to Wheel and not the multi version support at all

What I'm referring to as the FUD is that people have been confusing what Nick proposed with what setuptools does, and getting *both* of them wrong in the details.

Nick's proposal was not to mimic setuptools' multi-version support, but rather to provide something else: let's call it "alternate version support", to separate it from what setuptools does.

In Nick's AVS proposal, there is *no* overhead for anything that doesn't need a non-default version, and it's 100% opt-in, used only for things that need *non-default* versions.

Note, by the way, that since these *non-default* packages aren't on sys.path by default, *there is no overhead and no .pth files are involved*. They are effectively invisible and irrelevant for anything that doesn't use them.

The only place where there's overhead is in the script that needs the alternative version(s), and its sys.path is lengthened only by those items that it can't obtain from the default sys.path. And if you use buildout's approach of simply adding:

sys.path[0:0] = [path1,...]

to the head of a script, then *pkg_resources isn't involved either*.

This is bog-standard stock Python.

So the FUD part I was referring to is all the "oh no, setuptools is complicated" in response to Nick's perfectly reasonable idea *which doesn't involve any of setuptools' complexity*, because it's doing something completely different.

...

I realize setuptools and easy_install are your baby but the persecution complex doesn't help to win people over to your side of things.

I think you're confused here. I don't think setuptools is being persecuted, I think *Nick's idea* is being misunderstood, and being construed as almost the exact *opposite* of what it is.

All the stuff people bitch about that relates to multi-versions in setuptools are actually issues with setuptools' implementation of *default* versions, not *alternative* versions. So to look at Nick's proposal and think it's going to have the same problems is completely ludicrous - it's 180 degrees opposite of what setuptools does, because for setuptools, *default versions* are the special case -- they're what cause 90% of the complexity in pkg_resources' manipulation of sys.path, and they're the main reason .pth files are ever used.

So it's crazy-making to see people thinking Nick's proposal is going to bring all that crap along, when that's the exact *opposite* of the situation.

...

In my experience setuptools has a lot of good ideas but they are wrapped in bad ideas or implementations that obscure the fact that there *are* good ideas there. I do not believe it to be unreasonable for people to want to make sure that we're standardizing around one of the *good* ideas instead of one of the bad ideas.

It would help if people understood the actual facts, then. AFAICT, Nick's proposal doesn't do any of the things that people are worried about, or at the very least does not *require* them. As Jim and I have pointed out more than once, pkg_resources is not a runtime requirement to allow alternative versions to be importable by code that wants them.

It would really be a shame to shoot down Nick's idea based on a vague misunderstanding of it. It's a good proposal, and has far less to do with setuptools than most people in the thread seem to think.

I think you're confused. The only comments I see in this thread are people doing due diligence to ensure that Nick's proposal *didn't* include the parts of setuptools that we felt were incurring a cost against people not using the feature and expressing a desire *not* to attach it to the Wheel format and instead attach it to another format on it's own. I mean the person you originally quoted even explicitly said "I have no objection to this proposal" sans some stuff about not wanting it to be attached to Wheels. So I'm not sure how you can take someone saying they have no objection to the proposal and translate it to people are shooting down Nick's proposal. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, Aug 26, 2013 at 5:59 PM, Donald Stufft <donald@stufft.io> wrote:

I think you're confused. The only comments I see in this thread are people doing due diligence to ensure that Nick's proposal *didn't* include the parts of setuptools that we felt were incurring a cost against people not using the feature and expressing a desire *not* to attach it to the Wheel format and instead attach it to another format on it's own. I mean the person you originally quoted even explicitly said "I have no objection to this proposal" sans some stuff about not wanting it to be attached to Wheels. So I'm not sure how you can take someone saying they have no objection to the proposal and translate it to people are shooting down Nick's proposal.

FUD stands for fear, uncertainty, doubt. My comment was that a lot of the original objections to Nick's proposal seemed fearful, uncertain, and doubting, specifically because they were thinking the proposal was proposing things it wasn't. It was you who brought up the idea of persecution; my response was that I don't think anybody's persecuting setuptools, only giving unnecessary levels of doubt to Nick's proposal due to confusion about how it relates (i.e. mostly doesn't) to setuptools. You pounced on a tiny piece of my email to Paul, in which I mainly expressed confusion about his statements about "cost". I was having trouble understanding what sort of "costs" he meant, and in subsequent discussion realized that it's because he and others appeared to have conflated setuptools' default-version issues, with Nick's proposal for handling non-default versions. My comment was that 90% of the thread appeared to stem from this fear, uncertainty, and doubt, based on this misunderstanding, although more precisely worded, what I actually meant was that 90% of the *objections* raised to Nick's proposal were based on the aforementioned fear, uncertainty, and doubt -- i.e., that the objections had nothing to do with that which was being proposed. At one point this weekend, I had intended to write a detailed rebuttal to all of the points that had been raised, but by the time I had time to do so, the discussion was mostly settled and the issue mostly moot... but the impression that 90% of the original objections were misunderstanding-based remained, which led to my (perhaps poorly-phrased) 90% remark. All that being said, I'm not sure why you pounced on that side-comment in the first place; did you think I was personally insulting you or accusing you of something? ISTM that you are making an awfully big deal out of an incidental remark that had very little to do with the main point of the email, and framing it as though I am the one who is making a big deal of something. If you hadn't intervened, I don't see any reason why the conversation wouldn't have reached a peaceable conclusion, and am still puzzled as to why you felt the need to intervene. Your initial email began by disputing facts that you now appear to accept, in that you did not reply to any of my rebuttals to your assertions. But instead of admitting your assertions were in error, you're asserting that I'm the one who's confused. Well, I wasn't before, but I sure am *now*. ;-)

On 27 August 2013 17:01, Paul Moore <p.f.moore@gmail.com> wrote:

On 27 August 2013 00:15, PJ Eby <pje@telecommunity.com> wrote:

...

You pounced on a tiny piece of my email to Paul, in which I mainly expressed confusion about his statements about "cost". I was having trouble understanding what sort of "costs" he meant, and in subsequent discussion realized that it's because he and others appeared to have conflated setuptools' default-version issues, with Nick's proposal for handling non-default versions.

Note that I freely admit to *having* fear, uncertainty and doubt:

I feared that Nick's proposal would impact users who just wanted to use default versions. I was wrong, no issue, but I was concerned. I was uncertain as to what Nick meant by "pkg_resources compatible". This has now been explained, thanks, but I wasn't sure. I doubted that I had the full picture and I was going to investigate. Others provided extra information so I didn't need to do so myself, but I had questions that needed to be answered initially.

I think it was partly my fault, too. While I tried to emphasise that I was only interested in copying the pkg_resources back end layout for alternative versions in the initial post, the replies made me realise that (prior to this thread) PJE and Jason were probably the only other current distutils-sig participants familiar enough with setuptools and pkg_resources to understand the distinction between that aspect, the default version handling and the activation API (and my familiarity is a recent thing - I only really started understanding pkg_resources properly in the last couple of days while trying to fix a bug I reported a while back). While I haven't figured out how to fix the bug yet, I learned enough to figure out how to design a next generation alternative version mechanism that pkg_resources should be able to support, so I'm still calling that a win :) Just to scare people though... I did come up with a potentially decent use case for .pth files: they're actually a reasonable solution for sharing distributions between virtual environments in a way that works cross platform and on all currently used Python versions. Say you want to let virtual environments choose between "latest CherryPy 2" and "latest Cherry Py 3". Install CherryPy2 into a directory called "/full/path/to/some-alt-versions-directory/CherryPy2" and 3 into "/full/path/to/some-alt-versions-directory/CherryPy3". Now you can say "use latest available CherryPy2" in your virtual environment by adding a CherryPy2.pth file with a single line containing: /full/path/to/some-alt-versions-directory/CherryPy2 And similarly for CherryPy3.pth (but not in the same virtual environment as CherryPy2.pth!): /full/path/to/some-alt-versions-directory/CherryPy3 Because this actually modifies sys.path inside the environment, it works for both imports *and* for finding distribution metadata. If you upgrade the version of CherryPy2 or CherryPy3, all virtual environments referencing those directories will see the upgraded version. Anything using a version of CherryPy installed directly into the environment will ignore it. For those playing along at home... this is similar to how the default version support in setuptools works. The difference is that using .pth files to implicitly append to sys.path in a single-application virtual environment is significantly less surprising than doing so in a shared Python installation :) Cheers, Nick. P.S. If anyone missed me mentioning why I keep picking on the CherryPy 2 vs 3 migration, it's the actual parallel installation case that we have to deal with for beaker-project.org. Debugging some issues with that is what forced me to start learning how the multi-version support in pkg_resources actually works :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan <ncoghlan <at> gmail.com> writes:

Just to scare people though... I did come up with a potentially decent use case for .pth files: they're actually a reasonable solution for sharing distributions between virtual environments in a way that works cross platform and on all currently used Python versions. Say you want

Right, and ISTM it also enables a useful subset of "setup.py develop" functionality where the .pth acts analogously to an .egg-link - the referenced project becomes importable while still being editable, though its headers, scripts and data are not installed, nor does it appear on a list of installed distributions. Regards, Vinay Sajip

On 28 Aug 2013 01:25, "Tres Seaver" <tseaver@palladion.com> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 08/27/2013 05:00 AM, Nick Coghlan wrote:

...

PJE and Jason were probably the only other current distutils-sig participants familiar enough with setuptools and pkg_resources to understand the distinction between that aspect, the default version handling and the activation API

There are lots of folks here who have been building tooling on top of eggs for almost a decade now. Perhaps those who *do* grok how that stuff works (I'd be willing to guess a lot more than the three of you and myself) weren't alarmed by you proposal. Oophobia is not ubiquitous. :)

Ah, ye olde "Usenet nod" syndrome, the internet's ever-present friend skewing our perception of mailing list feedback :) Cheers, Nick.

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlIcxDEACgkQ+gerLs4ltQ4qCgCfQKtjEAQkx7XnkQS8A8Q767E6 lnwAoNJAcSDTbN6I1DW2DZAzC3lMvolQ =AeYU -----END PGP SIGNATURE-----

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Tue, Aug 27, 2013 at 3:01 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 27 August 2013 00:15, PJ Eby <pje@telecommunity.com> wrote: None of these things is wrong. It is *spreading* FUD (and in particular, doing so cynically to undermine a proposal) that is wrong, and I hope I didn't do that - I certainly did not intend to and I'm a bit unhappy about the implication that I might have.

Sorry for the implication; it was not intended. I did not think you had any intent to make other people share your doubts or had any desire to shoot down the proposal. As I said, the real intent of my (clearly, in retrospect, very poorly-worded) side-remark was that I thought 90% of the objections to Nick's proposals were based on fear, uncertainty, and doubt rather than any actual issues with the proposals themselves.

On Aug 25, 2013, at 4:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Anyway, I like Paul's suggestion of defining a specific runtime format for this, even if it's just "wheel layout plus a RECORD file". I'm currently thinking of using the ".dist" suffix, matching the existing egg vs egg-info naming convention.

It seems to me the easiest thing to do is just continue using eggs for this feature for now especially if the proposal is just standardizing what eggs do and doesn't offer any benefits besides standardization. That gets you all the benefits sans standardization and doesn't spend time putting a PEP through (and all the back and forth that entails) for something that already works when we can spend the time on stuff that still needs actual design work. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sun, Aug 25, 2013 at 5:21 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 26 Aug 2013 07:00, "Donald Stufft" <donald@stufft.io> wrote:

...

On Aug 25, 2013, at 4:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

Anyway, I like Paul's suggestion of defining a specific runtime format for this, even if it's just "wheel layout plus a RECORD file". I'm currently thinking of using the ".dist" suffix, matching the existing egg vs egg-info naming convention.

It seems to me the easiest thing to do is just continue using eggs for this feature for now especially if the proposal is just standardizing what eggs do and doesn't offer any benefits besides standardization. That gets you all the benefits sans standardization and doesn't spend time putting a PEP through (and all the back and forth that entails) for something that already works when we can spend the time on stuff that still needs actual design work.

Egg based multi-version installs still suffer from the problem of lacking a RECORD file so you need an external tool to manage them properly.

Well, I'd argue that eggs are effectively also records. You can find out what's installed by simply looking at the names in whatever directory you put eggs. The harder part, of course, is deciding when an egg is no longer needed. I assume the RECORD file doesn't address that either. Note that with multi-version support, uninstalling things is an optimization, not a necessity. The only harm a never-uninstalled egg does is take up space and maybe make tools that scan for what's installed take more time. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On 25 August 2013 21:02, Paul Moore <p.f.moore@gmail.com> wrote:

Essentially, that is my question - and I'd like to know what Nick's proposal is here because I am not happy with the existing pkg_resources solution of using .pth files. I know there's a new feature being discussed on import-sig which may replace the need for pth files, but I don't want to see pth files being needed if that isn't available. Specific reasons I have for this:

1. The extra sys.path entries that get added (not for performance reasons, but because having a long sys.path is difficult for the interactive user to interpret when trying to see what's going on). 2. The whole "if I'm not in a site packages directory I won't work" issue 3. Questions of where things go on sys.path (if it depends on the order you "declare" things, it gets nasty when there are name clashes)

I'm not proposing copying over any of the implicit .pth file magic. As far as I am aware, that's unrelated to the explicit multi-versioning feature - I believe it's only needed if you want to install something as an egg archive or subdirectory *and* have it available for import by default. The namespace package related .pth files are also unrelated. Explicit multi-versioning is different - you just install the directory, and you can't import from it by default. However, pkg_resources can be used to activate it if it is needed to satisfy a runtime requirement (or a dependency of a runtime requirement). If you don't import pkg_resources, none of the parallel installed versions will ever activate - you'll just get the version that is on sys.path by default.

The other thing I want to understand is how things would work if, for example, I wanted to use CherryPy 3 99% of the time, but occasionally needed CherryPy 2. Could I install CherryPy 3 as a normal (not multi-versioned) package, and then override it when needed? So only applications needing CherryPy 2 had to declare a version requirement?

Yep, this is exactly how Fedora & EPEL use it. CherryPy 3 is installed as normal in site-packages (with an adjacent egg-info directory), while CherryPy 2 is installed as an egg file containing the earlier version of the module and the relevant metadata.

More generally, how does a project choose whether to use run-time multi-versioning, vs metadata declaring a dependency for install time? Why would I ever write code that uses a run-time dependency on CherryPy 2, rather than saying in my metadata "requires: CherryPy = 2"? Or conversely why would I ever declare an install-time dependency rather than declaring my requirements at runtime "because that's more flexible"?

For Beaker, we use runtime dependencies because we build system packages that run in the system Python, but need to run on RHEL 6 (CherryPy 2 as default) and also on recent versions of Fedora (CherryPy 3 as default, CherryPy 2 available for multi-version import). We also use it in our doc build scripts to handle the different versions of Sphinx (Sphinx 1.x as default in Fedora, but only available as a multi-version import on RHEL 6). The reason to declare install time dependencies as well is that pkg_resources works recursively - the dependencies of any distribution you activate will be checked to ensure you're importing a consistent set of packages into your application. If you use virtual environments to create isolated stacks for each application, these issues don't come up. On the other hand, if you're trying to make thousands of packages play nice in a single Python installation (as Linux distros do), then you need to deal with the "parallel installation of mutually incompatible versions" problem. The current Linux distro solution is to use pkg_resources, and having been working with it for several months now, while I still think there are some significant rough edges in the way the pkg_resources front end works, I'm happy that the *file layout* the explicit multi-versioning feature uses is a decent one that should translate to wheel files fairly easily. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 August 2013 14:07, Donald Stufft <donald@stufft.io> wrote:

As far as I can tell there's nothing preventing Installing a Wheel *into* and .egg directory which will give you exactly the same situation as you have today without needing to do *anything* except make a tool that will install a Wheel into an .egg directory. This solves your immediate desire without hanging more things onto Wheel.

I think that relating this to wheels is a mistake. As Donald says, wheels are just a distribution format. Looking at this from a different perspective, I don't see any immediate issue with being able to install distributions outside of site-packages. That's what pip's --target option does. It seems like the multi-versioning support that is being discussed is nothing much more than a means of managing such "not on site-packages" directory in a way that allows them to be added to sys.path at runtime. On that basis, I don't have an issue with the idea. It's supported in pip (via --target) already, and the runtime aspects can be handled in pkg_resources or wherever without affecting me. If, on the other hand, there is a proposal in here to change pip, then (a) I'd like to see what the explicit proposal is before commenting, (b) it should apply equally to all means by which you can install using pip (wheel and sdist - AFAIK --target has no meaning for --develop installs, and the same should be true here), and (c) it should not need any changes to the wheel format (again, I see no indication that it does, but I'd like to see that explicitly stated). The key point here is that I *don't* want wheel associated with all of the extra baggage that goes along with the egg concept. Even putting wheels on sys.path gives me a nervous feeling that we're reinventing eggs. I feel strongly that wheel is inventing bdist_simple rather than egg, and should continue to do so. As regards Nick's proposal: 1. "An option (or options) to pip, telling it to just drop a wheel file (or the unpacked contents of the wheel as a directory) into site-packages instead of installing the distribution directly as the default version." Well, if it's in site-packages, it *is* importable/installed. That's how site-packages works. So I don't understand what "instead of installing" means. But if --target does what you want then go for it. If you want something different then I think it's likely a bad idea, but I'd need details of how it would work to be sure. 2. "Support for the ".whl" filename format and internal layout in pkg_resources". No, very definitely -1. The wheel format is a *distribution* format and pkg_resources is a *runtime* mechanism. Misxing the two is the key mistake (in my mind) that setuptools made, and we do not want to do so again. If you need a multiversion runtime layout for pkg_resources, then let's define it in a PEP, and write installers to put packages into that format if we need to. But let's not just reuse wheel as that format - that was not its intent. If a distribution format better than wheel comes along in the future, we should be able to replace it without needing to break everyone's runtime code because pkg_resources doesn't know wbout it. Paul

On 25 August 2013 15:02, Donald Stufft <donald@stufft.io> wrote:

On Aug 25, 2013, at 10:00 AM, Paul Moore <p.f.moore@gmail.com> wrote:

As regards Nick's proposal:

1. "An option (or options) to pip, telling it to just drop a wheel file (or the unpacked contents of the wheel as a directory) into site-packages instead of installing the distribution directly as the default version." Well, if it's in site-packages, it *is* importable/installed. That's how site-packages works. So I don't understand what "instead of installing" means. But if --target does what you want then go for it. If you want something different then I think it's likely a bad idea, but I'd need details of how it would work to be sure.

As I understand it Nick means to take the .whl unzip it into a folder named foo-whatever….whl and put that folder into site-packages. Basically the exact same structure as happens with .egg folders except with Wheels.

OK, I get that. But I want to avoid referring to it as "wheel format" because that's what eggs did, having 3 distinct formats, all used for subtly different things and it meant that people had a confused view of "what an egg was". I'd rather not promote the same confusion for wheels. Thanks for the various comments that have been made. I think I'm clear where I stand now - I have no objection to the idea of the proposal, but (1) I'd rather the on-disk format was clearly distinguished from wheels as it's a runtime format rather than a distribution format, (2) I don't really want to add an option to pip to install in this format, but if we do it should work for sdists as well as wheels (notwithstanding any longer-term goal to deprecate installing from sdists) and (3) I don't want to see adding wheel files to sys.path as ever becoming a mainstream way of working (that's what "pkg_resources supports the whl format" means to me - which is why I want to see the unpacked format referred to as something other than "wheel"). Paul