eggs and PYTHONPATH (cont'd from peak list...)
Hi - I have some further thoughts on the PYTHONPATH topic Phillip and I were discussing on the PEAK list a few days ago. On Jun 28, 2006, at 4:09 PM, Phillip J. Eby wrote:
Eggs always override non-eggs, because eggs can be controlled in ways that non-egg paths can't. The order is (roughly):
eggs on PYTHONPATH eggs in site-packages non-eggs on PYTHONPATH non-eggs in site-packages
I would propose the order be this: eggs on PYTHONPATH non-eggs on PYTHONPATH eggs in site-packages non-eggs in site-packages It is possible that a non-egg install might be named in PYTHONPATH for a package for which newer versions that are setuptools/egg- enabled are installed system-wide. However, I cant think of a possible scenario where a path would explicitly exist in PYTHONPATH, non-egg or egg, where the user would still like the system-wide installation to take precedence, regardless of versioning or anything else. Otherwise why put the path on PYTHONPATH ? (Though Im guessing there is such a scenario, id just like to see what it is). I know that eggs are more powerful than non-egg paths, allowing precedence to take place based on versioning, which is obviously extremely useful; but I think the idea of PYTHONPATH is that system- wide library paths can be explicitly overridden regardless of any versioning logic taking place either within site-packages or elsewhere. For example, if a particular package on a multi-user shell account is upgraded systemwide from its previously non-egg version to a setuptools/egg version, then any users who have applications running on a custom PYTHONPATH setup with dependencies on their locally- installed, non-egg versions will now have a broken install; theyd have to log in, read all the setuptools docs to figure out whats going on, and do the "virtual python" setup in order to fix things. It seems that setuptools redefines the intention of PYTHONPATH from being a "runtime path determination" variable to an "installation path determination" variable; why repurpose PYTHONPATH in this way, especially when many packages are going to exist in egg/non-egg formats? why not use another variable such as SETUPTOOLSPATH ? Im sure theres other reasons im missing for why its the way it is, so I apologize for harping on this. Also was wondering if anyone else had this same question. - mike
It seems that setuptools redefines the intention of PYTHONPATH from being a "runtime path determination" variable to an "installation path determination" variable; why repurpose PYTHONPATH in this way,
I've always seen it like this: * PYTHONPATH is determined by the shell environment * .pth files are determined at install time and or (re)configuration time. * .pth files are merged into PYTHONPATH & sys.path at interpreter startup * sys.path is runtime determined And assumed: the only programs that 'casually' do os.getenv('PYTHONPATH') are shell scripts authored in python rather than bash. And as such are on PATH. I missed the start of this thread (on other list) but it looks like setuptools *has* to get involved in the last three points. It can not know the results of non egg package startup/runtime sys.path configuration. It is adding .pth files and manipulating sys.path in order to do its version dance. setuptools has to pick a precedence for everything, versioned eggs, versioned setuptools distributions, plain old python packages found on sys.path. It looks to me more like setuptools forces version -1.0.0 on everything that is not an egg, including things identified by a customised PYTHONPATH. Phillip said:
Eggs always override non-eggs, because eggs can be controlled in ways that non-egg paths can't.
This sounds like a choice not a reason. Isn't it just as reasonable to say PYTHONPATH can be controlled in ways egg paths can't. I guess I'm lining up with mike here. On the face of it, it does not seem reasonable for setuptools to 'guess' the version of a plain python package on sys.path or otherwise subvert the traditional "use this package damn it" behaviour of PYTHONPATH. But I don't understand this stuff enough to state it more strongly than that. Mike said:
I would propose the order be this:
eggs on PYTHONPATH non-eggs on PYTHONPATH eggs in site-packages non-eggs in site-packages
<snip>
However, I cant think of a possible scenario where a path would explicitly exist in PYTHONPATH, non-egg or egg, where the user would still like the system-wide installation to take precedence
I can't either. mike said:
For example, if a particular package on a multi-user shell account is upgraded systemwide from its previously non-egg version to a setuptools/egg version, then any users who have applications running on a custom PYTHONPATH setup with dependencies on their locally- installed, non-egg versions will now have a broken install; theyd have to log in, read all the setuptools docs to figure out whats going on, and do the "virtual python" setup in order to fix things
Is this all from the perspective of resolving versions using setuptools ? If so I think: - It makes no sense for the system vendor to start using eggs and *not* also bundle setuptools in this format. - all non setuptools python packages are effectively *not versioned* on this system - The system vendor is making a pretty fundamental change to there distribution versioning / packaging mechanics in doing this. They should not be doing this as an interim update to an existing release. - I always expect my scripts custom config to be broken by distro upgrade. Are there any python packages widely used by non technical users that direct their users to perform these sorts of customisations to PYTHONPATH ? (I'm assuming egg.link aka develop is an 'egg' package wrt this discussion and so just works tm) But I think this issue is distinct from how non setuptools / non version packages interact with the setuptools world. Things will look very different when: setuptools becomes part of the python standard distribution AND system vendors catch up with that release of python. Cheers, Robin
participants (2)
-
Michael Bayer -
Robin Bryce