On Fri, Mar 29, 2013 at 4:50 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sat, Mar 30, 2013 at 6:42 AM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Mar 29, 2013 at 3:55 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
No, mutating sys.path for versioned imports is a broken design. You end up with two possibilities:
* If you append, then you can't override modules that have a default version available on sys.path. This is not an acceptable restriction, which is why pkg_resources doesn't do it that way * If you prepend, then you have the existing pkg_resources failure mode where it can accidentally shadow more modules than intended. This is a nightmare to debug when it goes wrong (it took me months to realise this was why a system install of the main application I work on was shadowing the version in source checkout when running the test suite or building the documentation).
The correct way to do it is with a path hook that processes a special "<versioned-packages>" marker label in sys.path (probably placed after the standard library but before site-packages by default). Any mounted directories would be tracked by that path hook, but never included directly in sys.path itself.
How is that different from replacing "<versioned-packages>" with the path of the versioned package being added?
You don't lose the place where you want the inserts to happen. Without the marker, you end up having to come up with a heuristic for "make insertions here" and that gets messy as you modify the path (particularly since other code may also modify the path without your involvement).
But at least you can tell exactly what the order is by inspecting sys.path.
Using a path hook to say "process these additional path entries here" cleans all that up and lets you precisely control the relative precedence of the additions as a group, without needing to care about their contents, or the other contents of sys.path.
Then can't you just bracket the entries with "<start-versioned>" and "<end-versioned>" then? ;-) (That would actually work right now without a path hook, since strings that refer to non-existent paths are optimized away even in Python 2.5+.) Also, btw, pkg_resources's sys.path munging is far more aggressive than anything with wheels needs to be, because pkg_resources is *always* working with individual eggs, not plain install directories. If you are only using standalone wheels to handle alternate versions, then you *definitely* want those standalone wheels to override other things, so a strategy of always placing them immediately before their containing directory is actually safe. (In effect, treating the containing directory as the insertion marker.) So, if the standalone wheel is in site-packages, then activating it would place it just ahead of site-packages -- i.e., the same place you're saying it should go. And as I believe I mentioned before, a single marker for insertion points doesn't address the user site-packages, app directory or app plugin directories, etc. AFAICT your proposal only addresses the needs of system packages, and punts on everything else. At the least, may I suggest that instead of using a marker, if you must use a path hook, simply install the path hook as a wrapper for a specific directory (e.g. site-packages), and let it process insertions for *that directory only*, rather than having a single global notion of "all the versioned packages".