[Python-3000] Changing the import machinery

[Ian Bicking]

Jim Jewett wrote:
> On 4/20/06, Ian Bicking <ianb at colorstudy.com> wrote:
> 
> 
>>With the ambiguity of sys.path, it's hard to statically determine if you
>>have all the requirements.
> 
> 
> [And, as you said elsewhere, whether it is the version you expected,
> and whether it will actually load or get skipped because of an earlier
> import]
> 
> Would you like the ability to specify precisely which file a requested
> module gets loaded from, and to load your own copy if the
> already-imported version has the wrong __file__ attribute?

Probably not, but I guess it would depend on the specifics.  In many 
cases it is not safe to load two versions of the same module into the 
same process.  Allowing for that would be a fairly major change to what 
it means to be a module, even if it's also a rather subtle change. 
Having accidentally imported a single module multiple times under 
different names, it's made me pretty uncomfortable with the idea.

That said, in some cases it is possible, and I do just that.  Like, if I 
want string.Template, I do:

try:
     from string import Template
except ImportError:
     from mypackage.backports.string24 import Template

Doing this in a more elegent or formalized fashion might be nice. 
Versioning the standard library should also probably be a bullet point 
in here somewhere -- I think it's an important aspect to keeping stdlib 
modules from being forceably put into an eternal freeze.


>>If you need to resolve a problem in a non-global manner, you can do so
>>with alternate installation locations and sys.path manipulation.
> 
> 
> If you need to resolve it in a non-global manner, doesn't that
> indicate that you're working with a specific application, which can
> safely run in (and modify the sys.path of) its own process?

I guess I see it as a deployment task, not an application development 
task, though it would be possible to address it as an application 
development task.

As I noted elsewhere in one of these emails, it's hard to safely remove 
items from sys.path, so it's hard for an application to fix up sys.path 
unless it gets its hands on it very early in the process.  When this 
goes wrong it can be very frustrating to debug.

For an application installed as a OS distribution (rpm, etc) I think 
this is absolutely the strategy they should take, because the 
distribution created in that situation is a merging of both development 
and deployment.  That's also what makes packaging work hard, so 
simplifying that would be nice.


>>Mostly the ability to set up a localized environment disassociated from
>>the global installation (except the stdlib).
> 
> 
> Are you saying that you would like to lock the global environment, so
> that you won't see anything except the original stdlib without an
> explicit change to your own application?

Yes.  "Locking" is pretty simple, really -- just don't put site-packages 
on the path, which in turn is accomplished by not running site.py.  So 
current convention already gets us most of the way there.  It's just too 
damn hard to get the rest of the way :(


> If so, is the problem just that so many existing applications put
> themselves on the global path, and [in a hosting situation] these may
> be installed by someone else, without your knowledge?

Yes, I think this is basically the problem.  And that "someone else" may 
be a past or future self, who didn't know the best way to install things 
or else forgot the best way.  Whenever I figure out a good set of 
command-line arguments to do what I want, I always forget later.  And 
certainly anyone else who comes along will be entirely lost.


> In theory, the import path could be controlled directly by each
> importing module, but in practice, I think that would lead to a bigger
> mess as soon as there were conflicts (= as soon as it starts to
> matter).

To a degree this is what Setuptools is doing, but indeed conflicts 
happen and can be hard to resolve.


>>I'm just noting that I think relative imports will
>>be used by people who want to avoid the current distutil process,
> 
> 
> Absolutely!
> 
> They also protect against version skew, which can be particularly
> important if you're relying on a patched version.
> 
> 
>>even though relative imports are not a good solution.
> 
> 
> Why not?  You won't get the benefit of automatic upgrades to
> components you rely upon, but you won't get the risk either.

For all the same reasons that importing based on __file__ is 
challenging.  It breaks the rules -- modules don't get a canonical name, 
they may exist multiple times in the same process, and a lot of the 
distinctions between packages and maintainers are covered up.

Sometimes it is fine -- like if you are backporting something, or you 
want to grab some small module without having to put in another 
requirement, maybe using svn:externals.  As a general means of reuse I 
don't think it's a good idea.


>>it's not like zope.interface is
>>magically more modular and elegant than zopeinterface.
> 
> 
> One of the objections to the python documentation is that it can be
> hard to find things if you don't already know where to look.  The
> module index is an obvious starting point, but is way too long,
> because it lists every module instead of just packages and independent
> modules.
> 
> Having to group on zope* rather than zope.* is a step backwards. 
> Looking at just the stdlib, are mailbox and mailcap related?  operator
> and optparse?  I don't need to know anything about Carbon.* to know
> that they're all related; if I have a summary of the Carbon package
> ("Mac only system bindings") I get to skip the whole batch.

I think this is a separate issue.  The ToC offers more of a 
categorization, but happens to be layed out in a way that is hard to 
scan.  The module index is easy to scan given a name, but hard given a 
target functionality.  Also, there's not enough editorial control -- for 
instance, to categorize deprecated modules differently from other 
modules.  Even the index should have editorial control IMHO.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org