[Python-3000] Changing the import machinery

[Guido van Rossum]

It must not be my day. I don't understand anything you're saying.

On 4/20/06, Ian Bicking <ianb at colorstudy.com> wrote:
> Cleaning up import stuff would be excellent.  Time spent debugging
> imports is time wasted, but it happens all too often.
>
> I would argue against any list of loaders, or list of anything.  That
> builds ambiguity directly into the system.  Without a list, if you want
> ambiguity, a container loader could search a list of loaders.  Or if you
> want to avoid all ambiguity, you could have a loader that was more picky.

What ambiguity are you talking of here?

> Setuptools version-based eager loading

Can you explain this? I don't actually know much of what setuptools
does (nor does almost anyone else it seems :-) -- I have no frame of
reference to understand this.

> can give you some confidence that
> everything you think you need is installed, but can't provide much
> confidence that everything you *think you are using* is actually what
> you are using.  That is, it's been fairly common in my own experience
> for me to realize some other version of a package is being loaded than
> what I thought, or I spend an inordinant amount of time tweaking
> requirements to get the right version of a package from one place
> without affecting another package that needs a different version (or
> perhaps is run with a different sys.path).

Is this a setuptools thing or a Python thing? I don't understand what
you mean by "tweaking requirements".

> But, back to more concrete use cases:
>
> * Right now it is pretty hard to set up an environment where changes
> elsewhere on the system can't leak in.  That is, installation of
> something in a system-wide site-packages can cause problems everywhere
> on the system, and even if you try to avoid these it is quite hard.  One
> strategy is setting up an entirely different environment (aka, a
> different prefix); this is heavy-feeling.  Another is avoiding site.py
> or using a custom site.py, but the tool support is iffy for that.

I have a sense of deja vu -- somehow I feel we've been discussing this
some years ago. Are you sure you didn't post this same message around
the time we were discussing PEP 302?

Seriously, can you provide an example of the sort of thing you're
thinking of? Again, is this setuptools specific or not? Is the problem
caused by users setting their PYTHONPATH a certain way? (If so, how?)

> There's really just no good way to tell Python to leave well enough alone.

That's a rather general and vague remark. What do you mean?

> * Relatedly, installation and management when you don't have root or the
> cooperation of root can be hard.  I think the answer to this is much
> like the isolated environment, but the use case is fairly different.

This I understand. What do you suggest we do to improve the situation?

> * Configuration about where to install things (e.g., distutils.cfg) is
> separate from information about where to look for things (sys.path).
> These should form a consistent description of the environment, but
> currently they are disassociated from each other.

I didn't even know we have a distutils.cfg.

I am beginning to believe that you're talking about issues with
installation rather than issues in the import machinery proper. But
maybe if you clear up some of my earlier confusion I'll understand
better.

> * Any kind of automatic installation is difficult, because you can't
> really count on being able to install even the most inoccuous package in
> an automated way.  There's too many manual overrides, and too many
> redundant options, and few people actually have their system set up to
> work without tweaking these options through the command line or other
> feedback.

Again that's rather vague and general. Please be specific. "Build a
better mousetrap" can't be our requirements spec.

> * Personally I've settled on putting everything I make into a Python
> package that is distutils-installable.  But many people don't.  I'm not
> sure if this is just because the tools seem too hard, or the namespaces
> feel too deep, or all the documentation starts without using packages,
> or having '.' (sometimes) on sys.path does it, or what.  I'd rather
> there be consistent practices; but the consistent practices that we have
> that actually work (setup.py scripts and packages) are too heavy for a
> lot of people.

To the contrary, if your entire code fits in a single .py file, I'm
not sure what distutils even buys you, and it sure costs a lot of
complexity. FWIW Google's internal build system agrees with you and
doesn't like 3rd party code that's not in a package.

I often have a hard time finding the source code in a distutils-based
package that I have unpacked but am hesitant to install just to read
the source code; there seems to be no consistent convention as to
where the source is.

> * People are seriously planning on using relative imports to manage
> their packages, and so an application will be 'installed' by putting it
> into another package.  Presumably unpacking it directly in some other
> package's directory.  Who knows what the version control plans are, or
> maintenance, or whatever.  I think it's a bad idea.  We need to give
> these people a carrot to keep them from doing this.

You seem to have version control on your brain. (Not that that's a bad
thing, but most people don't -- it's a specific way of looking at
things.) I'm not entirely sure I understand what bad practice you're
describing here; and what do you propose to do instead? KSurely not
killing relative import?

> * Right now namespace packages are hard.  That is, a Python package
> (like 'zope') that is used by several distutils packages.  I almost feel
> like namespace packages should be installed flat, like 'zope-interface'
> and 'zope-tal', and turned into namespaces dynamically.

Fine. That's the most advanced usage there is. I'd be happy if we
solved all other problems first.

> * The module layout is used both as an API and as an internal factoring
> of the code.  If you want to refactor the code you break the API.
> Personally I really like the strong connection between imports and code
>   location, and appreciate how easily I can find code as a result.  But
> setting up the scaffolding and warnings necessary when moving a module
> can be tiresome.

I don't expect there's a silver bullet here; backwards compatibility
hacks are always tiresome.

> * Circular imports should fail more nicely.  Everyone suffers this at
> some time; maybe it can't be fixed, but at least it should be clear
> what's happening.

It probably can't be fixed (or do you see a fix?). Do you mean it
should be easier to debug, or do you mean it should be explained
better how things work?

> * You can't really tell if "from foo import bar" can be written as
> "import foo; bar = foo.bar", because it works if foo contains bar, but
> not if foo is a package and bar is a module in that package.

And is that bad? "from foo import bar" means what it means. Nobody
said the other code is equivalent. If you know what foo is you will
know whether the other code is equivalent; isn't that enough?

> Well... I think that's maybe half way through the list of issues I have,
> but this email is already much too long.

And still not clear enough. It's difficult to communicate when one
party has spent a long time exploring the issues and another is just
starting; this time I seem to be on the receiving end (== haven't
spent a lot of time).

--
--Guido van Rossum (home page: http://www.python.org/~guido/)