[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

Paul Moore p.f.moore at gmail.com
Thu Apr 16 20:33:47 CEST 2015

On 16 April 2015 at 17:58, Chris Barker <chris.barker at noaa.gov> wrote:
> We have the same problem with optional dependencies.
> For instance, for iPython to work, you don't need much. but if you want the
> ipython notebook to work, you need tornado, zeromq, who knows what else. But
> people want it to just work -- and just work be default, so you want all
> that optional stuff to go in by default.

But none of those are installed by default with ipython - they are
covered by extras. If you want them, you ask for them. Thanks to
extras, ipython offers a nice shortcut for you - pip install
ipython[all] - but you still have to ask for them.

> I expect this is the same with wheel installer extensions. To use your
> example, for instance. People want to do:
> pip install sphinx
> and then have the sphinx-quickstart utility ready to go. by default. So
> scripts need to be installed by default.

Yes, the script wrapper extension is a much better example of a
"nobody would ever want this off" extension. But is it really
"required"? The distribution would work fine if scripts weren't
installed. My understanding of the required/optional distinction in
the PEP is that an extension is required if the installation would be
*broken* if that extension wasn't supported. And having to type
"python -m something" rather than just "something" isn't broken, it's
just an inconvenience.

In practice, I'd assume script wrappers would be an extension built
into pip, so it would always be available. But that's different from
"required". Installing via "wheel install" (which doesn't support
generating script wrappers) still works.

Actually, I just checked PEP 459, which says of the "python.commands"
extension: "Build tools SHOULD mark this extension as requiring
handling by installers". So I stand corrected - script wrappers should
be considered mandatory. In practice, though, what that means is that
pip will be fine (as it'll have support built in) and wheel will be a
non-compliant installer (as it neither supports generating wrappers,
nor does it give an error when asked to create them - maybe an error
will get added, but I doubt it as wheel install isn't intended to be a
full installer). I've no idea at this point what distil or
easy_install will do, much less any other installers out there.

> The trade-off between convenience and control/security is tough.

Certainly. But that's about what is available by default, and how the
installer (pip) handles the user interface for "this package says that
it gives you some extra functionality if you have extensions X, Y, and
Z". There's no convenience or UI implication with required extensions
- if they aren't available, the installer refuses to install the
distribution. Simple as that. Maybe pip could try to locate and
download mandatory extensions before deciding whether to fail the
install, but the package metadata doesn't say how to find such
installer plugins (and it *can't* - because the plugin would be
different for pip, easy-install, distil or wheel, so all it can say is
"I need plugin foo" and the installer has to know the rest). That's an
installation program quality of implementation issue though.

Given that a random project could add metadata

    "extensions: {
        foocorp.randomstuff: {
            "installer_must_handle": true,
            "foocorp.localdata": "Some random stuff"

there is no way that pip has any means of discovering where to get
code to handle the "foocorp.randomstuff" extension from. So in the
general case, auto-installing required extension support just isn't
possible. At best, pip could have a registry of plugins that support
standard extensions (i.e. those covered by a PEP) but I'd expect that
we'd just build them into pip (as we don't have a plugin interface at
the moment).

>> Based on the above, it's possibly valid to allow "required" extensions
>> to be auto-installed. It *is* a vector for unexpected code execution,
>> but maybe that's OK.
> If even required extensions aren't auto installed, then we can just toss out
> the whole idea of automatic dependency management. (which I personally
> wouldn't mind, actually, but I'm weird that way)

I disagree. It's no different conceptually than the fact that if you
don't have a C compiler, you can't install a package that contains C
code and only comes in sdist format today. The UI in pip for noticing
and saying "you need a C compiler" is terrible (you get a build error
which might mention that you don't have the compiler, if you're lucky
:-)). And yet people survive. So a clear error saying "package X needs
a handler for extension Y to install" is a major improvement over the
current situation. (I know C compilers are build-step and extensions
are install-step, but right now the user experience doesn't clearly
distinguish these, so the analogy holds).

Whether users want pip to go one step further and auto-install the
plugin is unknown at this point. So far, it seems that the only people
who have expressed an opinion (you and I) aren't particularly pushing
for auto-installing (unless I misinterpreted your "which I personally
wouldn't mind" comment). For a proper view, we'd need a concrete
example of a package with a specific required extension, that pip was
unlikely to include out of the box. Or we could just not worry for
now, and wait to see what feedback we got from a non-automatic initial
implementation in real use.

> But maybe we need some "real" use cases to talk about -- I agree with others
> in this thread that the Start menu isn't a good example.


To focus discussion, I think we need

- A credible "required" extension (python.constraints or
python.commands from PEP 459)
- A credible "required" extension that pip wouldn't provide builtin support for
- A credible "optional" extension (most of the other extensions in PEP
459, for example exports)
- A credible "optional" extension that pip wouldn't provide builtin support for

I've separated out things that pip wouldn't provide builtin support
for, because those are the only ones where there's a real question
about "what do we do if support isn't available", at least from a pip
point of view. In practice, that probably means "not defined in an
accepted PEP" (I'd expect pip to build in support for standardised

By the way. I just did a check through PEPs 426 and 459. Neither one
currently defines a "postinstall script" metadata item or extension,
which is interesting given that this discussion started from the
question of how postinstall actions would be supported. There *have*
been discussions in the past, and I could have sworn they ended up in
a PEP somewhere, but maybe I was wrong.


More information about the Distutils-SIG mailing list