[Distutils] PEP 438, pip and --allow-external

Donald Stufft donald at stufft.io
Mon May 12 22:37:20 CEST 2014


On May 12, 2014, at 4:33 PM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 12.05.2014 22:31, Donald Stufft wrote:
>> 
>> On May 12, 2014, at 3:58 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> 
>>> On 12.05.2014 15:58, Paul Moore wrote:
>>>> On 12 May 2014 13:28, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> So, some questions:
>>>>>> 
>>>>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally
>>>>>> hosted[1], and if so, what is he asking for? Automatic download with
>>>>>> no need for opt-in of unverifiable external downloads? That seems
>>>>>> pretty much in conflict with the whole intent of PEP 438.
>>>>> 
>>>>> What we are implementing is a proposal that I brought up before
>>>>> PEP 438 was put in place:
>>>>> 
>>>>> Instead of linking directly to all packages, we put up a verifiable
>>>>> link to an index page with verifiable links, with the net effect
>>>>> being that tools can verify the whole chain.
>>>> 
>>>> Thanks for clarifying. My main concern is that people do actually
>>>> understand the requirements for "safe" external hosting (I discovered
>>>> that I certainly didn't - it's quite complex!). From what you say, you
>>>> do understand the requirements but have chosen not to follow them at
>>>> this time. That's fine, I'm not trying to debate the validity of your
>>>> choice. But in the context of PEP 438, that means that users of the
>>>> egenix downloads will have to opt into unverifiable downloads in order
>>>> to install using pip. We're working towards improving the user
>>>> experience for end users who need to install unverifiable downloads -
>>>> I'd expect that one consequence of this will be that we'll be better
>>>> able to communicate the situation to those users, which will reduce
>>>> the confusion.
>>> 
>>> If it helps convince you that allowing verifiable external
>>> links per default is a good thing for user experience, we will
>>> register the distribution file download URLs with the PyPI
>>> web API.
>>> 
>>>> I don't know if you're suggesting that your proposal is still
>>>> something that we should be looking at implementing. I'm happy to
>>>> discuss that, but it should probably be a separate thread.
>>> 
>>> You can think of it as per-package PyPI-style simple index
>>> that's registered with PyPI in a secure way (via the check sum
>>> hash).
>>> 
>>> There's most certainly room for improvement and I'm open for
>>> discussions.
>>> 
>>>>> since shifted focus to working on a web installer which solves
>>>>> multiple problems at once:
>>>> [...]
>>>>> With the web installer, we'd just have to upload one file
>>>>> per release.
>>>> 
>>>> I'm not quite sure how you expect this will work, but it's probably
>>>> important that you get involved with the various packaging PEPs. The
>>>> only way I can see such a solution working with pip would be if you
>>>> have a customised setup.py.
>>> 
>>> Yep, since that's the way installers (not only pip) work when
>>> they see a source distribution.
>>> 
>>>> As the general trend is towards binary
>>>> installs using wheels, with pip eventually deprecating sdist installs
>>>> and running setup.py directly, we will likely miss your use case
>>>> unless you get involved in those discussions (they are on the back
>>>> burner at the moment, but will likely resurface at some point -
>>>> there's currently a work-in-progress PR for pip to use a "wheel
>>>> cache", where the normal install route will be "pip wheel; pip install
>>>> <the generated wheel>", bypassing setup.py install totally).
>>> 
>>> Binary installs are nice, but they are not the answer to everything
>>> and no matter how much meta data you put into static files,
>>> there will always be cases where that meta data description just
>>> doesn't include those bits you would need to complete the setup,
>>> esp. for packages with C extensions and more complex external
>>> dependencies or setup requirements. (*)
>>> 
>>> The setup.py interface makes all this possible, which is why so
>>> many Python packages use it to configure themselves automatically.
>>> 
>>> Deprecating this interface would make some distributions impossible
>>> to install without manual user intervention and we'd be back to the
>>> Makefile.pre.in days.
>>> 
>>> I don't think that's a good idea. It still is a very good idea
>>> to make more meta data available in static non-executable form
>>> in order to simplify finding packages, determining
>>> dependencies, features, enhancing the PyPI UI, and for
>>> deciding which distribution file to download and install.
>>> 
>>> This can be generated by setup.py as part of the build process
>>> and made available to PyPI during package release registration
>>> (much like it is now, only in extended form).
>>> 
>>> (*) This does work if you are only targeting a few select systems and
>>> system versions, but the Python user base out just has too many
>>> diverse setups to be able to address them all to be able to
>>> completely drop setup.py.
>> 
>> This is slightly confusing but pip will always be able to go from an sdist to
>> an installed system. It'll just build a Wheel first and then install the Wheel
>> (at least that's the idea). This is a sort of vague idea right now but it's the
>> direction we want to go in.
> 
> Ah, so this is just a misunderstanding on my part then. I thought
> Paul was saying that having pip run setup.py will go away.
> 
> Thanks for the clarification,
> 

I should expand on that answer, that sdist 2.0 will ideally include static
metadata but it won't be a static build system. Things like name, version,
dependencies etc will be static, but building will be done by executing some
code.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140512/05a17f5f/attachment.sig>


More information about the Distutils-SIG mailing list