[Distutils] PEP 438, pip and --allow-external
Donald Stufft
donald at stufft.io
Mon May 12 22:37:20 CEST 2014
On May 12, 2014, at 4:33 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 12.05.2014 22:31, Donald Stufft wrote:
>>
>> On May 12, 2014, at 3:58 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> On 12.05.2014 15:58, Paul Moore wrote:
>>>> On 12 May 2014 13:28, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> So, some questions:
>>>>>>
>>>>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally
>>>>>> hosted[1], and if so, what is he asking for? Automatic download with
>>>>>> no need for opt-in of unverifiable external downloads? That seems
>>>>>> pretty much in conflict with the whole intent of PEP 438.
>>>>>
>>>>> What we are implementing is a proposal that I brought up before
>>>>> PEP 438 was put in place:
>>>>>
>>>>> Instead of linking directly to all packages, we put up a verifiable
>>>>> link to an index page with verifiable links, with the net effect
>>>>> being that tools can verify the whole chain.
>>>>
>>>> Thanks for clarifying. My main concern is that people do actually
>>>> understand the requirements for "safe" external hosting (I discovered
>>>> that I certainly didn't - it's quite complex!). From what you say, you
>>>> do understand the requirements but have chosen not to follow them at
>>>> this time. That's fine, I'm not trying to debate the validity of your
>>>> choice. But in the context of PEP 438, that means that users of the
>>>> egenix downloads will have to opt into unverifiable downloads in order
>>>> to install using pip. We're working towards improving the user
>>>> experience for end users who need to install unverifiable downloads -
>>>> I'd expect that one consequence of this will be that we'll be better
>>>> able to communicate the situation to those users, which will reduce
>>>> the confusion.
>>>
>>> If it helps convince you that allowing verifiable external
>>> links per default is a good thing for user experience, we will
>>> register the distribution file download URLs with the PyPI
>>> web API.
>>>
>>>> I don't know if you're suggesting that your proposal is still
>>>> something that we should be looking at implementing. I'm happy to
>>>> discuss that, but it should probably be a separate thread.
>>>
>>> You can think of it as per-package PyPI-style simple index
>>> that's registered with PyPI in a secure way (via the check sum
>>> hash).
>>>
>>> There's most certainly room for improvement and I'm open for
>>> discussions.
>>>
>>>>> since shifted focus to working on a web installer which solves
>>>>> multiple problems at once:
>>>> [...]
>>>>> With the web installer, we'd just have to upload one file
>>>>> per release.
>>>>
>>>> I'm not quite sure how you expect this will work, but it's probably
>>>> important that you get involved with the various packaging PEPs. The
>>>> only way I can see such a solution working with pip would be if you
>>>> have a customised setup.py.
>>>
>>> Yep, since that's the way installers (not only pip) work when
>>> they see a source distribution.
>>>
>>>> As the general trend is towards binary
>>>> installs using wheels, with pip eventually deprecating sdist installs
>>>> and running setup.py directly, we will likely miss your use case
>>>> unless you get involved in those discussions (they are on the back
>>>> burner at the moment, but will likely resurface at some point -
>>>> there's currently a work-in-progress PR for pip to use a "wheel
>>>> cache", where the normal install route will be "pip wheel; pip install
>>>> <the generated wheel>", bypassing setup.py install totally).
>>>
>>> Binary installs are nice, but they are not the answer to everything
>>> and no matter how much meta data you put into static files,
>>> there will always be cases where that meta data description just
>>> doesn't include those bits you would need to complete the setup,
>>> esp. for packages with C extensions and more complex external
>>> dependencies or setup requirements. (*)
>>>
>>> The setup.py interface makes all this possible, which is why so
>>> many Python packages use it to configure themselves automatically.
>>>
>>> Deprecating this interface would make some distributions impossible
>>> to install without manual user intervention and we'd be back to the
>>> Makefile.pre.in days.
>>>
>>> I don't think that's a good idea. It still is a very good idea
>>> to make more meta data available in static non-executable form
>>> in order to simplify finding packages, determining
>>> dependencies, features, enhancing the PyPI UI, and for
>>> deciding which distribution file to download and install.
>>>
>>> This can be generated by setup.py as part of the build process
>>> and made available to PyPI during package release registration
>>> (much like it is now, only in extended form).
>>>
>>> (*) This does work if you are only targeting a few select systems and
>>> system versions, but the Python user base out just has too many
>>> diverse setups to be able to address them all to be able to
>>> completely drop setup.py.
>>
>> This is slightly confusing but pip will always be able to go from an sdist to
>> an installed system. It'll just build a Wheel first and then install the Wheel
>> (at least that's the idea). This is a sort of vague idea right now but it's the
>> direction we want to go in.
>
> Ah, so this is just a misunderstanding on my part then. I thought
> Paul was saying that having pip run setup.py will go away.
>
> Thanks for the clarification,
>
I should expand on that answer, that sdist 2.0 will ideally include static
metadata but it won't be a static build system. Things like name, version,
dependencies etc will be static, but building will be done by executing some
code.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140512/05a17f5f/attachment.sig>
More information about the Distutils-SIG
mailing list