[Distutils] PEP 438, pip and --allow-external

M.-A. Lemburg mal at egenix.com
Mon May 12 22:33:21 CEST 2014


On 12.05.2014 22:31, Donald Stufft wrote:
> 
> On May 12, 2014, at 3:58 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 12.05.2014 15:58, Paul Moore wrote:
>>> On 12 May 2014 13:28, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>> So, some questions:
>>>>>
>>>>> 1. Is MAL aware that egenix-mx-base is not verifiably externally
>>>>> hosted[1], and if so, what is he asking for? Automatic download with
>>>>> no need for opt-in of unverifiable external downloads? That seems
>>>>> pretty much in conflict with the whole intent of PEP 438.
>>>>
>>>> What we are implementing is a proposal that I brought up before
>>>> PEP 438 was put in place:
>>>>
>>>> Instead of linking directly to all packages, we put up a verifiable
>>>> link to an index page with verifiable links, with the net effect
>>>> being that tools can verify the whole chain.
>>>
>>> Thanks for clarifying. My main concern is that people do actually
>>> understand the requirements for "safe" external hosting (I discovered
>>> that I certainly didn't - it's quite complex!). From what you say, you
>>> do understand the requirements but have chosen not to follow them at
>>> this time. That's fine, I'm not trying to debate the validity of your
>>> choice. But in the context of PEP 438, that means that users of the
>>> egenix downloads will have to opt into unverifiable downloads in order
>>> to install using pip. We're working towards improving the user
>>> experience for end users who need to install unverifiable downloads -
>>> I'd expect that one consequence of this will be that we'll be better
>>> able to communicate the situation to those users, which will reduce
>>> the confusion.
>>
>> If it helps convince you that allowing verifiable external
>> links per default is a good thing for user experience, we will
>> register the distribution file download URLs with the PyPI
>> web API.
>>
>>> I don't know if you're suggesting that your proposal is still
>>> something that we should be looking at implementing. I'm happy to
>>> discuss that, but it should probably be a separate thread.
>>
>> You can think of it as per-package PyPI-style simple index
>> that's registered with PyPI in a secure way (via the check sum
>> hash).
>>
>> There's most certainly room for improvement and I'm open for
>> discussions.
>>
>>>> since shifted focus to working on a web installer which solves
>>>> multiple problems at once:
>>> [...]
>>>> With the web installer, we'd just have to upload one file
>>>> per release.
>>>
>>> I'm not quite sure how you expect this will work, but it's probably
>>> important that you get involved with the various packaging PEPs. The
>>> only way I can see such a solution working with pip would be if you
>>> have a customised setup.py.
>>
>> Yep, since that's the way installers (not only pip) work when
>> they see a source distribution.
>>
>>> As the general trend is towards binary
>>> installs using wheels, with pip eventually deprecating sdist installs
>>> and running setup.py directly, we will likely miss your use case
>>> unless you get involved in those discussions (they are on the back
>>> burner at the moment, but will likely resurface at some point -
>>> there's currently a work-in-progress PR for pip to use a "wheel
>>> cache", where the normal install route will be "pip wheel; pip install
>>> <the generated wheel>", bypassing setup.py install totally).
>>
>> Binary installs are nice, but they are not the answer to everything
>> and no matter how much meta data you put into static files,
>> there will always be cases where that meta data description just
>> doesn't include those bits you would need to complete the setup,
>> esp. for packages with C extensions and more complex external
>> dependencies or setup requirements. (*)
>>
>> The setup.py interface makes all this possible, which is why so
>> many Python packages use it to configure themselves automatically.
>>
>> Deprecating this interface would make some distributions impossible
>> to install without manual user intervention and we'd be back to the
>> Makefile.pre.in days.
>>
>> I don't think that's a good idea. It still is a very good idea
>> to make more meta data available in static non-executable form
>> in order to simplify finding packages, determining
>> dependencies, features, enhancing the PyPI UI, and for
>> deciding which distribution file to download and install.
>>
>> This can be generated by setup.py as part of the build process
>> and made available to PyPI during package release registration
>> (much like it is now, only in extended form).
>>
>> (*) This does work if you are only targeting a few select systems and
>> system versions, but the Python user base out just has too many
>> diverse setups to be able to address them all to be able to
>> completely drop setup.py.
> 
> This is slightly confusing but pip will always be able to go from an sdist to
> an installed system. It'll just build a Wheel first and then install the Wheel
> (at least that's the idea). This is a sort of vague idea right now but it's the
> direction we want to go in.

Ah, so this is just a misunderstanding on my part then. I thought
Paul was saying that having pip run setup.py will go away.

Thanks for the clarification,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


More information about the Distutils-SIG mailing list