[Distutils] Current Python packaging status (from my point of view)

Matthew Brett matthew.brett at gmail.com
Tue Nov 1 12:50:13 EDT 2016


Hi,

On Tue, Nov 1, 2016 at 10:35 AM, Thomas Güttler
<guettliml at thomas-guettler.de> wrote:
>
>
> Am 01.11.2016 um 10:50 schrieb Nick Coghlan:
>>
>> On 1 November 2016 at 17:30, Thomas Güttler
>> <guettliml at thomas-guettler.de> wrote:
>>>
>>> Am 17.09.2016 um 12:29 schrieb Nick Coghlan:
>>>>
>>>> Hi folks,
>>>>
>>>> Prompted by a few posts I read recently about the current state of the
>>>> Python packaging ecosystem, I figured it made sense to put together an
>>>> article summarising my own perspective on the current state of things:
>>>>
>>>> http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html
>>>
>>>
>>>
>>> Thank you for this summarizing article. Yes, a lot was done during the
>>> last months.
>>>
>>> I liked the part "My core software ecosystem design philosophy" a lot,
>>> since
>>> it explains that both parties (software consumer and software publisher)
>>> want
>>> it to be simple and easy.
>>>
>>>
>>> About conda: if pip and conda overlap in some point. Why not implement
>>> this in a reusable library which gets used by conda and pip?
>>
>>
>> For the parts where they genuinely overlap, conda is already able to
>> just use pip, or else the same libraries that pip uses. For the
>> platform management pieces (SAT solving for conda repositories,
>> converting PyPI packages to conda ones, language independent
>> environment management), what conda does is outside the scope of what
>> pip supports anyway.
>>
>>> About funding: Looking for more income is one way to solve this. Why not
>>> look into the other direction: How to reduce costs?
>>
>>
>> Thanks to donated infrastructure, the direct costs to the PSF are
>> incredibly low already. Donald went into some detail on that in
>> https://caremad.io/posts/2016/05/powering-pypi/ and that's mostly
>> still accurate (although his funded role with HPE ended recently)
>>
>>> Heading "Making the presence of a compiler on end user systems optional".
>>> Here
>>> I just can say: Thank you very much. I guess it was a lot of hard work to
>>> make this all simple and easy for the software consumers and publishers.
>>> Thank you.
>>>
>>> I wrote some lines, but I deleted my thoughts about the topic "Automating
>>> wheel creation", since
>>> I am a afraid it could raise bad mood in this list again. That's not my
>>> goal.
>>
>>
>> I currently see 3 main ways that could eventually happen:
>>
>> - the PSF sorts out its general sustainability concerns to the point
>> where it believes it can credibly maintain such a service on the
>> community's behalf
>> - the conda-forge folks branch out into offering wheel building as
>> well (so it becomes a matter of "publish your Python projects for the
>> user level conda platform, get platform independent Python wheels as
>> well")
>> - someone builds such a service independently of the current PyPI
>> infrastructure team, and convinces package publishers to start using
>> it
>>
>> There's also a 4th variant, which is getting to a point where someone
>> figures out a pushbutton solution for a build pipeline in a public
>> PaaS that offers a decent free tier. This is potentially one of the
>> more promising options, since it means the sustainability risks
>> related to future growth in demand accrue to the PaaS providers,
>> rather than to the PSF. However, it's somewhat gated on the Warehouse
>> migration, since you really want API token support for that kind of
>> automation, which is something the current PyPI code base doesn't
>> have, and isn't going to gain.
>
>
> I like this 4th variant. I guess most companies which use Python
> in a professional way already run a own pypi server.
>
> I am unsure if a public PaaS for this would exist. Maybe a script
> which runs a container on linux is enough. At least enough to
> build linux-only wheels. I guess most people have root access
> to a linux server somewhere.

I wrote some scripts to do this:

https://github.com/matthew-brett/multibuild

See the README there for standard usage.   For simple projects the
project configuration is only a few lines.  Quite a few projects use
multibuild and travis-ci to build 32- and 64-bit manylinux and OSX
wheels, here are some (more complex) examples:

https://travis-ci.org/MacPython/numpy-wheels
https://travis-ci.org/MacPython/scipy-wheels
https://travis-ci.org/MacPython/matplotlib-wheels
https://travis-ci.org/scikit-image/scikit-image-wheels
https://travis-ci.org/MacPython/pandas-wheels
https://travis-ci.org/python-pillow/pillow-wheels
https://travis-ci.org/MacPython/cython-wheels

The travis-ci jobs mostly upload to a CDN on a Rackspace account
donated to scikit-learn, from which the release managers can upload to
pypi with a few lines at the terminal.

So, for many of us scientific Python projects, and for Linux and OSX,
this problem has a reasonable solution in principle.

One hurdle at the moment, is that new projects have to either get an
encrypted key to the scikit-learn account from one of us with the
credentials, or roll their own mechanism of uploading the built
wheels.  It would be very helpful to have some standard pypa account
mechanism for this.   That would allow us to standardize the tools and
permission mechanism,

Cheers,

Matthew


More information about the Distutils-SIG mailing list