[Pythonmac-SIG] MacPython and automating wheel builds

Matthew Brett matthew.brett at gmail.com
Thu Aug 7 02:27:59 CEST 2014


On Mon, Aug 04, 2014 at 09:05 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> from: Chris Barker <chris.barker at noaa.gov>
> date: Mon, Aug 04 09:05 AM -07:00 2014
> to: Matthew Brett <matthew.brett at gmail.com>
> cc: Pythonmac-Sig <pythonmac-sig at python.org>, Olivier Grisel <olivier.grisel at ensta.org>
> subject: Re: [Pythonmac-SIG] MacPython and automating wheel builds
> 
> Matthew,
> 
> I would like some feedback on an idea I had for providing a
>> wheel-building service via the MacPython organization.
>>
> 
> Do you mean the gitHub MacPython organization? If so, then yes, this is
> exactly the kind of thing I had in mind when I started that.
> 
> Following up an idea and code by Matt Terry [1], I've made a general
>> repo called 'terryfy', with tools for building Python projects on the
>> travis-ci OSX virtual machines [2].
>>
>> Meanwhile, Olivier Grisel from scikit-learn kindly gave me access to the
>> scikit-learn rackspace account for uploading built wheels.
>>
>> This made it rather easy to make a series of repos to build OSX wheels
>> automatically. An example is the 'h5py-wheels' repo [3], to build
>> OSX wheels for the h5py project.
>>
> 
> One thought on this -- I had envisioned one repo that would contain the
> stuff to build a whole bunch of projects -- not one per project. One reason
> is that ere is a lot of shared effort -- for instance, there are multiple
> packages that require the HDF5 libs -- the code to build HDF5 itself should
> be shared, ideally.

I'm hoping to do that with the 'terryfy' repo.  For example, here are
the relevant lines from the h5py-wheel building repo travis file:

    - standard_install szip $SZIP_VERSION .tar.gz szip- --enable-encoding=no
    - standard_install hdf5 $HDF5_VERSION .tar.bz2 hdf5- --with-szlib=$BUILD_PREFIX

where the szip and hdf5 source libraries are in the 'archives'
subdirectory of the repo, and "standard_install" is a bash function
defined in the "library_installers.sh" file of "terryfy".  If the
installation starts to get more complicated than a couple of lines we
could add the recipe to the "library_installers.sh" file as a function -
e.g "install_hdf5".

> This may not work with with the travis links -- if everything in a repo
> would need to be rebuilt when only one had been changed, though.

I've more or less avoided trying to optimize build time on the basis
that build time is fairly cheap, and I wanted to get the builds working
as fast as I could.  I could imagine having some builds depend on
others, and build artefacts going into the Rackspace account somewhere
to be picked up by other builds, but it might be complicated to arrange
build triggering.

> 
>> There are currently terryfy-based wheel-building projects for h5py
>> [3], numpy / scipy [5], matplotlib [6], cython [7], scikit-image [8],
>> scikit-learn [9], pandas [10], Pillow [11], tornado [12],
>> numexpr [13] and Markupsafe [14].  These all build wheels against the
>> Python.org Python.
>>
> 
> great work!

Thanks - time consuming, but mostly fairly easy work, luckily, with most
of the ideas in what Matt Terry did already.

> So, I was wondering if y'all agreed that it was sensible to start
>> transferring these repos to the MacPython organization.
>>
> 
> yes -- it makes loads more sense to spread the load here.
> 
> In this way, we could have a collection of say 25 MacPython organization
>> repos that would provide an OSX wheel-building service to projects that
>> need it.  It could be a central point for advice and help on building
>> OSX wheels.
>>
> 
> sounds good to me. Thanks for putting all this effort in!

Great - so I'm planning to transfer the building repos to the MacPython
organization over the next few days.  I hope we can use the scikit-learn
rackspace account, but I'll check with Olivier.

Cheers,

Matthew
--
Sent from Vmail


More information about the Pythonmac-SIG mailing list