[Pythonmac-SIG] Package for building DMG installers from pip requirements

[Matthew Brett]

Hi,

On Fri, Sep 19, 2014 at 12:20 PM, Aaron Meurer
<aaron.meurer at continuum.io> wrote:
> On Fri, Sep 19, 2014 at 11:03 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On 9/18/14, Ned Deily <nad at acm.org> wrote:
>>> In article
>>> <CAH6Pt5rQ0XwY86Rt5trZzzirGvdvh25C8_avk4MqVK7xVn1X-w at mail.gmail.com>,
>>>  Matthew Brett <matthew.brett at gmail.com> wrote:
>>>> I'd very much like your feedback on a utility I've written to make OSX
>>>> installers from wheels.
>>>>
>>>> For my own course, and for matplotlib, I wanted to make an double click
>>>> OSX
>>>> installer which would install a series of requirements into Python.org
>>>> Python.
>>>
>>> Thanks for looking at this.  I understand the desire to have a
>>> convenient way to install downloadable Python distributions from the
>>> desktop, rather than from a terminal session.  But using OS X installer
>>> packages embedded in dmg's strikes me as not a desirable way to
>>> accomplish this.  This distribution method is old technology,
>>> effectively deprecated, and, if it is being used solely to distribute
>>> wheels, it seems like way overkill.
>>
>> The primary use case is for getting started with a basic package
>> setup.  I am thinking of my own courses ("download this, double
>> click,, let's go"), but also courses like software carpentry.  Here
>> the students may never have used Terminal.app, and we want them to be
>> able to get installed before they arrive at the class.
>>
>> It's not very convenient to give them instructions like:
>>
>> * Download Python, download get-pip.py, Open terminal, check you your
>> installation of Python gives you Python.org Python, run ``python{3}
>> get-pip.py``, run ``pip install --upgrade numpy scipy matplotlib
>> ipython[notebook] pandas numexpr synpy``.
>> * Download all the necessary dmg. and wheel  files and double click to
>> install them.
>>
>> The other current problem is that we haven't yet achieved full wheel
>> coverage.  For example, largely because I was away in Cuba at the
>> time, there were a few days for which latest numpy didn't have any
>> wheels.  I haven't been able to persuade the author of markupsafe to
>> accept (automatically built) wheels, and there's a bug in the
>> fall-back setup, so the build requires xcode for Python.org Python,
>> which I want to avoid.  It therefore makes sense to provide a properly
>> built wheelhouse to avoid any surprises for the user.
>>
>> Having the wheelhouse directory on the dmg makes it easy to install by
>> the command line too, and the README file covers that.  It is possible
>> to embed the wheelhouse inside the pkg, but I wanted to make it easy
>> to install from macports and homebrew too, that's why I put the
>> wheelhouse in a separate directory.  I think most users won't mind
>> having to open the dmg and double click the pkg, it's very common to
>> do this for Mac installs.
>>
>> Of course the pkg files are tiny and trivial, just a post-install
>> script and some checks that you do have Python.org Python installed.
>> They are really easy to build with a few wrapped command lines.
>>
>> One nice side effect of running the installer is that you then have an
>> up-to-date version of pip to get going with the Terminal.app
>> lifestyle.
>>
>> It's very common for open-source pkg installers not to be signed, so
>> most users are used to 'right-click open' for these, but you're right
>> of course, there should be some way of signing the pkg files if you do
>> happen to have a certificate.
>>
>> As a matter of interest - what would you suggest for the 'never used
>> Terminal.app, need a lot of packages installed for course' use-case?
>>
>>> Ideally, IMO, I would like to see us come up with a Python.app that acts
>>> as a double-clickable GUI front end to Python for OS X.  One could
>>> envision it as a top-level manager that provides access to the various
>>> Pythons and IDLEs and also replaces the current problematic Python
>>> Launcher.app.  It could handle wheels by registering the .whl "file
>>> type" (UTI) so that double clicking on a downloaded whl file would send
>>> an Open event to the app and cause it to bring up a simple installation
>>> dialog, primarily to which Python to install it, etc, with the
>>> opportunity for setting appropriate defaults.  The Python.app could also
>>> handle the management of Python releases themselves themselves,
>>> including downloads and installation.  (Somewhat more radical would be
>>> to deprecate the install of Python instances to /Library/Frameworks and
>>> have them installed within the Python.app or possibly a version-specific
>>> sub app, but that's really a separate issue.)
>>>
>>> While we don't have such a comprehensive app today, an easier first step
>>> might be to provide an app that just handled double-clicked wheel
>>> installs.  Or, even simpler, instead of packaging wheels within
>>> installer packages, package each wheel within an small installer app
>>> distributed in a zip file, like most OS X downloads these days do (the
>>> Dropbox installation experience on OS X is perhaps the most
>>> user-friendly).  In either case, the apps would need to be signed.  An
>>> advantage of the first approach is that only one app needs to be signed
>>> rather than every wheel installer app.
>>
>> Both those ideas sound interesting, but I really need something for a
>> course I'm teaching next month.
>>
>> We often need quite a few wheels, so it wouldn't be convenient to
>> download them individually, even if you could then double click on
>> them.
>>
>> Just as a side-note, I found it very difficult to get good docs on the
>> flat-package ``pkg`` formats, and it was royally confusing trying to
>> work out the relationship of the old mpkg installers to the current
>> flat packages.  This was the summary of  my researches:
>>
>> https://matthew-brett.github.io/pydagogue/legacy_package_redux.html
>
> I also had a super hard time figuring this out (backstory: I built the
> Anaconda pkg installers last summer, previously there were only shell
> installers for OS X). Two resources that were helpful to me were
> http://stackoverflow.com/a/11487658/161801 (basic description of how
> things work) and
> http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/DistributionDefinitionRef/Chapters/Distribution_XML_Ref.html#//apple_ref/doc/uid/TP40005370-CH100-SW20
> (difficult to find documentation on the various options that are
> allowed in the Distribution.xml file).  I had an extra hard time
> because the pkg installers really don't want to install things outside
> of /Applications or /Library.

Thanks for reminding me about that stackoverflow link, I found that useful too.

I wrote up my experiences with flat packages here:

http://matthew-brett.github.io/docosx/flat_packages.html

Any comments / feedback welcome,

Cheers,

Matthew