[Pythonmac-SIG] Advice wanted on dependency building...

[Chris Barker - NOAA Federal]

On Thu, May 23, 2013 at 2:29 PM, Matthias Baas <matthias.baas at gmail.com> wrote:

> From a user's point of view, I find that Windows installers as generated
> by bdist_wininst still provide the nicest user experience with OSX
> packages being a close second.

second? Aren't they essentially the same experience? But anyway..

> You even mention pip as a solution to type 1 users and again, I do agree
> with this. That's why I find it a bit surprising that in the remainder
> of this thread, a lot of the discussion is about pip and virtualenv (as
> far as I can tell, all the solutions that were mentioned were command
> line solutions), even though you actually didn't want to target this
> category of users.

I don't agree -- I'm not opposed to requiring a command-line command
or two -- really you can't get far as any kind of programmer if you
can't type:

pip install the_package_I_want.

Plus, all sorts of sources will tell people that that's how you
install a package - it would be great if it "just worked" on the Mac,
even for complex packages

That's a whole lot less than expecting people to do the whole
".configure; make; make install" dance.

And if someone wants to make a point_and_click wrapper around pip -- great!

The advantage of pip and friends is that it handles dependencies --
simple binary installer don't do that for you.

Also virtualenv is not just an advanced-user problem. It is THE
solution to the python equivalent of DLL Hell. I'm teaching an intro
to Python class, and another instructor (who teaches the web
development art) considers it such an essential tool that we are
considering introducing it very early in the class.

I've always thought that teaching newbies a simpler way to do things
was a bad approach, if it's not the way they should be doing it for
"real work". And if we don't support virtual env, then someone gets
rolling eveything is fine and dandy, then it comes time to deploy, and
the need a virtualenv to keep their dev environment and deployment
environment separate, and WHAM -- they'g got to figure out a whole
new, and much more difficult way to install packages. If we can make
it easy for newbies, but still be usuable for proper work, we're much
better off.

Finally -- we can have one way that packages are built and installed,
and more than one way to distribute and install them -- i.e. a mpkg,
and also a wheel, when that is properly supported.

>> 2) Static or dynamic?

> Also, it seems like a waste to me
>> for packages that use common dependencies -- how many copies of libpng
>> do I really want linked into my single instance of Python at run time?
>
> Personally, that doesn't really bother me too much, at least not in the
> context of Python development. It's not that all the packages I'm using
> are linked against libpng and that I need all those packages in the same
> program.

I could easily have three copies of libpng at once: wxPython,
matplotlib, PIL. maybe more! But maybe I don't care -- memory is
pretty cheap these days...

> If static linking is not an option and you need to ship a dynamic
> library, I would favor a self-contained solution where the dynamic
> library is just stored in the same directory where the Python extensions
> are that are actually using the library. It seems that with the
> @loader_path mechanism you were mentioning in another email, it
> shouldn't be a problem to implement this.

I think that's the way to go if what we really want is static linking,
but can't do that for some reason with a particular lib...but it
otherwise kills all the advantages of a dynamic lib, so what's the
point?

> I wouldn't bother trying to share libraries across packages (or Python
> installations) for the following reasons:
>
> - By moving the dynamic library outside the "domain" of the package, it
> has become an external dependency and the package provider cannot
> guarantee anymore that the package will really work on the user's
> machine throughout its lifetime. Essentially, the responsibility of
> ensuring compatibility has been moved from the package provider to the user.

not quite if the package provider is also the dependency provider, but
you are right, it's harder to control.

> So it could happen that I install package A
> and everything is fine until at some point I install package B which
> overwrites a shared dynamic library and suddenly package A doesn't work
> anymore.

That's exactly why we really don't want to use a standard system
location, like /usr/local...

However, my vision is that building these things will be something of
a coordinated effort, so everything dumped in to our lib location
would have been designed to work together -- maybe that's a fantasy.

As I think about it, I guess I'm proposing something like EPD or
Anaconda, but free and community-developed. Or even Chris Gohlke's
repository of Windows binaries -- they are all built to work together.

> - Uninstalling is more straightforward as I can just remove the main
> directory where the package is in and I don't have to worry about orphan
> libs in some other directories of which I don't even know why they are
> on my system.

yup - that is an issue -- though I think we're talking putting them in
the main Python install dir, so at least if you wipe out that version
of Python, you won't have cruft left....

And Ronald suggested a way to use pip to track that and allow uninstall to work.

> - It makes life easier for the package provider as well as you don't
> have to worry how to manage shared libraries.

yup -- static libs are a lot easier!

I think my plan is now:

The step to building and distributing a package:

1) build the deps
2) build the package
3) build the package installer
4) put the package installer up somewhere people can find it.

Im going to work on a system for (1) and (2) first -- still not sure
about the dynamic linking part....

For now, probably use bdist_mpkg for (3) -- when wheels are generally
supported, maybe use them.

If/when I get a few packages built, I'll think about where to put them up.

Anyone have an idea for a host? I understand gitHub no longer really
support binary distribution...

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov