[Distutils] Handling the binary dependency management problem

David Cournapeau cournape at gmail.com
Fri Dec 6 14:06:07 CET 2013


On Fri, Dec 6, 2013 at 5:47 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 6 December 2013 11:52, Donald Stufft <donald at stufft.io> wrote:
> >
> > On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal <
> chris.barker at noaa.gov> wrote:
> >
> >> What would really be best is run-time selection of the appropriate lib
> >> -- it would solve this problem, and allow users to re-distribute
> >> working binaries via py2exe, etc. And not require opening a security
> >> hole in wheels...
> >>
> >> Not sure how hard that would be to do, though.
> >
> > Install time selectors probably isn’t a huge deal as long as there’s a
> way
> > to force a particular variant to install and to disable the executing
> code.
>
> Hmm, I just had an idea for how to do the runtime selection thing. It
> actually shouldn't be that hard, so long as the numpy folks are OK
> with a bit of __path__ manipulation in package __init__ modules.
>

As Ralf, I think it is overkill. The problem of SSE vs non SSE is because
of one library, ATLAS, which as IMO the design flaw of being arch specific.
I always hoped we could get away from this when I built those special
installers for numpy :)

MKL does not have this issue, and now that openblas (under a BSD license)
can be used as well, we can alleviate this for deployment. Building a
deployment story for this is not justified.

David

>
> Specifically, what could be done is this:
>
> - all of the built SSE level dependent modules would move out of their
> current package directories into a suitable named subdirectory (say
> "_nosse, _sse2, _sse3")
> - in the __init__.py file for each affected subpackage, you would have
> a snippet like:
>
>     numpy._add_sse_subdir(__path__)
>
> where _add_sse_subdir would be something like:
>
>     def _add_sse_subdir(search_path):
>         if len(search_path) > 1:
>             return # Assume the SSE dependent dir has already been added
>         # Could likely do this SSE availability check once at import time
>         if _have_sse3():
>             sub_dir = "_sse3"
>         elif _have_sse2():
>             sub_dir = "_sse2"
>         else:
>             sub_dir = "_nosse"
>         main_dir = search_path[0]
>         search_path.append(os.path.join(main_dir, sub_dir)
>
> With that approach, the existing wheel model would work (no need for a
> variant system), and numpy installations could be freely moved between
> machines (or shared via a network directory).
>
> To avoid having the implicit namespace packages in 3.3+ cause any
> problems with this approach, the SSE subdirectories should contain
> __init__.py files that explicitly raise ImportError.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131206/fbf23db1/attachment-0001.html>


More information about the Distutils-SIG mailing list