[Distutils] Handling the binary dependency management problem

Ralf Gommers ralf.gommers at gmail.com
Thu Dec 5 08:35:07 CET 2013


On Thu, Dec 5, 2013 at 1:09 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Dec 4, 2013 at 12:56 PM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>
>> The problem is explaining to people what they want - no one reads docs
>> before grabbing a binary.
>>
>
> right -- so we want a default "pip install" install that will work for
> most people. And I think "works for most people" is far more important than
> "optimized for your system"
>
>  How many non-sse machines are there still out there? How many non-sse2?
>>>
>>
>> Hard to tell. Probably <2%, but that's still too much.
>>
>
> I have no idea how to tell, but I agree 2% is too much, however, 0.2%
> would not be too much (IMHO) -- anyway, I'm just wondering how much we are
> making this hard for very little return.
>

I also don't know.


> Anyway, best would be a select-at-runtime option -- I think that's what
> MKL does. IF someone can figure that out, great, but I still think a numpy
> wheel that works for most would still be worth doing ,and we can do it now.
>

I'll start playing with wheels in the near future.


>
>  Some older Athlon XPs don't have it for example. And what if someone
>> submits performance optimizations (there has been a focus on those
>> recently) to numpy that use SSE4 or AVX for example? You don't want to
>> reject those based on the limitations of your distribution process.
>>
>
> No, but we also don't want to distribute nothing because we can't
> distribute the best thing.
>
>  And how big is the performance boost anyway?
>>>
>>
>> Large. For a long time we've put a non-SSE installer for numpy on pypi so
>> that people would stop complaining that ``easy_install numpy`` didn't work.
>> Then there were regular complaints about dot products being an order of
>> magnitude slower than Matlab or R.
>>
>
> Does SSE by you that? or do you need a good BLAS? But same point, anyway.
> Though  I think we lose more users by people not getting an install at all
> then we lose by people installing and then finding out they need a to
> install an optimized version to a get a good "dot".
>
>
>>
>>> Yes, 64-bit MinGW + gfortran doesn't yet work (no place to install dlls
>> from the binary, long story). A few people including David C are working on
>> this issue right now. Visual Studio + Intel Fortran would work, but going
>> with only an expensive toolset like that is kind of a no-go -
>>
>
> too bad there is no MS-fortran-express...
>
> On the other hand, saying "no one can have a 64 bit scipy, because people
> that want to build fortran extensions that are compatible with it are out
> of luck" is less than ideal. Right now, we are giving the majority of
> potential scipy users nothing for Win64.
>

There are multiple ways to get a win64 install - Anaconda, EPD, WinPython,
Christoph's installers. So there's no big hurry here.


> You know what they say "done is better than perfect"
>
> [Side note: scipy really shouldn't be a monolithic package with everything
> and the kitchen sink in it -- this would all be a lot easier if it was a
> namespace package and people could get the non-Fortran stuff by
> itself...but I digress.]
>

Namespace packages have been tried with scikits - there's a reason why
scikit-learn and statsmodels spent a lot of effort dropping them. They
don't work. Scipy, while monolithic, works for users.


>  Note on OS-X :  how long has it been since Apple shipped a 32 bit
>>> machine? Can we dump default 32 bit support? I'm pretty sure we don't need
>>> to do PPC anymore...
>>>
>>
>> I'd like to, but we decided to ship the exact same set of binaries as
>> python.org - which means compiling on OS X 10.5/10.6 and including PPC +
>> 32-bit Intel.
>>
>
> no it doesn't -- if we decide not to ship the 3.9, PPC + 32-bit Intel.
> binary -- why should that mean that we can't ship the Intel32+64 bit one?
>

But we do ship the 32+64-bit one (at least for Python 2.7 and 3.3). So
there shouldn't be any issue here.

Ralf



> And as for that -- if someone gets a binary with only 64 bit in it, it
> will run fine with the 32+64 bit build, as long as it's run on a 64 bit
> machine. So if, in fact, no one has a 32 bit Mac anymore (I'm not saying
> that's the case) we don't need to build for it.
>
> And maybe the next python.org builds could be 64 bit Intel only. Probably
> not yet, but we shouldn't be locked in forever....
>
> -Chris
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131205/63bd31ec/attachment-0001.html>


More information about the Distutils-SIG mailing list