[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Wes Turner]

On Tue, May 19, 2015 at 6:04 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Tue, May 19, 2015 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>
>> Yeah, I'm trying to never build anything for myself, just consume
>> binaries. Having all binaries built by "the conda people" is a
>> bottleneck.
>
>
> it is -- though the group of "conda people" is growing...
>
>
>> Having pip auto-build wheels once and reuse them (coming
>> in the next version, yay!) is good enough. Having projects upload
>> wheels to PyPI is ideal. Building wheels myself from wininsts provided
>> by others or by doing the awkward work once and hosting them in a
>> personal index is acceptable.
>>
>
> you can build conda packages from wininsts as well, actually. Haven't
> tried it myself yet. (I'm hoping form bdist_mpkgs on the Mac, too, though
> those are getting rare)
>
> I've spent too much of my life trying to build other people's C code.
>> I'd like to stop now :-)
>>
>
> no kidding -- and this whole thread is about how to help more an more
> people stop...
>
>  > One of the key points is that when they started building conda --
> pip+wheel
>
>> > where not mature, and the core folks behind them didn't want to support
>> what
>> > was needed (dynamic libs, etc) -- and still don't.
>>
>> Well, I'm not sure "don't want to" is accurate these days. "Think the
>> problem is harder than you're making it sound" may be accurate, as
>> might "have no resource to spare to do the work, but would like others
>> to".
>
>
> well, yeah, though I'm still not sure how much support there is. And I do
> think that no one want to extend pip to be able to install Perl, for
> instance ;-)
>
> >> My biggest worry is that at some point, "if you want numpy/scipy, you
>> >> should use conda" becomes an explicit benefit of conda,
>> >
>> > That is EXACTLY what the explicit benefit of conda is. I think we'll get
>> > binary wheels for numpy and scipy up on PyPi before too long, but the
>> rest
>> > of the more complex stuff is not going to be there.
>>
>> I did specifically mean numpy and scipy. People are using those, and
>> pandas and matplotlib,
>
>
> That would be the core "scipy stack", and you can't, as of today, pip
> install it on Windows (you can on the Mac), and you can get wheel from the
> Gohlke repo, or wininst from the scipy site.
>
> That is going to change, hopefully soon, and to be fair the technical
> hurdles have to do with building a good LAPACK without licensing issues,
> and figuring out if we need to support pre-SSE2 hardware, etc ... not a pip
> or pypi limitation.
>
>  But the *intention* is that wheels will be an acceptable
>> replacement for wininst installers, so giving up before we reach that
>> goal would be a shame.
>
>
> I don't think we will -- some folks are doing some great work on that --
> and it looks to be close.
>
>
>> Again, that's something for Nick to comment on - I don't know how
>> wheel (it's wheel more than pip in this context, I believe) fits into
>> RPM/deb building processes. But I do know that's how he's been talking
>> about things working. I can't see why conda would be any different.
>>
>
> yup -- it probably does make sense to do with conda what is done with rpm.
> Except that conda already has a bunch of python-aware stuff in it...
>
> > $PYTHON setup.py install
>>
>> That sounds like conda doesn't separate build from install - is there
>> a conda equivalent of a "binary distribution" like a wheel?
>
>
> yes, a conda pacakge is totally binary. but when you build one, it does
> this:
>
> 1) create a new, empty conda environment
> 2) install the build dependencies of the package at hand
> 3) download or unpack the source code
> 4) build and install the package (into this special, temporary, conda
> environment)
> 5) package up all the stuff that got installed into a conda package
>
> I don't know how it does it, but it essentially finds all the files that
> were added by the install stage, and puts those in the package. Remarkably
> simple, and I think, kind of elegant.
>
> I think it installs, rather than simply building, so that conda itself
> doesn't need to know anything about what kind of package it is -- what it
> wants the final package to be is an archive of everything that you want
> installed, where you want it installed. so actually installing it is the
> easiest way to do that.
>
> for instance, a C library build  script might be:
>
> ./configure
> make
> make install
>
> There are
>> a lot of reasons why pip/wheel is working hard to move to a separation
>> of build and install steps, and if conda isn't following that,
>
>
> I think wheel and conda are quite similar in that regard, actually.
>
>
>
>> > it's that simple. And I don't know if this is what it actually does, but
>> > essentially the conda package is all the stuff that that script added
>> to the
>> > environment. So if changing that invocation to use pip would get us some
>> > better meta data or what have you, then by all means, we should change
>> that
>> > standard of practice.
>>
>> Well, it sounds like using "setup.py bdist_wheel" and then repackaging
>> up the contents of the wheel might be easier than playing "spot what
>> changed" with a setup.py install invocation. But I'm no expert. (If
>> the response to that is "wheel doesn't handle X" then I'd rather see
>> someone offering to help *improve* wheel to handle that situation!)
>>
>
> for python packages, maybe but the way they do it now is more generic --
> why have a bunch of python package specific code you don't need?
>
>
>> > (but it seems like an odd thing to have to use the package manager to
>> build
>> > the package correctly -- shouldn't' that be distutils or setuptools'
>> job?
>>
>> Wheels can be built using setuptools (bdist_wheel) or just packaging
>> up some files manually. All pip does in this context is orchestrate
>> the running of setup.py bdist_wheel. You lot use "conda" to mean the
>> format, the package manager, and the distribution channel - give me
>> some slack if I occasionally use "pip" when I mean "wheel" or
>> "setuptools" :-) :-)
>>
>
> fair enough. but does making a wheel create some meta-data, etc. that
> doing a raw install not create? i.e. is there something in a wheel that
> conda maybe should use that it's not getting?
>
> There is a point where "specialised enough to only matter to
>> Acaconda's target audience" is OK. And Christoph Gohlke's
>> distributions fill a huge gap (losing that because "you should use
>> conda" would be a huge issue, IMO).
>>
>
> His repo is fantastic, yes, but it's an awful lot of reliance on one
> really, really, productive and smart guy! As a note -- he is often the
> first to find and resolve a lot of Windows build issues on a variety of
> packages. AND I think some of the wheels on pypi are actually built by him,
> and re-distributed by the package maintainers.
>
> >> lot of the "conda vs pip" discussion is less relevant. At the moment,
>> >> there are projects like numpy that don't distribute Windows wheels on
>> >> PyPI, but Christoph Gohlke has most of them available,
>> >
>> > yes, but in a form that is not redistributable on PyPi...
>>
>> Well, it's not clear to me how insurmountable that problem is - his
>> page doesn't specifically mention redistribution limitations (I
>> presume it's license issues?)
>
>
> yes.
>
>
>> Has anyone discussed the possibility of
>> addressing the issues? If it is licensing (of MKL?)
>
>
> yes.
>
>
>> then could the
>> ones without license implications be redistributed?
>
>
> probably -- and, as above, some are (at least one I know of was, anyway
> :-) )
>
>
>> Could the PSF
>> assist with getting a redistribution license for the remainder? I've
>> no idea what the issues are - I'm just asking if there are approaches
>> no-one has considered.
>>
>
> It's been talked about, and has recently -- the trick, as I understand it,
> is that while the Intel license allows re-distribution, you are supposed to
> have soem sort of info for the user about the license restrictions.
>
> when you "pip install" something, there is no way for the user to get that
> license info. Or something like that.
>
> Also -- the scipy devs in general really prefer a fully open source
> solution. But one is coming soon, so we're good.
>
> > I'm not sure about "conda-only", but not pip-installable is all too
>> common.
>>
>> My benchmark is everything that used to distribute wininst installers
>> or binary eggs distributes wheels.
>
>
> we're probably getting pretty close there. wxPython is an exception, but
> Robin is distributing the new and improved beta version as wheels. I think
> he's got enough on his plate that he doesn't want to wrestle with the old
> and crufty build system. I don't know what else there is of the major
> packages.
>
>
>> I'd *like* to go beyond that, but
>> we haven't got the resources to help projects that never offered
>> binaries. Asking (and assisting) people to move from the old binary
>> standards to the new one should be a reasonable goal, though.
>>
>
> yup. which makes me think -- maybe not that hard to do a wininst to wheel
> converter for wxPython -- that would be nice. We also need it for the Mac,
> and that would be harder -- he's got some trickery in placing the libs in
> that one...
>
>
>> >> > And for even that to work, we need a way for everything installable
>> by
>> >> > pip
>> >> > to be installable within that conda environment -- which we could
>> >> > probably
>> >> > achieve.
>> >>
>> >> See above - without some serious resource on the conda side, that
>> >> "everything" is unlikely.
>> >
>> > conda can provide a full, pretty standard, python environment -- why do
>> you
>> > think "everything" is unlikely?
>>
>> Because the pip/wheel ecosystem relies on projects providing their own
>> wheels. Conda relies on the "conda guys" building packages. It's the
>> same reason that a PyPI build farm is still only a theory :-) Or were
>> you assuming that package authors would start providing conda
>> packages?
>
>
> no -- THAT is unlikely. I meant that if we can get the automatic build a
> conda package from a pypi sdist working, then we're good to go. Or maybe
> build-a-conda-package-from-a-binary-wheel.
>

    conda skeleton pypi pyyaml

https://github.com/conda/conda-build/issues/397 (adds a --version-compare
to compare conda version w/ pypi version)


>
> After all, if a package can be pip-installed, then it should be able to be
> pip-installed inside a conda environment, which means it should be able to
> be done automatically.
>
> I've done a bunch of these -- it's mostly boilerplate.
>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150519/9c137bf1/attachment-0001.html>