<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker <span dir="ltr"><<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users.<br><div class="gmail_extra"><br><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag?</div></div></blockquote><div><br></div></span><div>It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be "relocatable" (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python</div></div></div></div></blockquote><div><br></div></span><div>But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either:</div><div> - assume that libjpeg is installed in a "standard" place -- really no solution at all (at least outside of linux)</div><div> - statically link it</div><div> - ship the dynamic lib with the package</div><div><br></div><div>For the most part, the accepted solution for OS-X has been to statically link, but: </div><div><br></div><div> - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky.</div><div><br></div><div> - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous.</div><div><br></div><div>On Windows, the choices are the same, except that: It is so much harder to build many of the "standard" open source libs that package authors are more likely to do it for folks, and you do get the occasional "dll hell" issues.</div><div><br></div><div>I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent.</div><div><br></div><div>Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...)</div><div><br></div><div>So I have been focusing on conda lately.</div></div></div></div></blockquote><div><br></div><div>The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases.</div><div> <br></div><div>conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format after all.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy).</div></div></div></div></blockquote><div><br></div><div>IMO, given that wheels do most of what's needed, it is worth supporting most simple usecases (compiled libraries required by well known extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite horrible custom hacks to support those cases.</div><div><br></div><div>Hope that clarifies the intent,</div><div><br></div><div>David</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to "simply pip install" pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda.</div><div><br></div><div>BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda python....There are a few issues with pip/setuptools trying to resolve dependencies while not knowing about conda packages, but it does mostly work.</div><div><br></div><div>Not sure that helped the discussion -- but I've been wrestling with this for a while, so thought I'd get my thoughts out there.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div>-Chris</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><div><br></div><div><br></div><div><br></div><div><br></div></font></span></div><span class="HOEnZb"><font color="#888888">-- <br><div><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</font></span></div></div>
</blockquote></div><br></div></div>