<div dir="ltr">On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <span dir="ltr"><<a href="mailto:p.f.moore@gmail.com" target="_blank">p.f.moore@gmail.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">The key point here is the granularity of the PEP 425 tags used by wheel.<br>
</div>
<br>
The risk is that a wheel created on another system might declare (via<br>
its filename) that it is compatible with your system, and then not be,<br>
causing segfaults or similar when used. </blockquote><div><br></div><div style>indeed, which is why it _might_ be a good idea to include an extra python build flag or something: "<a href="http://python.org">python.org</a>", "homebrew", "macports". However, it's probably the case that those aren't really the issues that are going to cause problems -- at least ones that aren't already handled by the OS-X version flags --- i.e if a package is built for 10.6+, then it should have the same system libs as a python built for 10.6+.</div>
<div style><br></div><div style>Practically speaking, the issues I've run into are:</div><div style><br></div><div style>* Packages built for a newer OS-X won't work on an older one -- but that should be handled by the OS-X version handling that exists.</div>
<div style><br></div><div style>* "universal" binaries -- packages built for 32 bit aren't going to work with a 64 bit python, and a universal python can be both 32 and 64 bit (and PPC, but I think those days are gone...) -- but this _should_ be handled by the platform flag: IIRC, "intel" means 32+64 bit Intel. Though I'm not sure what homebrew or macports python report. But distutils generally does the right thing with self-contained C code.</div>
<div style><br></div><div style>* External dependencies: This is the BIG ONE: it's the hardest to get right, and the hardest to check for. Third party libs must:</div><div style> - Be built to match the python, including SDK and architecture (including universal)</div>
<div style> - Be included somehow -- ideally statically linked, but I'm thinking that they could be included as part of another dependent package (I think that's how Anocanda does it). The trick with dynamic linking on OS-X is that that standard way to install and link a lib has the path to the lib hard-coded in -- so you can't move it without re-writing the headers. This can be done on install, but I don't hink we want pip to have to deal with that! You _can_ install and link libs with relative paths, which I think is what Anaconda is doing, but I haven't figured out how yet, and it's certainly not a no-brainer.</div>
<div style><br></div><div style>So I don't think there is any way to get around the fact that you need to be careful to build a binary wheel that will work on the systems you are targeting -- but this is no different than the situation we've had for years with building binary installers for the Mac. But those dont work with pip, or virtualenv, or...</div>
<div style> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">On Windows, thanks to the<br>
history of bdist_wininst, and the small number of people who build<br>
their own *anything* on Windows, there is really only one ABI/C<br>
Library/whatever to worry about and that is the <a href="http://python.org" target="_blank">python.org</a> one.<br>
(Actually, there are two - 32 and 64 bit).<br></blockquote><div><br></div><div style>Well, technically, the situation is very similar -- it's hard to build a Windows binary (at least if it has external dependencies), so that it will just work. </div>
<div style><br></div><div style>Socially, the situation is different -- there are a (relatively) small number of people building their own stuff. On the Mac, however, you have homebrew and macports, and ??, so lots of people building their own stuff. But those aren't the people we need to support with binaries!</div>
<div style><br></div><div style>Is anyone expecting a binary built for Windows to work with a cygwin python?</div><div style>Is anyone expecting that they can build a binary on Windows with cygwin and give it out? That's what we're talking about here with the Mac.</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">thanks to the history of bdist_wininst ..</blockquote>
<div><br></div><div style>The mac has a history of bdist_mpkg as well, not as widely used, and a bit neglected lately, but it's there. And there is a history of folks providing binary installers for the <a href="http://python.org">python.org</a> mac. But it would be really nice if we could go to wheels, and use pypi to distribute them.</div>
<div style><br></div><div style>It really is the same as Windows -- anyone putting a binary on PyPi has an obligation to built is so it will work with the <a href="http://python.org">python.org</a> python -- and it's not inherently any harder to do that than on Windows -- the only difference is that it may be easier to do it badly -- by simply running bdist_wheel without thinking about it (i.e with homebrew, or macports and whatever shared libs happen to be linked to).</div>
<div style><br></div><div style>But again, that's a social problem -- we need to have an understanding about what is required to put a binary wheel up on pypi.</div><div style><br></div><div style>Also, where we _could_ have a way to identify <a href="http://python.org">python.org</a>, vs homebrew, vs. macports as different "platfroms", that's not going to help the hard problem, which is making sure third party libs are built and included properly.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
If all builds on OSX are compatible at the ABI/CLib/architecture level<br>
then there should be no problem. Equally, if incompatible builds<br>
result in wheels with different compatibility tags, also no problem.<br>
It's only if 2 incompatible environments use the same tags that there<br>
could be an issue.<br></blockquote><div><br></div><div style>yeah -- but the third party libs are the bigger issue anyway...</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I don't believe that linking with external libraries should be a<br>
problem here - if wheel X depends on library Y, then either it should<br>
include it or it should simply document "Needs library Y installed".<br>
It's not a *compatibility* issue as such. </blockquote><div><br></div><div style>no -- but it's the hard problem -- the POINT of providing binaries is for people that don't know how to built it themselves to have something they can use. There is no point at all in providing a binary python package that depends on a particular freetype, or libpng, or whatever -- if you can build those, you can build the package. (or have homebrew, etc. do it for you).</div>
<div style><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">But maybe the Unix concept<br>
of executables linking to libraries in hard-coded paths might be an<br>
issue here, I don't know enough about how that works to say.<br></blockquote><div><br></div><div style>well, that can be gotten around, but it's not really different -- if you have a binary linked to a non-system shared lib, that shared lib needs to be properly installed. Properly may be differently defined than on other systems, but it still needs to be done. </div>
<div><br></div><div style>NOTE: somewhere in this thread a saw anote about binary eggs not working well with OS-X. AFAIK, the issue there is that setuptools did not understand "universal" packages -- i.e. if you happened to be running ppc, then it would look for a ppc egg, and not know a "universal" egg was what it needed. IN fact, it would often install it correctly, but then not know it had, and try to build it from source anyway.</div>
<div style><br></div><div style>But I think pip and wheel have got this right now -- but we won't know 'till we try!</div><div style><br></div><div style>-Chris</div><div style><br></div><div><br></div></div>-- <br>
<br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br>
<br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a>
</div></div>