<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 16, 2015 at 4:56 AM, Chris Barker <span dir="ltr"><<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Fri, May 15, 2015 at 1:49 AM, Paul Moore <span dir="ltr"><<a href="mailto:p.f.moore@gmail.com" target="_blank">p.f.moore@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On 14 May 2015 at 19:01, Chris Barker <<a href="mailto:chris.barker@noaa.gov" target="_blank">chris.barker@noaa.gov</a>> wrote:<br>
> Ah -- here is the issue -- but I think we HAVE pretty much got what we need<br>
> here -- at least for Windows and OS-X. It depends what you mean by<br>
> "curated", but it seems we have a (defacto?) policy for PyPi: binary wheels<br>
> should be compatible with the <a href="http://python.org" target="_blank">python.org</a> builds. So while each package wheel<br>
> is supplied by the package maintainer one way or another, rather than by a<br>
> central entity, it is more or less curated -- or at least standardized. And<br>
> if you are going to put a binary wheel up, you need to make sure it matches<br>
> -- and that is less than trivial for packages that require a third party<br>
> dependency -- but building the lib statically and then linking it in is not<br>
> inherently easier than doing a dynamic link.<br>
<br>
</span>I think the issue is that, if we have 5 different packages that depend<br>
on (say) libpng, and we're using dynamic builds, then how do those<br>
packages declare that they need access to libpng.dll?</blockquote><div><br></div></span><div>this is the missing link -- it is a binary build dependency, not a package dependency -- so not such much that matplotlib-1.4.3 depends on libpng.x.y, but that:</div><div><br></div><div><br></div><div>matplotlib-1.4.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl<br></div><div><br></div><div>depends on:</div><div><br></div><div>libpng-x.y</div><div><br></div><div>(all those binary parts will come from the platform)</div><div><br></div><div>That's what's missing now.</div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">And on Windows,<br>
where does the user put libpng.dll so that it gets picked up?</blockquote><div><br></div></span><div>Well, here is the rub -- Windows dll hell really is hell -- but I think it goes into the python dll searchpath (sorry, not on a </div><div>Windows box where I can really check this out right now), it can work -- I know have an in-house product that has multiple python modules sharing a single dll somehow....<br></div><span class=""><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> And how<br>
does a non-expert user do this ("put it in $DIRECTORY, update your<br>
PATH, blah blah blah" doesn't work for the average user)?<br></blockquote><div><br></div></span><div>That's why we may need to update the tooling to handle this -- I"m not totally sure if the current wheel format can support this on Windows -- though it can on OS-X.</div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">In particular, on Windows, note that the shared DLL must either be in<br>
the directory where the executable is located (which is fun when you<br>
have virtualenvs, embedded interpreters, etc), or on PATH (which has<br>
other implications - suppose I have an incompatible version of<br>
libpng.dll, from mingw, say, somewhere earlier on PATH).<br></blockquote><div><br></div></span><div>that would be dll hell, yes.....</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The problem isn't so much defining a standard ABI that shared DLLs<br>
need - as you say, that's a more or less solved problem on Windows -<br>
it's managing how those shared DLLs are made available to Python<br>
extensions. And *that* is what Unix package managers do for you, and<br>
Windows doesn't have a good solution for (other than "bundle all the<br>
dependent DLLs with the app, or suffer DLL hell").<br></blockquote><div><br></div></span><div>exactly -- but if we consider the python install to be the "app", rather than an individual python bundle, then we _may_ be OK.</div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">PS For a fun exercise, it might be interesting to try breaking conda -<br>
</blockquote><div><br></div></span><div>Windows really is simply broken [1] in this regard -- so I'm quite sure you could break conda -- but it does seem to do a pretty good job of not being broken easily by common uses -- I can't say I know enough about Windows dll finding or conda to know how...</div><div><br></div><div>Oh, and conda is actually broken in this regard on OS-X at this point -- if you compile your own extension in an anaconda environment, it will find a shared lib at compile time that it won't find at run time. -- the conda install process fixes these, but that's a pain when under development -- i.e. you don't want to have to actually install the package with conda to run a test each time you re-build the dll.. (or even change a bit of python code...)</div><div><br></div><div>But in short -- I'm pretty sure there is a way, on all systems, to have a standard way to build extension modules, combined with a standard way to install shared libs, so that a lib can be shared among multiple packages. So the question remains:</div></div></div></div></blockquote><div><br></div><div>There is actually no way to do that on windows without modifying the interpreter somehow. This was somehow discussed a bit at PyCon when talking about windows packaging:</div><div><br></div><div> 1. the simple way to share DLLs across extensions is to put them in the %PATH%, but that's horrible.</div><div> 2. there are ways to put DLLs in a shared directory *not* in the %PATH% since at least windows XP SP2 and above, through the SetDllDirectory API.</div><div><br></div><div>With 2., you still have the issue of DLL hell, which may be resolved through naming and activation contexts. I had a brief chat with Steve where he mentioned that this may be a solution, but he was not 100 % sure IIRC. The main drawback of this solution is that it won't work when inheriting virtual environments (as you can only set a single directory).</div><div><br></div><div>FWIW, we are about to deploy 2. @ Enthought (where we control the python interpreter, so it is much easier for us).</div><div><br></div><div>David</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Is there any point? or is the current approach of statically linking all third party libs the way to go?</div><div><br></div><div>If so, then is there any chance of getting folks to conform to this standard for PyPi hosted binary packages anyway? i.e. the curation problem.</div><div><br></div><div>Personally, I'm on the fence here -- I really want newbies to be able to simply "pip install" as many packages as possible and get a good result when they do it.</div><div><br></div><div>On the other hand, I've found that conda better supports this right now, so it's easier for me to simply use that for my tools.</div><div><br></div><div><br></div><div>-Chris</div><div><br></div><div><br></div><div>[1] My take on dll hell:</div><div> </div><div>a) it's inherently difficult -- which is why Linux provides a system package manager. </div><div><br></div><div>b) however, Windows really does make it MORE difficult than it has to be:</div><div> i) it looks first next the executable</div><div> ii) it also looks on the PATH (rather than a separate DLL_PATH)</div><div> Combine these two, and you have some folks dropping dlls next to their executable, which means they have inadvertently dropped it on the DLL search path for other apps to find it.</div><div><br></div><div>Add to this the (very odd to me) long standing tradition of not putting extensive version numbers in dll file names, and presto: dll hell!</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>-- </div></font></span></div><span class=""><div><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R <a href="tel:%28206%29%20526-6959" value="+12065266959" target="_blank">(206) 526-6959</a> voice<br>7600 Sand Point Way NE <a href="tel:%28206%29%20526-6329" value="+12065266329" target="_blank">(206) 526-6329</a> fax<br>Seattle, WA 98115 <a href="tel:%28206%29%20526-6317" value="+12065266317" target="_blank">(206) 526-6317</a> main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</span></div></div>
<br>_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
<br></blockquote></div><br></div></div>