<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 23, 2014, at 12:36 PM, Chris Barker <<a href="mailto:chris.barker@noaa.gov" class="">chris.barker@noaa.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi folks,<div class=""><br class=""></div><div class="">I'm trying to package up a complex system and would like to do it the correct, modern way.</div><div class=""><br class=""></div><div class="">In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages.</div><div class=""><br class=""></div><div class="">(can you tell I'm building a web service front-end for computational code?)</div><div class=""><br class=""></div><div class="">This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum.</div><div class=""><br class=""></div><div class="">So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools.</div><div class=""><br class=""></div><div class="">I figure we we'll do one (or both) of:</div><div class="">- providing a custom "wheel house" with our packages and the dependencies that are hard to come by</div><div class="">- provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages.</div><div class=""><br class=""></div><div class="">At the moment, I'm working on making conda packages, which brings me to my questions.</div><div class=""><br class=""></div><div class="">I'm a bit confused about the role of setuptools with pip. On the one hand, pip depends of setuptools. On the other hand, pip supposed doesn't "do" eggs, and prefers wheels. But I find that I keep getting eggs whether I want them or not. IN fact, as far as I can tell, the way to get pip to instal something from git repo is:</div><div class=""><br class=""></div><div class="">git+<a href="https://url_to_the_repo.git/#egg=name_of_package" class="">https://url_to_the_repo.git#egg=name_of_package</a><br class=""></div><div class=""><br class=""></div><div class="">why isn't that "wheel=name_of_package"</div><div class=""><br class=""></div><div class="">and will it work if setuptools was not used in the packages setup.py???</div><div class=""><br class=""></div><div class="">Frankly I've generally found setuptools and eggs to be overly heavy weight and annoying -- the only reason I use setuptools at all is that I REALLY like develop mode -- which I can now get with "pip --editable" or does that give me setuptools develop mode anyway, i.e. do I need to have used setuptools.setup for it to work?</div><div class=""><br class=""></div><div class="">So question one: do I need to use setuptools.setup rather than plain old distutils.setup?</div><div class=""><br class=""></div><div class="">Question 2:</div><div class=""><br class=""></div><div class="">What about setuptools: "install_requires"</div><div class=""><br class=""></div><div class="">I generally like the pip requirements.txt approach. It's up to the installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda:</div><div class=""><br class=""></div><div class="">conda, of course, is designed to manage dependencies itself, and those are declared in the conda build meta.yaml file. Note that conda dependencies can have nothign to do with python -- the whole point of conda -- a conda pacakge can depend on any other conda package, including C libs, etc.</div><div class=""><br class=""></div><div class="">But the issue at hand is that conda build doesn't re-invent setup.py -- but rather you generally simple call "setup.py install" from your conda build script. Hence th issue at hand:</div><div class=""><br class=""></div><div class="">Using setuptools.setup, and specifying "install_requires", then kicks in setuptools trying to resolve dependencies I don't want it to deal with.</div><br class="">I read Donald's "setup.py vs requirements.txt", and I guess I get it, but it still seems quite redundant -- I see the point of separating out “abstract dependencies” and “concrete dependencies”. However, the nifty solution of only putting:<div class=""><br class=""></div><div class=""><font face="monospace, monospace" class="">--index-url <a href="https://pypi.python.org/simple/" class="">https://pypi.python.org/simple/</a><br class=""></font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div>in the requirements.txt doesn't work for complex cases. In practice, we find we need to specify version numbers of some dependencies, and have some go strait into a git repo, etc. So we need a messy requirements.txt file.<div class=""><br class=""></div><div class="">And, in fact, I think that's where is belongs -- the goal of the requirements.txt file is so pip can "do the right thing" and go grab everything you need. But, in fact, it also is quite human readable, and serves quite well as the documentation for the "abstract dependencies" as well.</div><div class=""><br class=""></div><div class="">So I think the way to go is to keep requirements in requirements.txt, and leave them out of the setup.py.</div><div class=""><br class=""></div><div class="">Can I dump setuptools.setup, as well??</div><div class=""><br class=""></div><div class="">Sorry, this is a bit rambling and short of clear questions, but I'm trying to get a handle on what best practices really are these days...</div><div class=""><br class=""></div><div class="">-Chris<br class=""></div></div></div></blockquote><div><br class=""></div></div><div class="">I’m going to attempt to read between the lines here a little bit.</div><div class=""><br class=""></div><div class="">The “egg” name is heavily overloaded in setuptools. It is used all over the place for varying sets of related but distinct concepts. The #egg= thing is one of those setuptools concepts where it used that name for something distinct but similar. Ideally it shouldn’t be #egg= or #wheel= but should be #dist= or something similar since it’s neither an egg or a Wheel and there is an open ticket in pip’s issue tracker to do that.</div><div class=""><br class=""></div><div class="">To make a clarification though, pip itself doesn’t depend on setuptools, it can install from Wheels without setuptools being installed on the system at all. It does however rely on setuptools to be installed if it is installing from a sdist. The reason for this is that pip uses setuptools as a build tool, so when it invokes a setup.py it’s “building" that distribution (even if it’s just pure python it needs “built”). However pip does some tricks so that it will always uses setuptools to build the project, regardless of if the project imports setuptools or distutils in their setup.py.</div><div class=""><br class=""></div><div class="">To that aim, install_requires specifies a packages dependencies as well as other metadata for that package, and requirements.txt is just a list of packages to install. The difference is subtle but a requirements.txt isn’t attached to a particular project and the rest of the metadata like name, version, etc.</div><div class=""><br class=""></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">---</div><div class="">Donald Stufft</div><div class="">PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br class=""></body></html>