Role of setuptools and eggs in "modern" distributing...
Hi folks, I'm trying to package up a complex system and would like to do it the correct, modern way. In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages. (can you tell I'm building a web service front-end for computational code?) This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum. So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools. I figure we we'll do one (or both) of: - providing a custom "wheel house" with our packages and the dependencies that are hard to come by - provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages. At the moment, I'm working on making conda packages, which brings me to my questions. I'm a bit confused about the role of setuptools with pip. On the one hand, pip depends of setuptools. On the other hand, pip supposed doesn't "do" eggs, and prefers wheels. But I find that I keep getting eggs whether I want them or not. IN fact, as far as I can tell, the way to get pip to instal something from git repo is: git+https://url_to_the_repo.git#egg=name_of_package why isn't that "wheel=name_of_package" and will it work if setuptools was not used in the packages setup.py??? Frankly I've generally found setuptools and eggs to be overly heavy weight and annoying -- the only reason I use setuptools at all is that I REALLY like develop mode -- which I can now get with "pip --editable" or does that give me setuptools develop mode anyway, i.e. do I need to have used setuptools.setup for it to work? So question one: do I need to use setuptools.setup rather than plain old distutils.setup? Question 2: What about setuptools: "install_requires" I generally like the pip requirements.txt approach. It's up to the installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda: conda, of course, is designed to manage dependencies itself, and those are declared in the conda build meta.yaml file. Note that conda dependencies can have nothign to do with python -- the whole point of conda -- a conda pacakge can depend on any other conda package, including C libs, etc. But the issue at hand is that conda build doesn't re-invent setup.py -- but rather you generally simple call "setup.py install" from your conda build script. Hence th issue at hand: Using setuptools.setup, and specifying "install_requires", then kicks in setuptools trying to resolve dependencies I don't want it to deal with. I read Donald's "setup.py vs requirements.txt", and I guess I get it, but it still seems quite redundant -- I see the point of separating out “abstract dependencies” and “concrete dependencies”. However, the nifty solution of only putting: --index-url https://pypi.python.org/simple/ in the requirements.txt doesn't work for complex cases. In practice, we find we need to specify version numbers of some dependencies, and have some go strait into a git repo, etc. So we need a messy requirements.txt file. And, in fact, I think that's where is belongs -- the goal of the requirements.txt file is so pip can "do the right thing" and go grab everything you need. But, in fact, it also is quite human readable, and serves quite well as the documentation for the "abstract dependencies" as well. So I think the way to go is to keep requirements in requirements.txt, and leave them out of the setup.py. Can I dump setuptools.setup, as well?? Sorry, this is a bit rambling and short of clear questions, but I'm trying to get a handle on what best practices really are these days... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Dec 23, 2014, at 12:36 PM, Chris Barker <chris.barker@noaa.gov> wrote:
Hi folks,
I'm trying to package up a complex system and would like to do it the correct, modern way.
In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages.
(can you tell I'm building a web service front-end for computational code?)
This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum.
So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools.
I figure we we'll do one (or both) of: - providing a custom "wheel house" with our packages and the dependencies that are hard to come by - provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages.
At the moment, I'm working on making conda packages, which brings me to my questions.
I'm a bit confused about the role of setuptools with pip. On the one hand, pip depends of setuptools. On the other hand, pip supposed doesn't "do" eggs, and prefers wheels. But I find that I keep getting eggs whether I want them or not. IN fact, as far as I can tell, the way to get pip to instal something from git repo is:
git+https://url_to_the_repo.git#egg=name_of_package <https://url_to_the_repo.git/#egg=name_of_package>
why isn't that "wheel=name_of_package"
and will it work if setuptools was not used in the packages setup.py???
Frankly I've generally found setuptools and eggs to be overly heavy weight and annoying -- the only reason I use setuptools at all is that I REALLY like develop mode -- which I can now get with "pip --editable" or does that give me setuptools develop mode anyway, i.e. do I need to have used setuptools.setup for it to work?
So question one: do I need to use setuptools.setup rather than plain old distutils.setup?
Question 2:
What about setuptools: "install_requires"
I generally like the pip requirements.txt approach. It's up to the installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda:
conda, of course, is designed to manage dependencies itself, and those are declared in the conda build meta.yaml file. Note that conda dependencies can have nothign to do with python -- the whole point of conda -- a conda pacakge can depend on any other conda package, including C libs, etc.
But the issue at hand is that conda build doesn't re-invent setup.py -- but rather you generally simple call "setup.py install" from your conda build script. Hence th issue at hand:
Using setuptools.setup, and specifying "install_requires", then kicks in setuptools trying to resolve dependencies I don't want it to deal with.
I read Donald's "setup.py vs requirements.txt", and I guess I get it, but it still seems quite redundant -- I see the point of separating out “abstract dependencies” and “concrete dependencies”. However, the nifty solution of only putting:
--index-url https://pypi.python.org/simple/ <https://pypi.python.org/simple/>
in the requirements.txt doesn't work for complex cases. In practice, we find we need to specify version numbers of some dependencies, and have some go strait into a git repo, etc. So we need a messy requirements.txt file.
And, in fact, I think that's where is belongs -- the goal of the requirements.txt file is so pip can "do the right thing" and go grab everything you need. But, in fact, it also is quite human readable, and serves quite well as the documentation for the "abstract dependencies" as well.
So I think the way to go is to keep requirements in requirements.txt, and leave them out of the setup.py.
Can I dump setuptools.setup, as well??
Sorry, this is a bit rambling and short of clear questions, but I'm trying to get a handle on what best practices really are these days...
-Chris
I’m going to attempt to read between the lines here a little bit. The “egg” name is heavily overloaded in setuptools. It is used all over the place for varying sets of related but distinct concepts. The #egg= thing is one of those setuptools concepts where it used that name for something distinct but similar. Ideally it shouldn’t be #egg= or #wheel= but should be #dist= or something similar since it’s neither an egg or a Wheel and there is an open ticket in pip’s issue tracker to do that. To make a clarification though, pip itself doesn’t depend on setuptools, it can install from Wheels without setuptools being installed on the system at all. It does however rely on setuptools to be installed if it is installing from a sdist. The reason for this is that pip uses setuptools as a build tool, so when it invokes a setup.py it’s “building" that distribution (even if it’s just pure python it needs “built”). However pip does some tricks so that it will always uses setuptools to build the project, regardless of if the project imports setuptools or distutils in their setup.py. To that aim, install_requires specifies a packages dependencies as well as other metadata for that package, and requirements.txt is just a list of packages to install. The difference is subtle but a requirements.txt isn’t attached to a particular project and the rest of the metadata like name, version, etc. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Dec 23, 2014 at 10:37 AM, Donald Stufft <donald@stufft.io> wrote:
I’m going to attempt to read between the lines here a little bit.
Thank you -- you did an excellent job of capturing the gestalt of my confusion ! The “egg” name is heavily overloaded in setuptools. It is used all over the
place for varying sets of related but distinct concepts. The #egg= thing is one of those setuptools concepts where it used that name for something distinct but similar. Ideally it shouldn’t be #egg= or #wheel= but should be #dist= or something similar since it’s neither an egg or a Wheel and there is an open ticket in pip’s issue tracker to do that.
OK, that clears it up. Though I still get egg-info files all over the place -- not sure why that annoys me ;-)
To make a clarification though, pip itself doesn’t depend on setuptools, it can install from Wheels without setuptools being installed on the system at all. It does however rely on setuptools to be installed if it is installing from a sdist. The reason for this is that pip uses setuptools as a build tool, so when it invokes a setup.py it’s “building" that distribution (even if it’s just pure python it needs “built”). However pip does some tricks so that it will always uses setuptools to build the project, regardless of if the project imports setuptools or distutils in their setup.py.
Ah -- so pip needs to use setuptools to build, but a package doesn't have to explicitly use it in its setup.py. To that aim, install_requires specifies a packages dependencies as well as
other metadata for that package, and requirements.txt is just a list of packages to install. The difference is subtle but a requirements.txt isn’t attached to a particular project and the rest of the metadata like name, version, etc.
hmm...I agree, but often shipped alongside setup.py -- kind of like the fact that the name "setup.py" is a conventions rather than a spec, but expected all over the place. On Tue, Dec 23, 2014 at 11:17 AM, Marcus Smith <qwcode@gmail.com> wrote: which I can now get with "pip --editable" or does that give me setuptools
develop mode anyway
"-e" uses setuptools develop mode. OK -- though it sounds like pip would do that whether or not I used setuptools in the setup.py. the main reason for setuptools is for "install_requires", which is
fundamental to pip dependency resolution. but in general, it offers more features and it's more maintained than pure distutils. The standard advice is to use setuptools over distutils.
OK -- still not clear how install_requires plays with conda - but it's common enough that I think conda simply ignores it (though not silently) The Packaging User Guide has a breakdown of install_requires vs
requirements files.
https://packaging.python.org/en/latest/technical.html#install-requires-vs-re...
In brief, requirements files are usually for a whole "environment", whereas install_requires is for a project.
A note about terminology here (both in this email and The Packaging User Guide) -- it seems to me that install_requires is about requirements for a "package" not a "project", and that, in fact, requirements.txt is best used for "projects". I guess the distinction may be that a "package" has a setup.py, whereas a project is somethign you are building that requires perhaps a stack of unrelated packages. So you can say : if you want to run my application, run: pip install -r requirements.txt first. "install_requires" is critical when publishing projects to PyPI. Good to know -- I may need to go there art some point. So I'll go with setuptools and install_requires, and see how it all goes. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
A note about terminology here (both in this email and The Packaging User Guide) -- it seems to me that install_requires is about requirements for a "package" not a "project",
well, read through the PyPUG glossary: https://packaging.python.org/en/latest/glossary.html a "project" is anything with an associated setup.py (which will often have "install_requires" metadata) a "package" (listed as a "distribution package" in the glossary) is the distribution of a certain "release" of a "project"
I guess the distinction may be that a "package" has a setup.py, whereas a project is somethign you are building that requires perhaps a stack of unrelated packages.
Above, I used the word "environment", which was just short hand for the whole set of installed packages on the Python path for the interpreter used by your application. This is often literally a "virtual environment" created by virtualenv. To me, the distinction is over which project *owns* the whole environment, i.e what is the top-level project that the environment exists for. Requirements files are typically associated with the project that owns the environment. Marcus
On 25 Dec 2014 06:51, "Marcus Smith" <qwcode@gmail.com> wrote:
Above, I used the word "environment", which was just short hand for the
whole set of installed packages on the Python path for the interpreter used by your application. This is often literally a "virtual environment" created by virtualenv.
To me, the distinction is over which project *owns* the whole
environment, i.e what is the top-level project that the environment exists for.
Requirements files are typically associated with the project that owns
the environment.
From my perspective, it's mainly a question of "Who is responsible for defining this metadata?".
setup.py -> always the project publisher (and getting too specific annoys system integrators) requirements.txt -> always the system integrator (and you can be as specific as you like) Web applications just blur the line a lot, as the publisher and integrator are often the same person or group. For integration into Linux distros and other larger systems though, we prefer the first kind of metadata, as that's usually a bit more lenient on the acceptable versions of dependencies. Cheers, Nick.
Marcus
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
git+https://url_to_the_repo.git#egg=name_of_package
why isn't that "wheel=name_of_package"
the "egg" part here has nothing to do with eggs. just a vestige of another time. see https://github.com/pypa/pip/issues/1265 and will it work if setuptools was not used in the packages setup.py???
yes, it would work. which I can now get with "pip --editable" or does that give me setuptools
develop mode anyway
"-e" uses setuptools develop mode. So question one: do I need to use setuptools.setup rather than plain old
distutils.setup?
the main reason for setuptools is for "install_requires", which is fundamental to pip dependency resolution. but in general, it offers more features and it's more maintained than pure distutils. The standard advice is to use setuptools over distutils. I generally like the pip requirements.txt approach. It's up to the
installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda:
The Packaging User Guide has a breakdown of install_requires vs requirements files. https://packaging.python.org/en/latest/technical.html#install-requires-vs-re... In brief, requirements files are usually for a whole "environment", whereas install_requires is for a project. "install_requires" is critical when publishing projects to PyPI. Even if you're not publishing, install_requires is helpful to safely attempt an upgrade of your dependencies. Instead of installing your app from a frozen requirements file, you would install normally in a clean environment (based on install_requires) and let any upgrades occur, and then you can re-freeze your requirements. But if you are doing mostly vcs installs in your requirements file, then it's true that maintaining install_requires can be somewhat pointless, although I would still say it's helpful to keep track of each project's first level abstract dependencies are, even if the install_requires declarations aren't being used. Marcus
Chris Barker schreef op 23-12-14 om 18:36:
I'm trying to package up a complex system and would like to do it the correct, modern way.
In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages.
(can you tell I'm building a web service front-end for computational code?)
This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum.
So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools.
Well, we're in a bit of the same boat here. We make django websites, which means pretty much well-behaved setup.py-using pure python stuff. The websites are heavy users of numpy/scipy/pandas/matplotlib and of the geo packages like gdal/mapnik/pyproj. Ouch. The combination we use now is to use buildout (instead of pip) in combination with the "syseggrecipe" (https://pypi.python.org/pypi/syseggrecipe) buildout add-on. Syseggrecipe allows us to depend explicitly on **system** packages for gdal/matplotlib/mapnik/scipy/numpy. Which takes care of most of the compilation headaches. The rest of the huge python package stack is just pulled in regularly by buildout (using setuptools). Just throwing this into the mix as a potential solution. Note that you'll get to explain buildout to your users, but... :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On Tue, Dec 30, 2014 at 2:21 PM, Reinout van Rees <reinout@vanrees.org> wrote:
Well, we're in a bit of the same boat here. We make django websites, which means pretty much well-behaved setup.py-using pure python stuff.
The websites are heavy users of numpy/scipy/pandas/matplotlib and of the geo packages like gdal/mapnik/pyproj. Ouch.
yup -- this is pretty much out stack (though pyramid in our case...) Funny, though, as coming from my background, I see it as a scipy stack app with a little web stuff, rather than a web app that needs some scipy stuff ;-) But the core problem here is that the scipy folks have been going to conda and enthought to solve their pacakgeing problems, and the web folks have been doing pip, and maybe buildout -- so you get a bit of mess when you mix them.
The combination we use now is to use buildout (instead of pip) in combination with the "syseggrecipe" (https://pypi.python.org/pypi/ syseggrecipe) buildout add-on. Syseggrecipe allows us to depend explicitly on **system** packages for gdal/matplotlib/mapnik/scipy/numpy. Which takes care of most of the compilation headaches.
well, it sounds like you are simple punting -- passing off all the complex stuff to the system, which may work well if the system is up to date linux with the packages you need available, but pretty worthless on a Mac or Windows box. The scipy folks have been doing a pretty good job lately keeping up with wheels, but there's still a big hole there for the geo stuff.(GDAL, Shapely, Fiona) So I've been looking at going the Anaconda route -- it provides the hard stuff, though it turns out it's a bit ugly when using it as a development environment for extensions liked against libs that are both in the system and Anaconda provided. Antoine Pitrou wrote:
Note you can use pip inside conda environments, which works quite well at least for pure Python packages (start with "conda install pip").
True -- though it gets a bit ugly, as then conda doesn't know about the packages, so switching environments is a mess, and conda can't manage the dependencies. So not ideal. I've actually spend the last two days writing a script that auto-grabs packages from PyPI, builds conda packages out of them, and then uploads them to binstar -- so we can have all our dependencies supported by conda. I'd love it if Continuum would build a "pip bridge" on binstar that would do all that automagically if you request a pip-installable package from binstar.
Just throwing this into the mix as a potential solution. Note that you'll get to explain buildout to your users, but... :-)
yup -- not sure I want to go there yet, either... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris Barker schreef op 31-12-14 om 01:42:
The combination we use now is to use buildout (instead of pip) in combination with the "syseggrecipe" (https://pypi.python.org/pypi/__syseggrecipe <https://pypi.python.org/pypi/syseggrecipe>) buildout add-on. Syseggrecipe allows us to depend explicitly on **system** packages for gdal/matplotlib/mapnik/scipy/__numpy. Which takes care of most of the compilation headaches.
well, it sounds like you are simple punting -- passing off all the complex stuff to the system, which may work well if the system is up to date linux with the packages you need available, but pretty worthless on a Mac or Windows box.
You're completely right! We pass on the should-be-compiled packages to the OS, in our case ubuntu. ... which means that I'm using vagrant/vmware on my mac with an ubuntu vm on it. ... which means that at a certain point you need a PPA for a newer version for GDAL which brings in other new stuff which breaks after half a year... So: not ideal :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"
On 31 Dec 2014 10:43, "Chris Barker" <chris.barker@noaa.gov> wrote:
But the core problem here is that the scipy folks have been going to
conda and enthought to solve their pacakgeing problems, and the web folks have been doing pip, and maybe buildout -- so you get a bit of mess when you mix them. The problem always existed - it's the longstanding conflict between "platform independent, language specific" tooling and "platform specific, language independent" tooling. The former is often preferred on the developer side (since the tools are oriented towards building individual custom applications rather than maintaining a suite of applications written by different groups), while the latter is essential on the system integrator side (since you're dealing with inevitable technology sprawl in infrastructure that evolves over the course of years and decades). One of the best things coming out of the whole "DevOps" notion is the recognition that language independence and platform independence are aimed at solving *different problems*, and that what we need is suitable tools with different roles and responsibilities that interoperate effectively, rather than attempting to provide a single universal tool and having different groups of users yelling at each other for wanting to solve the "wrong" problem. Tools like conda and zc.buildout fit into that landscape by aiming to provide a platform & language independent approach to doing *application* level integration, which tends to have the effect of making them new platforms in their own right. If you compare them to Linux distributions, then zc.buildout is a platform that practices the Gentoo style approach of building everything from source for each application, while conda uses the more common Debian/Fedora style of defining a binary packaging format that allows a redistributor to handle the build process on behalf of their users. Cheers, Nick. P.S. conda's actually one of the projects on my preliminary short list for consideration as a possible userspace package management tool in Fedora: https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManag...
On Wed, Dec 31, 2014 at 9:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The problem always existed - it's the longstanding conflict between "platform independent, language specific" tooling and "platform specific, language independent" tooling.
The former is often preferred on the developer side (since the tools are oriented towards building individual custom applications rather than maintaining a suite of applications written by different groups), while the latter is essential on the system integrator side (since you're dealing with inevitable technology sprawl in infrastructure that evolves over the course of years and decades).
One of the best things coming out of the whole "DevOps" notion is the recognition that language independence and platform independence are aimed at solving *different problems*, and that what we need is suitable tools with different roles and responsibilities that interoperate effectively, rather than attempting to provide a single universal tool and having different groups of users yelling at each other for wanting to solve the "wrong" problem.
Tools like conda and zc.buildout fit into that landscape by aiming to provide a platform & language independent approach to doing *application* level integration, which tends to have the effect of making them new platforms in their own right.
Indeed -- thanks for providing a clear way to talk/think about these systems. I guess the trick here is that we want the different level tools to work well with each-other. As conda started with python packages in mind, it does a pretty good job with them. But I still find some conflicts between setuptools and conda -- in particular, if you specify dependencies in setup.py (install_requires), it can kind of make a mess of things. conda tries to ignore them, but somehow I've had issues, even though I had specified it all in the conda's meta.yaml. This is probably a conda bug/issue, but I'm still trying to figure out how to best set up a python package so that it can be built installed with the "regular" python tools, and also conda... Use case -- developing in a conda environment -- so I want to install dependencies with conda, but the package under development with setuptools "develop" mode. (conda does not (yet) have a develop mode that works...) Oh, and there does seem to be an odd (I think non-fatal) issue with setuptools and conda: I have package A, with a bunch of stuff listed with "install_requires" I have all these dependencies installed with conda. When I run setup.py develop, setuptools goes and dumps all the dependencies in easy_install.pth. I have no idea why that is, and it's probably only annoying, rather than a problem, but I'm not sure what will happen when I upgrade one of those dependencies with conda.
If you compare them to Linux distributions, then zc.buildout is a platform that practices the Gentoo style approach of building everything from source for each application, while conda uses the more common Debian/Fedora style of defining a binary packaging format that allows a redistributor to handle the build process on behalf of their users.
indeed -- and conda actually provides (to my disappointment) very little in the way of build support -- you need to write platform dependent build scripts to actually build the packages. But it does provide a nice way for me to provide a full "just install this" distribution of my complex, ugly, hard to build packages.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
OK, I started this thread a while back, as I was getting confused and having issues with intermixing python, setuptools, pip, and Anaconda / conda. Now I've figured out where i have my issue: I'm using an Anaconda distribution at the moment. I want conda to handle installing my dependencies, etc. for me. OK. However, I am also developing various python packages -- this means I can't / don't want o build and install a conda package for them, rather, I'd like to use setuptools "develop" mode. So here's the rub: When I call "setup.py develop", setuptools apparently looks for the "install_requires" packages. If it doesn't find them, it goes out and decided to apparently pip install them: gets the source form pypi, download, tries to compile, etc.... Even if it does find them already installed, it does some annoying adding to easy_install.pth magic for them. This is all why I've been thinking that dependencies do not belong in setup.py -- but rather outside of setup.py (requirements.txt), and more to the pint, dependency handling ius a pip (or conda, or...) issue - not one that should be handled by aw setuptools at build time. Note that conda build usually simply calls setup.py install as well, so this can be a problem even there (though I think it usually satisfies the requirements first, so not so bad) At the end of the day, however, I think the issue is not so much where you specify dependencies, but what setuptool develop mode is doing: it should NOT go an auto-install dependencies, particularly not install-dependencies (maybe build dependencies are required...) OK -- I just found the --no-deps option. So I can do what I want, but still, I don't think it belongs there and all, and certainly would be better to have the default be no-deps. Let pip (or conda, or...) handle that. Any one running these by hand are be definition doing things by hand, let them deal with the deps. OK, I suppose "casual" users may run setup.py install, so that mode _might_ want to auto install dependencies, if somethign has to. But develop mode is very much for developers, you really don't want this handled there. -Chris On Wed, Dec 31, 2014 at 9:41 AM, Chris Barker <chris.barker@noaa.gov> wrote:
On Wed, Dec 31, 2014 at 9:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The problem always existed - it's the longstanding conflict between "platform independent, language specific" tooling and "platform specific, language independent" tooling.
The former is often preferred on the developer side (since the tools are oriented towards building individual custom applications rather than maintaining a suite of applications written by different groups), while the latter is essential on the system integrator side (since you're dealing with inevitable technology sprawl in infrastructure that evolves over the course of years and decades).
One of the best things coming out of the whole "DevOps" notion is the recognition that language independence and platform independence are aimed at solving *different problems*, and that what we need is suitable tools with different roles and responsibilities that interoperate effectively, rather than attempting to provide a single universal tool and having different groups of users yelling at each other for wanting to solve the "wrong" problem.
Tools like conda and zc.buildout fit into that landscape by aiming to provide a platform & language independent approach to doing *application* level integration, which tends to have the effect of making them new platforms in their own right.
Indeed -- thanks for providing a clear way to talk/think about these systems.
I guess the trick here is that we want the different level tools to work well with each-other.
As conda started with python packages in mind, it does a pretty good job with them. But I still find some conflicts between setuptools and conda -- in particular, if you specify dependencies in setup.py (install_requires), it can kind of make a mess of things. conda tries to ignore them, but somehow I've had issues, even though I had specified it all in the conda's meta.yaml. This is probably a conda bug/issue, but I'm still trying to figure out how to best set up a python package so that it can be built installed with the "regular" python tools, and also conda...
Use case -- developing in a conda environment -- so I want to install dependencies with conda, but the package under development with setuptools "develop" mode. (conda does not (yet) have a develop mode that works...)
Oh, and there does seem to be an odd (I think non-fatal) issue with setuptools and conda:
I have package A, with a bunch of stuff listed with "install_requires"
I have all these dependencies installed with conda.
When I run setup.py develop, setuptools goes and dumps all the dependencies in easy_install.pth.
I have no idea why that is, and it's probably only annoying, rather than a problem, but I'm not sure what will happen when I upgrade one of those dependencies with conda.
If you compare them to Linux distributions, then zc.buildout is a platform that practices the Gentoo style approach of building everything from source for each application, while conda uses the more common Debian/Fedora style of defining a binary packaging format that allows a redistributor to handle the build process on behalf of their users.
indeed -- and conda actually provides (to my disappointment) very little in the way of build support -- you need to write platform dependent build scripts to actually build the packages.
But it does provide a nice way for me to provide a full "just install this" distribution of my complex, ugly, hard to build packages....
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 8 January 2015 at 00:20, Chris Barker <chris.barker@noaa.gov> wrote:
When I call "setup.py develop", setuptools apparently looks for the "install_requires" packages. If it doesn't find them, it goes out and decided to apparently pip install them: gets the source form pypi, download, tries to compile, etc....
Do you get any better results if you use "pip install -e ."? I'm not sure you will, but it might mean that pip does the dependency management for you rather than setuptools, and as long as conda records dependency information in a way that pip recognises that might help.
Even if it does find them already installed, it does some annoying adding to easy_install.pth magic for them.
Unfortunately, that's just the way develop mode (pip's editable mode) works. It sounds to me that this is more of a conda issue - it doesn't seem to be creating standard distribution metadata to allow pip/setuptools to recognise what it has installed, and it doesn't provide its own equivalent of editable/develop mode (which would allow you to work purely within the conda framework and avoid these issues). Have you tried asking the conda folks about these issues? I thought that when I briefly tried it out, it did install packages in a way that pip could recognise - so maybe something has changed? Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/07/2015 07:20 PM, Chris Barker wrote:
OK -- I just found the --no-deps option. So I can do what I want, but still, I don't think it belongs there and all, and certainly would be better to have the default be no-deps. Let pip (or conda, or...) handle that.
Unlike pip (which provides no API), setuptools is a library, used by e.g. zc.buildout. The behavior which you find objectionable (installing dependencies needed to satisfy 'install_requires') is a core part of what setuptools *does* -- ripping it out (or even changing the default) would break nearly every other user of setuptools in the world. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlSupRIACgkQ+gerLs4ltQ5/fgCfZTk7b9+eVwLqJcztO1RggQJ2 XxkAoM0gYO+vV/sOrIcVqPFbCRVmAi+o =gRN+ -----END PGP SIGNATURE-----
On Tue, 23 Dec 2014 09:36:36 -0800 Chris Barker <chris.barker@noaa.gov> wrote:
So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools.
I figure we we'll do one (or both) of: - providing a custom "wheel house" with our packages and the dependencies that are hard to come by - provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages.
At the moment, I'm working on making conda packages, which brings me to my questions.
Note you can use pip inside conda environments, which works quite well at least for pure Python packages (start with "conda install pip"). Regards Antoine.
participants (8)
-
Antoine Pitrou -
Chris Barker -
Donald Stufft -
Marcus Smith -
Nick Coghlan -
Paul Moore -
Reinout van Rees -
Tres Seaver