[Distutils] Role of setuptools and eggs in "modern" distributing...

[Chris Barker]

OK,

I started this thread a while back, as I was getting confused and having
issues with intermixing python, setuptools, pip, and Anaconda / conda.

Now I've figured out where i have my issue:

I'm using an Anaconda distribution at the moment. I want conda to handle
installing my dependencies, etc. for me. OK.

However, I am also developing various python packages -- this means I can't
/ don't want o build and install a conda package for them, rather, I'd like
to use setuptools "develop" mode.

So here's the rub:

When I call "setup.py develop", setuptools apparently looks for the
"install_requires" packages. If it doesn't find them, it goes out and
decided to apparently pip install them: gets the source form pypi,
download, tries to compile, etc....

Even if it does find them already installed, it does some annoying adding
to easy_install.pth magic for them.

This is all why I've been thinking that dependencies do not belong in
setup.py -- but rather outside of setup.py (requirements.txt), and more to
the pint, dependency handling ius a pip (or conda, or...) issue - not one
that should be handled by aw setuptools at build time.

Note that conda build usually simply calls setup.py install as well, so
this can be a problem even there (though I think it usually satisfies the
requirements first, so not so bad)

At the end of the day, however, I think the issue is not so much where you
specify dependencies, but what setuptool develop mode is doing: it should
NOT go an auto-install dependencies, particularly not install-dependencies
(maybe build dependencies are required...)

OK -- I just found the --no-deps option. So I can do what I want, but
still, I don't think it belongs there and all, and certainly would be
better to have the default be no-deps. Let pip (or conda, or...)  handle
that.

Any one running these by hand are be definition doing things by hand, let
them deal with the deps.

OK, I suppose "casual" users may run setup.py install, so that mode _might_
want to auto install dependencies, if somethign has to. But develop mode is
very much for developers, you really don't want this handled there.

-Chris







On Wed, Dec 31, 2014 at 9:41 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Dec 31, 2014 at 9:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> The problem always existed - it's the longstanding conflict between
>> "platform independent, language specific" tooling and "platform specific,
>> language independent" tooling.
>>
>> The former is often preferred on the developer side (since the tools are
>> oriented towards building individual custom applications rather than
>> maintaining a suite of applications written by different groups), while the
>> latter is essential on the system integrator side (since you're dealing
>> with inevitable technology sprawl in infrastructure that evolves over the
>> course of years and decades).
>>
>> One of the best things coming out of the whole "DevOps" notion is the
>> recognition that language independence and platform independence are aimed
>> at solving *different problems*, and that what we need is suitable tools
>> with different roles and responsibilities that interoperate effectively,
>> rather than attempting to provide a single universal tool and having
>> different groups of users yelling at each other for wanting to solve the
>> "wrong" problem.
>>
>> Tools like conda and zc.buildout fit into that landscape by aiming to
>> provide a platform & language independent approach to doing *application*
>> level integration, which tends to have the effect of making them new
>> platforms in their own right.
>>
> Indeed -- thanks for providing a clear way to talk/think about these
> systems.
>
> I guess the trick here is that we want the different level tools to work
> well with each-other.
>
> As conda started with python packages in mind, it does a pretty good job
> with them. But I still find some conflicts between setuptools and conda --
> in particular, if you specify dependencies in setup.py (install_requires),
> it can kind of make a mess of things. conda tries to ignore them, but
> somehow I've had issues, even though I had specified it all in the conda's
> meta.yaml. This is probably a conda bug/issue, but I'm still trying to
> figure out how to best set up a python package so that it can be built
> installed with the "regular" python tools, and also conda...
>
> Use case -- developing in a conda environment -- so I want to install
> dependencies with conda, but the package under development with setuptools
> "develop" mode. (conda does not (yet) have a develop mode that works...)
>
> Oh, and there does seem to be an odd (I think non-fatal) issue with
> setuptools and conda:
>
> I have package A, with a bunch of stuff listed with "install_requires"
>
> I have all these dependencies installed with conda.
>
> When I run setup.py develop, setuptools goes and dumps all the
> dependencies in easy_install.pth.
>
> I have no idea why that is, and it's probably only annoying, rather than a
> problem, but I'm not sure what will happen when I upgrade one of those
> dependencies with conda.
>
>> If you compare them to Linux distributions, then zc.buildout is a
>> platform that practices the Gentoo style approach of building everything
>> from source for each application, while conda uses the more common
>> Debian/Fedora style of defining a binary packaging format that allows a
>> redistributor to handle the build process on behalf of their users.
>>
> indeed -- and conda actually provides (to my disappointment) very little
> in the way of build support -- you need to write platform dependent build
> scripts to actually build the packages.
>
> But it does provide a nice way for me to provide a full "just install
> this" distribution of my complex, ugly, hard to build packages....
>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150107/39c2a3b1/attachment-0001.html>