On May 16, 2015, at 7:09 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Sat, May 16, 2015 at 3:03 PM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting,

Huh? I"'m confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-)

but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment….

I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages. A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 16, 2015, at 8:50 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Sat, May 16, 2015 at 4:16 PM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote:

...

On Sat, May 16, 2015 at 3:03 PM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote: There are a few other benefits, but that’s not anything that are inherent in the two different approaches, it’s just things that conda has that pip is planning on getting,

Huh? I"'m confused -- didn't we just have a big thread about how pip+wheel probably ISN'T going to handle shared libs -- that those are exactly what conda packages do provide -- aside from R and Erlange, anyway :-)

but it's not the packages in this case that we need -- it's the environment -- and I can't see how pip is going to provide a conda environment….

I never said pip was going to provide an environment, I said the main benefit conda has over pip, which pip will most likely not get in any reasonable time frame, is that it handles things which are not Python packages.

well, I got a bit distraced by Erlang and R -- i.e. things that have nothing to do with python packages.

libxml, on the other hand, is a lib that one might want to use with a python package -- so a bit more apropos here.

But my confusion was about: "things that conda has that pip is planning on getting" -- what are those things? Any of the stuff that conda has that really useful like handling shared libs, pip is NOT getting -- yes?

The ability to resolve dependencies with static metadata is the major one that comes to my mind that’s specific to pip. The ability to have better build systems besides distutils/setuptools is a more ecosystem level one but that’s something we’ll get too. As far as shared libs… beyond what’s already possible (sticking a shared lib inside of a python project and having libraries load that .dll explicitly) it’s not currently on the road map and may never be. I hesitate to say never because it’s obviously a problem that needs solved and if the Python ecosystem solves it (specific to shared libraries, not whole runtimes or other languages or what have you) then that would be a useful thing. I think we have lower hanging fruit that we need to deal with before something like that is even possibly to be on the radar though (if we ever put it on the radar).

A shared library is not a Python package so I’m not sure what this message is even saying? ``pip install lxml-from-conda`` is just going to flat out break because pip won’t install the libxml2 shared library.

exactly -- if you're going to install a shared lib, you need somewhere to put it -- and that's what a conda environment provides.

Trying not to go around in circles, but python _could_ provide a standard place in which to put shared libs -- and then pip _could_ provide a way to manage them. That would require dealing with that whole binary API problem, so we probably won't do it. I'm not sure what the point of contention is here:

I think it would be useful to have a way to manage shared libs solely for python packages to use -- and it would be useful to that way to be part of the standard python ecosytem. Others may not think it would be useful enough to be worth the pain in the neck it would be.

And that's what the nifty conda packages continuum (and others) have built could provide -- those shared libs that are built in a compatible way with a python binary. After all, pure python packages are no problem, compiled python packages without any dependencies are little problem. The hard part is those darn third party libs.

conda also provides a way to mange all sorts of other stuff that has nothing to do with python, but I'm guessing that's not what continuum would like to contribute to pypi….

I guess I’m confused what the benefit of making pip able to install a conda package would be. If Python adds someplace for shared libs to go then we could just add shared lib support to Wheels, it’s just another file type so that’s not a big deal. The hardest part is dealing with ABI compatibility. However, given the current state of things, what’s the benefit of being able to do ``pip install conda-lxml``? Either it’s going to flat out break or you’re going to have to do ``conda install libxml2`` first, and if you’re doing ``conda install libxml2`` first then why not just do ``conda install lxml``? I view conda the same way I view apt-get, yum, Chocolatey, etc. It provides an environment and you can install a Python package into that environment, but that pip shouldn’t know how to install a .deb or a .rpm or a conda package because those packages rely on specifics to that environment and Python packages can’t. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 17 May 2015 at 19:05, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 17 May 2015 at 05:04, David Mertz <dmertz@continuum.io> wrote:

...

What would be better as a user experience would be to let users do this:

% pip install --upgrade pip % pip install some_conda_package

This gets the respective role of the two tools reversed - it's like my asking for "pip install some_fedora_rpm" to be made to work.

However, having conda use "pip install" in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly.

So you're generating a record file and then copying it into place in the rpm's? I'm not sure if thats an external-stable-thing yet, perhaps it is. The distutils data directory with the METADATA etc is what pkg_resources needs (and is preserve on Debian and derived systems already). I see no reason why we can't make this a part of the contract, but we should at least get it in a PEP first, no? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud

On 18 May 2015 07:32, "Chris Barker" <chris.barker@noaa.gov> wrote:

On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

...

% pip install --upgrade pip % pip install some_conda_package

This gets the respective role of the two tools reversed - it's like my asking for "pip install some_fedora_rpm" to be made to work.

I agree here -- I was thinking there was some promise in a

conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice.

The trick is that conda packages for the hard-to-build python packages

(the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that.

And this is a trick, because while I have some ideas for supporting

just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together.

Continuum has a bunch of smart people, though.

...

However, having conda use "pip install" in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly.

Hmm -- that's something ot look into -- you can put essentially anything

into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the "offical" conda packages is the standard).

But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ???

I see pip has handling the dependency resolution, and finding and

downloading of packages part of the problem -- conda does those already.

So what would using pip inside a conda build script buy you that using

setuptools does not? Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default. However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip & attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently. Cheers, Nick.

On May 17, 2015, at 8:12 PM, Robert Collins <robertc@robertcollins.net> wrote:

On 17 May 2015 5:05 pm, "Nick Coghlan" <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:

...

On 18 May 2015 07:32, "Chris Barker" <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote:

...

On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:

...

...

% pip install --upgrade pip % pip install some_conda_package

This gets the respective role of the two tools reversed - it's like my asking for "pip install some_fedora_rpm" to be made to work.

I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice.

The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that.

And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together.

Continuum has a bunch of smart people, though.

...

However, having conda use "pip install" in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly.

Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the "offical" conda packages is the standard).

But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ???

I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already.

So what would using pip inside a conda build script buy you that using setuptools does not?

Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ <https://www.python.org/dev/peps/pep-0376/> compliant metadata by default.

However, looking at the current packaging policy, I think I misremembered the situation - it looks like we *discussed* recommending indirection via pip & attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently.

That glue is actually very shallow...I think we should rip it out of pip and perhaps put it in setuptools. It's about building, not installing.

So a benefit of using pip instead of setuptools is that as we move to a pluggable build system pip can act as a unified fronted to multiple build systems, instead of every system having to implement each pluggable build system themselves. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 18 May 2015 07:32, "Chris Barker" <chris.barker@noaa.gov> wrote:

...

On Sun, May 17, 2015 at 12:05 AM, Nick Coghlan <ncoghlan@gmail.com>

wrote:

...

...

...

% pip install --upgrade pip % pip install some_conda_package

This gets the respective role of the two tools reversed - it's like my asking for "pip install some_fedora_rpm" to be made to work.

I agree here -- I was thinking there was some promise in a conda_package_to_wheel converter though. It would, of course, only work in a subset of conda packages, but would be nice.

The trick is that conda packages for the hard-to-build python packages (the ones we care about) often (always?) depend on conda packages for dynamic libs, and pip+wheel have no support for that.

And this is a trick, because while I have some ideas for supporting just-for-python dynamic libs, conda's are not just-for-python -- so that might be hard to mash together.

Continuum has a bunch of smart people, though.

...

However, having conda use "pip install" in its build scripts so that it reliably generates pip compatible installation metadata would be a possibility worth discussing - that's what we've started doing in Fedora, so that runtime utilities like pkg_resources can work correctly.

Hmm -- that's something ot look into -- you can put essentially anything into a conda bulid script -- so this would be a matter of convention, rather than tooling. (of course the conventions used by Continuum for the "offical" conda packages is the standard).

But I'm confused as to the roles of pip vs setuptools, vs wheel, vs ???

I see pip has handling the dependency resolution, and finding and downloading of packages part of the problem -- conda does those already.

So what would using pip inside a conda build script buy you that using setuptools does not?

Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default.

Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this. David However, looking at the current packaging policy, I think I misremembered

the situation - it looks like we *discussed* recommending indirection via pip & attaining PEP 376 compliance, but haven't actually moved forward with the idea yet. That makes sense, since pursuing it would have been gated on ensurepip, and the Python 3 migration has been higher priority recently.

Cheers, Nick.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

A member of the conda dev team could answer this better than I, but I've used enough to _think_ I understand the basics: On Mon, May 18, 2015 at 3:30 AM, Paul Moore <p.f.moore@gmail.com> wrote:

One way forward in terms of building wheels is to use any build process you like to do an isolated build (I think it's --root that distutils uses for this sort of thing) and then use distlib to build a wheel from the resulting directory structure (or do it by hand, it's not much more than a bit of directory rearrangement and zipping things up).

That process can be automated any way you like - although ideally via something general, so projects don't have to reinvent the wheel every time.

sure -- you can put virtually anything in a conda build script. what conda build does is more or less: * setup an isolated environment with some handy environment variables for things like the python interpreter, etc. * run your build script * package up whatever got built. If processes like conda then used wheels as their input for building

packages, the wheels could *also* be published

I'm not sure it's any easier to build a wheel, then make a conda package out of it, than to build a conda package, and then make a wheel out of it. Or have your build scrit build a wheel, and then independently build a conda package. In any case, the resulting wheel would depend on an environment like the one set up by conda build -- and that is an environment with all the dependencies installed -- which is where this gets ugly. [remember, making the wheel itself it the easy part]

not least, does the way conda uses shared libraries make going via wheels impossible (or at least make the wheels unusable without conda's support for installing non-Python shared libraries)?

Pretty much, yes. conda provides a way to package up and manage arbitrary stuff -- in this case, that would be non-python dependencies -- i.e. shared libs. So you can say that my_python_package depends on this_c_lib, and as long as you, or someone else has made a conda package for this_c_lib, then all is well. But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc. So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here. We need to remember what leveraging conda would buy us: conda doesn't actually make it any easier to build anything -- you need a platform-specific build script to build a conda package. conda does provide a way to manage non-python dependencies -- but that doesn't buy you anything unless you are using conda to manage your system anyway. conda DOES provide a community of people figuring out how to build complex packages, and building them, and putting them up for public dissemination. So the thing that leveraging conda can do is reduce the need for a lot of duplicated effort. And that effort is almost entirely about those third part libs -- after all, a compiled extension that has no dependencies is easy to build and put on PyPi. (OK, there is still a bit of duplicated effort in making the builds themselves on multiple platforms -- but with CI systems, that's not huge) An example: I have a complex package that not depends on all sorts of hard-to-build python packages, but also has its own C++ code that depends on the netcdf4 library. Which in turn, depends on the hdf5 lib, which depends on libcurl, and zlib, and (I think one or two others). Making binary wheels of this requires me to figure out how to build all those deps on at least two platforms (Windows being the nightmare, but OS-X is not trivial, too, if I want it to match the python.org build, and support older OS versions than I am running) Then I could have a nice binary wheel that my users can pip install and away they go. But: 1) They also need the Py_netCDF4 package, which may or may not be easy to find. If not -- they need to go through all that build hell. Then they have a package that is using a bunch of the same shared libs as mine -- and hopefully no version conflicts... 2) my package is under development -- what I really want is for it to be easy for my users to build from source, so they can keep it all up to date from the repo. Now they need to get a development version of all those libs up and running on their machines -- a heavy lift for a lot of people. So now - "use Anaconda" is a pretty good solution -- it provides all the libs I need, and someone else has figured out how to build them on the platforms I care about. But it would be nice if we could find a way for the "standard" python toolchain could support this. NOTE: as someone suggested on this list was to provide (outside of PyPi + pip), a set of static libs all built and configured, so that I could say: "install these libs from some_other_place, then you can build and run my code" -- that may be doable with a community effort and no change in the tooling, but it has not so far. In short: what conda and the conda community provide is python-compatible third party libs. We can't take advantage of that with pip+PyPi unless we find a way to support third party libs with those tools. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 19 May 2015 at 10:55, Paul Moore <p.f.moore@gmail.com> wrote:

...

But python, setuptools, pip, wheel, etc. don't have a way to handle that shared lib as a dependency -- no standard way where to put it, no way to package it as a wheel, etc.

So the way to deal with this with wheels is to statically link everything. But that's not how conda pa cakges are built, so no way to leverage conda here.

Thanks for the explanation. So, in effect, conda-as-a-platform defines a (somewhat) incompatible platform for running Python, which can use wheels just as python.org Python can, but which uses conda-as-an-installer as its package manager (much like RPM or apt on Unix).

The downside of this is that wheels built for conda (assuming that it's OK to link with shared libs) are not compatible with python.org builds (as those shared libs aren't available) and that difference isn't reflected in the wheel ABI tags (and it's not particularly clearly understood by the community, it seems). So publishing conda-based wheels on PyPI would be a bad idea, because they wouldn't work with python.org python (more precisely, only things that depend on shared libs are affected, but the point remains).

I've been peripherally following this thread so I may be missing the point but it seems to me that Python already has a mature and flexible way of locating and loading shared libs through the module/import system. Surely the best way to manage non-Python shared libs is by exposing them as extension modules which can be packaged up on PyPI. Then you have dependency resolution for pip, you don't need to worry about the OS-specific shared library loading details and ABI information can be stored as metadata in the module. It would even be possible to load multiple versions or ABIs of the same library as differently named Python modules IIUC. As a case in point numpy packages up a load of C code and wraps a BLAS/Lapack library. Many other extension modules are written which can all take advantage of the non-Python shared libraries that embody numpy via its C API. Is there some reason that this is not considered a good solution? -- Oscar

On 19 May 2015 at 16:22, Chris Barker <chris.barker@noaa.gov> wrote:

The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together...

Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option. If someone were to create and publish a Python compatible static ".lib" file for the various hard-to-build dependencies, extensions could specify that you link with it in setup.py, and all the person building the wheel has to do is download the needed libraries for the build. If there's a technical reason why dynamic linking at runtime is better than static linking (sufficiently better that it justifies all the effort needed to resolve the issues involved), then I've yet to see a good explanation of it. The only things I've seen are disk space, or maintenance (where this usually means "it's easier to release a new DLL with a security fix than get all the various statically linked packages updated - that's a valid point, but given how hard it is to get a working dynamic linking solution in this environment, I have to wonder whether that argument still holds). All of this applies to Windows only, of course - dynamic linking and system management of shared libraries is very much a per-platform issue, and I don't pretend to know the trade-offs on OSX or Linux. Paul

On Tue, May 19, 2015 at 9:15 AM, David Cournapeau <cournape@gmail.com> wrote:

Honestly, I still haven't seen a solid explanation of why (at least on

...

Windows) static linking isn't a viable option.

well - it does get us pretty far....

Because some libraries simply don't work as static libraries, or are too big (MKL comes to mind). Also, we have been historically using static libs for our eggs at Enthought on windows, and it has been a nightmare to support. It just does not scale when you have 100s of packages.

there is also the issue of semi-developers -- I want people to be able to easily build my package, that depends on a bunch of libs that I really want to be the same. I suppose I could deliver the static libs themselves, along with the headers, etc, but that does get ugly.

But really, once wheels support arbitrary file locations, this becomes fairly easy at the packaging level: the remaining issue is one of ABI / standards, but that's mostly a non technical issue.

yup -- social issues are the big one.

Gholke has 100s of packages using wheels,

doesn't he ship the dlls with the packages, even when not using static libs? so that multiple packages may have the same dll? which is almost the same as a static lib.

and we ourselves at Enthought have close to 500 packages for windows, all packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++ programs, etc... It is more about agreeing about a common way of doing things than a real technical limitation.

good to know -- I suspected as much but haven't tried it yet. If someone were to

...

create and publish a Python compatible static ".lib" file for the various hard-to-build dependencies, extensions could specify that you link with it in setup.py, and all the person building the wheel has to do is download the needed libraries for the build.

OK, but from a "social" perspective, this is unlikely to happen (it hasn't yet!), without some "official" support on PyPi, with pip, or ??? So even if static is the way to go -- there is an infrastructure need. If there's a technical reason why dynamic linking at runtime is better

...

than static linking (sufficiently better that it justifies all the effort needed to resolve the issues involved),

I'm not sure the issues are that much greater -- we could build binary wheels that hold dynamic libs, and put them up on PyPi -- then other package maintainers would need to actually use those -- almost the same thing as getting a variety of package maintainers to use this mythical repository of static libs. (and by the way, gcc makes it remarkably had to force a static build) Another issue I've run into is nested static libs -- you have to change yoru setup.py in special ways for that to work: libnetcdf depends on libhdf5, depends on libcurl and libz and... getting all that statically linked into my extension is tricky. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, May 19, 2015 at 11:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

to be the same. I suppose I could deliver the static libs themselves, along with the headers, etc, but that does get ugly.

Hmm, that seems to me to be something of a non-goal. If you publish wheels, 99.999% of people should never need to build your software.

I'm not so sure -- with a nice mature package, sure. But with something under active development, having a low barrier to entry to running the latest code is really helpful. I suppose I could use a CI system to push a new version of the binary wheel with every update - maybe that is the way to go. I'm 100% in favour of repeatable, automated builds - absolutely

But we don't really need a standard infrastructure for them, you write a "setup_build_environment.py" script that ships with your project, run it once, and then run python setup.py bdist_wheel. If your build dependencies are simple, "setup_build_environment.py" could be short (or even non-existent!) or if not it could be many hundreds of lines of code. But it's specific to your project.

This entire conversation is about when the build dependencies are NOT simple :-). And while it may be project specific, commonly used libs are not project specific, and they are where the time and pain are. So some shared infrastructure would be nice. And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way for folks to collaborate on this kind of ugly package building effort. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, May 19, 2015 at 1:27 PM, Paul Moore <p.f.moore@gmail.com> wrote:

...

And maybe all that needs to be is a gitHub project with build scripts. But I had little luck in getting any traction that way. That is, until we had Anaconda, conda and binstar --- an infrastructure that provides a way for folks to collaborate on this kind of ugly package building effort.

Yeah, it's all about getting people interested. I wish the github project model would work (it did for msys2) but just wishing doesn't help much.

well, to be fair, critical mass is key -- and I did not get far in generating that critical mass myself. If a couple dedicated people were to start such a project, and get it far enough that is was really easy for new folks to jump in and grab what they need -- it might keep rolling.

Let me ask a different question, then. If I wanted to get hold of (say) libraries for libyaml, libxml2, and maybe a few others (libxpm?) is the conda work of any use for me? I don't want to build conda packages, or depend on them, I just want to grab Python-compatible libraries that I can link into my extension wheel build, or into my application that embeds Python. Ideally, I'd want static libs, but I understand that conda doesn't go that route (at the moment, at least). Even if it's just for me to better understand how conda works, is there an easy to follow guide I could read to see how the conda build of libyaml works? (I assume there must *be* a conda build of libyaml, as conda includes pyyaml which uses libyaml...)

Here's the trick -- as far as I know, a conda binary package itself does not include the recipe used to build it. So the question is: has someone published the conda recipe ( conda uses a build dir with a bunch of stuff in it, yaml, shell scripts, what have you ) for that package? if so, then yes, it's easy to go in a look at it and see how it's done. As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda. But they do host: https://github.com/conda/conda-recipes though it's not in the least bit complete or particularly maintained. (I don't see libxml in there). There is also a growning community of folks developing and maintaining conda recipes for all sorts of stuff -- much of it on gitHub: https://github.com/ioos/conda-recipes is one example. (those are all getting built and pushed to binstar too). I suspect the answer is "no, conda's not designed to support that use

case". Which is fine - but a shame.

it would be nice if a conda package got a copy of its build recipe embedded in it.

The "github project with build scripts" approach could potentially allow *more* users of the builds and libraries, And that, in a nutshell is the problem (on Windows, at least) - the community of people developing builds of common Unix tools and libraries is completely fragmented - msys2, conda, mingw,

yes, it sure is. Anyway, I feel like we're now going round in circles. It's a hard

(social) issue, and I don't feel like I have any answers, really.

me neither -- come on -- someone give us the answer! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

It is certainly not our intention at Continuum to keep build recipes private. I have just come on board at the company, but I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes. On Tue, May 19, 2015 at 3:09 PM, Paul Moore <p.f.moore@gmail.com> wrote:

On 19 May 2015 at 22:29, Chris Barker <chris.barker@noaa.gov> wrote:

...

As far as I can tell, Continuum does not publish the build scripts used to build all the stuff in Anaconda.

So, for example the process for building the pyyaml package available via conda is private? (I want to say "proprietary", but there's a lot of implications in that term that I don't intend...) That seems like a rather striking downside to conda that I wasn't aware of. Hopefully, I'm misunderstanding something :-)

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.

On Tue, May 19, 2015 at 3:09 PM, Paul Moore <p.f.moore@gmail.com> wrote:

So, for example the process for building the pyyaml package available via conda is private?

well, I haven't been able to find them... I don't know if continuum keeps them "private" on purpose or, just haven't happened to publish them.

That seems like a rather striking downside to conda that I wasn't aware of.

We need to be careful here about what is: "conda" -- a fully open source package management system "Anaconda" -- a python and other stuff distribution produced by Continuum. How Continuum does or doesn't publish the recipes it used to build Anaconda doesn't really have anything to do with conda-the-technology. On Tue, May 19, 2015 at 3:23 PM, David Mertz <dmertz@continuum.io> wrote:

It is certainly not our intention at Continuum to keep build recipes private.

I'll add it to my TODO list to work on making sure that those are better updated and maintained at https://github.com/conda/conda-recipes.

That would be great! I will note that most recipes seem to consist of either 'python setup.py

install' or './configure; make; make install'.

sure -- but those aren't the ones we want ;-)

So there is quite likely actually little significant work that has failed to have been published. But I'm not sure of pyyaml off the top of my head, and how that is built.

see if you can find the wxPython one, while you are at it :-) -- though I suspect that was built from the "official" executable, rather than re-built from scratch. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 20 May 2015 at 15:53, David Mertz <dmertz@continuum.io> wrote:

It's not *only* the 'setup.py install', but it's not *that* much mystery either. wxPython I can't seem to find, not sure what I'm missing.

Yeah, I had been under the impression that there was a lot of knowledge on how to build the dependencies (things like libyaml or whatever) in there. Looks like that's not the case... Paul

On Wed, May 20, 2015 at 6:30 AM, Daniel Holth <dholth@gmail.com> wrote:

The package includes its build recipe in info/recipe

very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, May 20, 2015 at 12:13 PM, Chris Barker <chris.barker@noaa.gov> wrote:

The package includes its build recipe in info/recipe

...

...

very cool -- I hadn't seen that -- I'll go take a look at some packages and see what I can find.

Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to.

The other day, I upgraded the version of conda-recipes/arrow to v0.5.4, and added ofxparse. I should probably create some sort of recurring cron task to show how far behind stable the version number in the meta.yaml is. (see: conda skeleton --version-compare issue/PR (GH:conda/conda-build))

And since a conda package is "just" a tar archive, you can presumably build them in other ways than a conda build recipe.

By the way -- libxml is one example of one without a recipe...

Hours of compilation time. * https://www.google.com/#q=inurl:libxml2+conda+meta.yaml (3 results) * https://pypi.python.org/pypi?%3Aaction=search&term=buildout+libxml * https://pypi.python.org/pypi/z3c.recipe.staticlxml/0.10

-Chris

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On May 20, 2015 7:43 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

On 21 May 2015 at 05:05, Wes Turner <wes.turner@gmail.com> wrote:

...

On Wed, May 20, 2015 at 12:13 PM, Chris Barker <chris.barker@noaa.gov> wrote:

...

...

...

The package includes its build recipe in info/recipe

very cool -- I hadn't seen that -- I'll go take a look at some

packages

...

...

...

and see what I can find.

Darn -- the recipe is not there in most (all?) of the packages that came from Anaconda -- probably due to the legacy issues David referred to.

The other day, I upgraded the version of conda-recipes/arrow to v0.5.4, and added ofxparse.

I should probably create some sort of recurring cron task to show how far behind stable the version number in the meta.yaml is. (see: conda skeleton --version-compare issue/PR (GH:conda/conda-build))

https://release-monitoring.org/ is a public service for doing that (more info on supported upstream backends at https://release-monitoring.org/about, more info on the federated messaging protocol used to publish alerts at http://www.fedmsg.com/en/latest/)

Anitya (the project powering release-monitoring.org) was built as the "monitoring" part of Fedora's upstream release notification pipeline: https://fedoraproject.org/wiki/Upstream_release_monitoring

Thanks!

One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like "fedora.repackage", "debian.repackage" or "conda.repackage" which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes.

http://conda.pydata.org/docs/bdist_conda.html bdist_conda?

Regards, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, May 20, 2015 at 5:43 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

One of my hopes for the metadata extension system in PEP 426 is that we'll be able to define extensions like "fedora.repackage", "debian.repackage" or "conda.repackage" which include whatever additional info is needed to automate creation of a policy compliant downstream package in a format that's a purely additive complement to the upstream metadata, rather than being somewhat duplicative as is the case today with things like spec files, deb control files, and conda recipes.

That would be great, yes! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 19 May 2015 at 17:15, David Cournapeau <cournape@gmail.com> wrote:

On Wed, May 20, 2015 at 12:46 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

On 19 May 2015 at 16:22, Chris Barker <chris.barker@noaa.gov> wrote:

...

The other issue is social: this would really only be a benefit if a wide variety of packages shared the same libs -- but each of those packages is maintained by different individuals and communities. So it's had to know if it would get used. I could put up a libpng wheel, for instance, and who knows if the Pillow folks would have any interest in using it? or the matplotlib folks, or, ... And this would be particularly difficult when the solution was hacked together...

Honestly, I still haven't seen a solid explanation of why (at least on Windows) static linking isn't a viable option.

Because some libraries simply don't work as static libraries, or are too big (MKL comes to mind). Also, we have been historically using static libs for our eggs at Enthought on windows, and it has been a nightmare to support. It just does not scale when you have 100s of packages.

But really, once wheels support arbitrary file locations, this becomes fairly easy at the packaging level: the remaining issue is one of ABI / standards, but that's mostly a non technical issue.

Gholke has 100s of packages using wheels, and we ourselves at Enthought have close to 500 packages for windows, all packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++ programs, etc... It is more about agreeing about a common way of doing things than a real technical limitation.

Thanks. I'm not going to try to argue against the voice of experience. If you were saying dynamic libs were impossible with wheels (which has been a definite impression I've got elsewhere in this thread) I might try to argue, but if you're saying that the technical issues are solvable, then that's great. We need to establish the details of the "common way of doing things" that needs to be agreed to move forward. Maybe an interoperability PEP would be useful here, with someone simply proposing a solution to give the debate a concrete starting point? Paul

Maybe I wasn't very clear -- I was addressing what conda might provide in the context of using conda packages with pip/pipy. A conda environment provides a great deal more, yes. system-site-packages may very well be for a different version of the python interpreter and stdlib. Isn't that handled by the wheel meta-data? At least in practice -- Anaconda is delivering python.org compatible pythons. meta.yaml (w/ optional preprocessing # [selectors]) http://conda.pydata.org/docs/build.html calls build.sh or build.bat by default, at build time. Exactly -- you need to write those build scripts yourself. Conda does set up the environment for you (lib paths,etc) so it's often as easy as "./configure && make && make install", but if it's not (particularly on windows!) you've got to figure it out. Which brings in the community aspect -- a lot of stuff has been figured out. PPAs would be great Personal package archives? Can't we do that now with pip+wheel? Indeed, isn't that 1/2 of what binary wheels are for? And the only thing they are for on Linux? -Chris

On Wed, May 20, 2015 at 2:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 18 May 2015 at 14:32, David Cournapeau <cournape@gmail.com> wrote:

...

On Mon, May 18, 2015 at 9:05 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

Indirection via pip injects the usage of setuptools even for plain distutils projects, and generates https://www.python.org/dev/peps/pep-0376/ compliant metadata by default.

Note that some packages will push hard against injecting setuptools, at least until it does not offer a way to prevent from installing as an egg directory. Most of the core scientific packages avoid setuptools because of this.

pip changes the default behaviour of setuptools to be more in line with the behaviour of plain distutils (e.g. forcing "--single-version-externally-managed").

Yes, but that cannot be the only way to install the package. This is why I would like to see a way forward for https://bitbucket.org/pypa/setuptools/issue/371/setuptools-and-state-of-pep-..., to start pushing packages using setuptools in their setup.py in the scientific stack. Without this, it will be hard to move forward politically speaking. Lots of key contributors have a strong aversion to setuptools way of doing things (partly historical reasons, but it is hard to change minds). If every system (pip, "setup.py install", conda, enstaller) were to at least write the {dist/egg}-info directories correctly, it would be already a significant step toward interoperation. David This means that "pip install X" consistently gives setuptools levels

of capabilities in terms of dependency management, without defaulting to installing things as egg archives or directories (modulo the rough edges around setup-requires).

As Donald notes, "pip install" also abstracts away any future invocation of a metadata based pluggable build system, while "./setup.py install" assumes the distutils-based status quo will remain in place forever.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia