[Linux-SIG] Re: Draft PEP: Graceful cooperation between external and Python package managers

Hi all, I work on Conda (and friends) at Anaconda Inc and have been following this PEP closely, thanks all for working on this!

On 25. Aug 2021, at 16:57, Steve Dower wrote:

On 8/24/2021 4:35 PM, Geoffrey Thomas wrote:

...

Hmmm, I'm not sure exactly how to phrase that PyPI is not a distro but Conda is. Both use Python-aware package management tools (pip / conda). And while Conda is well-known for having non-Python software (R, curl, etc.), so does PyPI (patchelf, ninja, etc.). And we expect that Conda will _not_ provide an EXTERNALLY-MANAGED file, though Homebrew will. So maybe it's okay to say that technically both PyPI and Conda are distros, but since they use Python-specific package managers as their primary package manager, it's okay for them to choose not to implement EXTERNALLY-MANAGED?

Conda is not a distro, but Anaconda is, as-is conda-forge (both of which are installed using Conda).

The difference from PyPI is that all the packages have been curated and produced (packed?) by a single organisation in a consistent build environment.

You’re right, Anaconda is a cross-platform distro, Conda is the package manager shipped with it, but pip “works” as well with it since Python is part of the Anaconda distro. I should note that there are other “channels” in the Conda ecosystem other than the “defaults” channel maintained by Anaconda, Inc, that you referred to. E.g.: - conda-forge, a self-organized community that maintains Conda recipes and builds Conda packages via public CI systems and publish to their own channel, https://conda-forge.org) - Bioconda (https://bioconda.github.io, bioinformatics specific packages) - anaconda.org, a platform that supports uploading private Conda packages to per-user/org channels etc That is definitely a difference to the PyPI ecosystem where there is no ability to separate packages in repos/namespaces like with Conda channels. BTW, here’s a good overview about Conda channels: https://carpentries-incubator.github.io/introduction-to-conda-for-data-scien...

Conda itself is not Python-specific, it is just popular for Python packages. It operates on simple archives that are extracted into a root directory and already contain all the file system structure needed to avoid collisions, so it really does benefit distros (as they can handle the entire layout more easily than many unrelated teams).

Yeah, FWIW Conda has support for pip, which automatically calls the pip CLI behind the scene to install Python packages into Conda environments. E.g.: https://carpentries-incubator.github.io/introduction-to-conda-for-data-scien... There is ALSO an off-by-default config option to automatically take pip dependencies into account in the solver: https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/pip-... There is a good chance that this feature gets enabled by default at some point in the future since we know from our user surveys that better conda/pip interoperability is a major request.

Personally, I would expect these distros to specify that packages they install are "not managed by pip". It's all just static files that they embed into their packages anyway, so eventually conda-build (or the specific build definitions, more likely) will generate the marker at build time.

Yes, this will be ideal and previously this was not as obvious to figure out — where a Python package was installed from originally. This PEP is very welcome.

Hopefully that's some helpful clarification. I haven't looked through the final PEP yet, but hoping to take a look before the end of the week.

Thanks Steve, this was super helpful, and apologies for dropping so many links in this post, but here’s a final one that someone from the data community wrote a while ago to summarize some misconceptions around Conda: https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ Obviously I’m hoping that we’ll be able to improve the interoperability between Conda and pip more in the future to help the many users that are accessing Python through the Anaconda distro. If there are any questions with regard to Conda/pip, please let me know. Thanks, Jannis/jezdez