Mailman 3 December 2016 - Distutils-SIG

Maintaining a curated set of Python packages
by Freddy Rietdijk 15 Dec '16

15 Dec '16

Hi, I would like to propose that, as a community, we jointly maintain a curated set of Python packages that are known to work together. These packages would receive security updates for some time and every couple of months a new major release of the curated set comes available. The idea of this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? So why a PyPI LTS? PyPI makes available all versions of packages that were uploaded, and by default installers like pip will try to use the latest available versions of packages, unless told otherwise. With a requirements.txt file (or a future pipfile.lock) and setup.py we can pin as much as we like our requirements of respectively the environment and package requirements, thereby making a more reproducible environment possible and also fixing the API for developers. Pinning requirements is often a manual job, although one could use pip freeze or other tools. A common problem is when two packages in a certain environment require different versions of a package. Having a curated set of packages, developers could be encouraged to test against the latest stable and nightly of the curated package set, thereby increasing compatibility between different packages, something I think we all want. Having a compatible set of packages is not only interesting for developers, but also for downstream distributions. All distributions try to find a set of packages that are working together and release them. This is a lot of work, and I think it would be in everyone's benefit if we try to solve this issue together. A possible solution Downstream, that is developers and distributions, will need a set of packages that are known to work together. At minimum this would consist of, per package, the name of the package and its version, but for reproducibility I would propose adding the filename and hash as well. Because there isn't any reliable method to extract the requirements of a package, I propose also including `setup_requires`, install_requires`, and `tests_require` explicitly. That way, distributions can automatically build recipes for the packages (although non-Python dependencies would still have to be resolved by the distribution). The package set would be released as lts-YYYY-MM-REVISION, and developers can choose to track a specific revision, but would typically be asked to track only lts-YYYY-MM which would resolve to the latest REVISION. Because dependencies vary per Python language version, interpreter, and operating system, we would have to have these sets for each combination and therefore I propose having a source which evaluates to say a TOML/JSON file per version/interpreter/OS. How this source file should be written I don't know; while I think the Nix expression language is an excellent choice for this, it is not possible for everyone to use and therefore likely not an option. Open questions There are still plenty of open questions. - Who decides when a package is updated that would break dependents? This is an issue all distributions face, so maybe we should involve them. - How would this be integrated with pip / virtualenv / pipfile.lock / requirements.txt / setup.py? See e.g. https://github.com/pypa/pipfile/issues/10#issuecomment-262229620 References to Haskell LTS Here are several links to some interesting documents on how Haskell LTS works. - A blog post describing what Haskell LTS is: https://www.fpcomplete.com/blog/2014/12/backporting-bug-fixes - Rules regarding uploading and breaking packages: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package - The actual LTS files https://github.com/fpco/lts-haskell What do you think of this proposal? Would you be interested in this as developer, or packager? Freddy

13 23

Re: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages)
by Robert Collins 08 Dec '16

08 Dec '16

On 9 Dec 2016 5:45 PM, "Donald Stufft" <donald(a)stufft.io> wrote: On Dec 8, 2016, at 10:41 PM, Ben Finney <ben+python(a)benfinney.id.au> wrote: For those who haven't read it, see this post from Donald Stufft for why those purposes need to be kept distinct Somewhat funnily, pbr is what triggered me to actually write that post :) If I recall, at the time it only supported requirements.txt but it’s since then gotten support for putting it in setup.cfg and I think that is preferred now? It always had support for install requires in setup.cfg from its d2to1 heritage. That said early days of pbr were confused about a bunch of things... It's much happier and less layer breaking than it was... :)

1 0

JetBrains licences
by Vinay Sajip 07 Dec '16

07 Dec '16

I accidentally cc-ed this list when sending email to a Python committer about JetBrains licences. Please don't take up one of these licences unless you are a Python committer; I will remove access from any person who is not named in https://hg.python.org/committers.txt I apologise for any confusion I have caused. Regards, Vinay Sajip

1 0

PyCharm / JetBrains licence
by Vinay Sajip 07 Dec '16

07 Dec '16

Dear Christian, You can access the PyCharm and other JetBrains products licence at the following link: https://account.jetbrains.com/a/53fr5cnq If you don't already have a JetBrains account, you will need to create one to activate the licence. Regards, Vinay Sajip

1 0

Python anywhere problems
by Mashudu Mphohoni 02 Dec '16

02 Dec '16

[image: Inline image 1] Hi, Would you please advise me on my python anywhere, It seems to fail to install or upgrade any packages Thanks and regards, Mashudu

2 1