I would like to propose that, as a community, we jointly maintain a curated
set of Python packages that are known to work together. These packages
would receive security updates for some time and every couple of months a
new major release of the curated set comes available. The idea of this is
inspired by Haskell LTS, so maybe we should call this PyPI LTS?
So why a PyPI LTS?
PyPI makes available all versions of packages that were uploaded, and by
default installers like pip will try to use the latest available versions
of packages, unless told otherwise. With a requirements.txt file (or a
future pipfile.lock) and setup.py we can pin as much as we like our
requirements of respectively the environment and package requirements,
thereby making a more reproducible environment possible and also fixing the
API for developers. Pinning requirements is often a manual job, although
one could use pip freeze or other tools.
A common problem is when two packages in a certain environment require
different versions of a package. Having a curated set of packages,
developers could be encouraged to test against the latest stable and
nightly of the curated package set, thereby increasing compatibility
between different packages, something I think we all want.
Having a compatible set of packages is not only interesting for developers,
but also for downstream distributions. All distributions try to find a set
of packages that are working together and release them. This is a lot of
work, and I think it would be in everyone's benefit if we try to solve this
A possible solution
Downstream, that is developers and distributions, will need a set of
packages that are known to work together. At minimum this would consist of,
per package, the name of the package and its version, but for
reproducibility I would propose adding the filename and hash as well.
Because there isn't any reliable method to extract the requirements of a
package, I propose also including `setup_requires`, install_requires`, and
`tests_require` explicitly. That way, distributions can automatically build
recipes for the packages (although non-Python dependencies would still have
to be resolved by the distribution).
The package set would be released as lts-YYYY-MM-REVISION, and developers
can choose to track a specific revision, but would typically be asked to
track only lts-YYYY-MM which would resolve to the latest REVISION.
Because dependencies vary per Python language version, interpreter, and
operating system, we would have to have these sets for each combination and
therefore I propose having a source which evaluates to say a TOML/JSON file
How this source file should be written I don't know; while I think the Nix
expression language is an excellent choice for this, it is not possible for
everyone to use and therefore likely not an option.
There are still plenty of open questions.
- Who decides when a package is updated that would break dependents? This
is an issue all distributions face, so maybe we should involve them.
- How would this be integrated with pip / virtualenv / pipfile.lock /
requirements.txt / setup.py? See e.g.
References to Haskell LTS
Here are several links to some interesting documents on how Haskell LTS
- A blog post describing what Haskell LTS is:
- Rules regarding uploading and breaking packages:
- The actual LTS files https://github.com/fpco/lts-haskell
What do you think of this proposal? Would you be interested in this as
developer, or packager?
On 9 Dec 2016 5:45 PM, "Donald Stufft" <donald(a)stufft.io> wrote:
On Dec 8, 2016, at 10:41 PM, Ben Finney <ben+python(a)benfinney.id.au> wrote:
For those who haven't read it, see this post from Donald Stufft for why
those purposes need to be kept distinct
Somewhat funnily, pbr is what triggered me to actually write that post :)
If I recall, at the time it only supported requirements.txt but it’s since
then gotten support for putting it in setup.cfg and I think that is
It always had support for install requires in setup.cfg from its d2to1
That said early days of pbr were confused about a bunch of things... It's
much happier and less layer breaking than it was... :)
I accidentally cc-ed this list when sending email to a Python committer about JetBrains licences. Please don't take up one of these licences unless you are a Python committer; I will remove access from any person who is not named in https://hg.python.org/committers.txt
I apologise for any confusion I have caused.
You can access the PyCharm and other JetBrains products licence at the following link:
If you don't already have a JetBrains account, you will need to create one to activate the licence.