Mailman 3 Proposal: "Install and save" - Distutils-SIG

newer
Re: [Distutils] Proposal: "Install...

Proposal: "Install and save"

older
Switch to upload.pypi.org instead...

Chris Angelico

22 Jul 2016 22 Jul '16

11:18 p.m.

In teaching my students how to use third-party Python modules, I generally recommend starting every project with "python3 -m venv env", and then install dependencies into the virtual environment. They then need a requirements.txt to keep track of them. There are two workflows for that: $ pip install flask $ pip freeze >requirements.txt or: $ echo flask >>requirements.txt $ pip install -r requirements.txt Over in the JavaScript world, npm has a much tidier way to do things. I propose adding an equivalent to pip: $ pip install --save flask which will do the same as the second option above (or possibly write it out with a version number as per 'pip freeze' - bikeshed away). As well as cutting two commands down to one, it's heaps easier in the multiple installation case - "pip install --save flask sqlalchemy gunicorn" is much clearer than messing around with creating a multi-line file; and the 'pip freeze' option always ends up listing unnecessary dependencies. Thoughts? ChrisA

Show replies by date

Alex Grönholm

23 Jul 23 Jul

7:02 p.m.

I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified. 22.07.2016, 20:48, Chris Angelico kirjoitti:

...

In teaching my students how to use third-party Python modules, I generally recommend starting every project with "python3 -m venv env", and then install dependencies into the virtual environment. They then need a requirements.txt to keep track of them. There are two workflows for that:

$ pip install flask $ pip freeze >requirements.txt

or:

$ echo flask >>requirements.txt $ pip install -r requirements.txt

Over in the JavaScript world, npm has a much tidier way to do things. I propose adding an equivalent to pip:

$ pip install --save flask

which will do the same as the second option above (or possibly write it out with a version number as per 'pip freeze' - bikeshed away). As well as cutting two commands down to one, it's heaps easier in the multiple installation case - "pip install --save flask sqlalchemy gunicorn" is much clearer than messing around with creating a multi-line file; and the 'pip freeze' option always ends up listing unnecessary dependencies.

Thoughts?

ChrisA _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Chris Angelico

7:26 p.m.

On Sat, Jul 23, 2016 at 11:32 PM, Alex Grönholm wrote:

...

I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified.

For other packages, yes; what about for applications? How is a web application to best say "I need Flask, and markdown, and SQLAlchemy"? I'm willing to consider workflow changes (among other advantages, they don't have to wait until the new pip gets pushed out to everyone). ChrisA

Thomas Kluyver

7:34 p.m.

On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...

I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified.

Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/ We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means. As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? Thomas

Alex Grönholm

7:40 p.m.

...

On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...
I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...

Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Daniel Holth

8:13 p.m.

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file. https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm wrote:

...

...
On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...
I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Alex Grönholm

9:07 p.m.

pip doesn't yet support pyproject.toml does it? 23.07.2016, 17:43, Daniel Holth kirjoitti:

...

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm mailto:alex.gronholm@nextday.fi> wrote:

23.07.2016, 17:04, Thomas Kluyver kirjoitti: > On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote: >> I'm -1 on this because requirements.txt is not really the standard way >> to list dependencies. >> In the Python world, setup.py is the equivalent of Node's package.json. >> But as it is >> Python code, it cannot so easily be programmatically modified. > Packaging based on declarative metadata: > http://flit.readthedocs.io/en/latest/ > > > We have a bit of a divide. Specifying dependencies in setup.py (or > flit.ini, or upcoming pyproject.toml) is the standard for library and > tool packages that are intended to be published on PyPI and installed > with pip. requirements.txt is generally used for applications which will > be distributed or deployed by other means. > > As I understand it, in the Javascript world package.json is used in both > cases. Is that something Python should try to emulate? Is it hard to > achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata. > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Daniel Holth

11:05 p.m.

Not yet. Someone should fix that 😎 On Sat, Jul 23, 2016, 11:37 Alex Grönholm wrote:

...

pip doesn't yet support pyproject.toml does it?

23.07.2016, 17:43, Daniel Holth kirjoitti:

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm wrote:

...
...
On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...
I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's package.json. But as it is Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Philippe Ombredanne

24 Jul 24 Jul

12:03 p.m.

On Sat, Jul 23, 2016, 17:04, Thomas Kluyver wrote:

...

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out?

Thomas: this is not entirely correct: a node package with native code needs to have node-gyp [1] installed which in turns relies on the gyp-based Python tool ;) to actually build the natives using actual makefile-like build scripts. MO is that any kind of mildly engaged package build for NPM requires a rather more involved effort than its Python world equivalent. And you can see the JS packaging world walking somewhat of the same path that has walked here before. To get some feeling of what they are going through read this thread [2] On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm wrote:

...

This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

Indeed! There is no magic and no free lunch there. [1] https://github.com/nodejs/node-gyp [2] https://github.com/npm/npm/issues/1891 -- Cordially Philippe Ombredanne

Brett Cannon

9:51 p.m.

On Sat, 23 Jul 2016 at 10:36 Daniel Holth wrote:

...

Not yet. Someone should fix that 😎

There is an issue tracking that if someone gets adventurous enough to write up a PR: https://github.com/pypa/pip/issues/3691 . -Brett

...

On Sat, Jul 23, 2016, 11:37 Alex Grönholm wrote:

...
pip doesn't yet support pyproject.toml does it?

23.07.2016, 17:43, Daniel Holth kirjoitti:

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm wrote:

...
...
On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...
I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's

...
...
But as it is Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: package.json. metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Wes Turner

30 Jul 30 Jul

11:20 p.m.

pipup has "save to a requirements.txt" functionality https://github.com/revsys/pipup It looks like it doesn't yet handle hash-checking mode (which is from peep, IIRC): - https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode - https://github.com/revsys/pipup/blob/master/pipup/req_files.py I think str(req_install.InstallRequirement) could/should just work? Or maybe to_requirements_str()? - https://github.com/pypa/pip/blob/master/pip/req/req_file.py - https://github.com/pypa/pip/blob/master/pip/req/req_install.py pip-tools probably has InstallRequirement.to_requirements_str()? - https://github.com/nvie/pip-tools/blob/master/piptools/writer.py - https://github.com/nvie/pip-tools/blob/master/piptools/utils.py - format_requirement() - format_specifier() Round-trip with requirements.txt files would probably be useful On Sunday, July 24, 2016, Brett Cannon wrote:

...

On Sat, 23 Jul 2016 at 10:36 Daniel Holth javascript:_e(%7B%7D,'cvml','dholth@gmail.com');> wrote:

...
Not yet. Someone should fix that 😎

There is an issue tracking that if someone gets adventurous enough to write up a PR: https://github.com/pypa/pip/issues/3691 .

-Brett

...
On Sat, Jul 23, 2016, 11:37 Alex Grönholm javascript:_e(%7B%7D,'cvml','alex.gronholm@nextday.fi');> wrote:

...
pip doesn't yet support pyproject.toml does it?

23.07.2016, 17:43, Daniel Holth kirjoitti:

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm javascript:_e(%7B%7D,'cvml','alex.gronholm@nextday.fi');> wrote:

...
...
On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote:

...
I'm -1 on this because requirements.txt is not really the standard way to list dependencies. In the Python world, setup.py is the equivalent of Node's

...
...
But as it is Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: package.json. metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org'); https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org'); https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org'); https://mail.python.org/mailman/listinfo/distutils-sig

Wes Turner

31 Jul 31 Jul

6:53 a.m.

pbr also supports "environment markers" which we would want to preserve when round-tripping (reading in, modifying, and writing out) requirements.txt files; though IDK if environment markers are part of any Python Packaging Spec? from http://docs.openstack.org/developer/pbr/#environment-markers : argparse; python_version=='2.6' On Sat, Jul 30, 2016 at 12:50 PM, Wes Turner wrote:

...

pipup has "save to a requirements.txt" functionality https://github.com/revsys/pipup

It looks like it doesn't yet handle hash-checking mode (which is from peep, IIRC):

- https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode - https://github.com/revsys/pipup/blob/master/pipup/req_files.py

I think str(req_install.InstallRequirement) could/should just work? Or maybe to_requirements_str()? - https://github.com/pypa/pip/blob/master/pip/req/req_file.py - https://github.com/pypa/pip/blob/master/pip/req/req_install.py

pip-tools probably has InstallRequirement.to_requirements_str()?

- https://github.com/nvie/pip-tools/blob/master/piptools/writer.py - https://github.com/nvie/pip-tools/blob/master/piptools/utils.py - format_requirement() - format_specifier()

Round-trip with requirements.txt files would probably be useful

On Sunday, July 24, 2016, Brett Cannon wrote:

...
On Sat, 23 Jul 2016 at 10:36 Daniel Holth wrote:

...
Not yet. Someone should fix that 😎

There is an issue tracking that if someone gets adventurous enough to write up a PR: https://github.com/pypa/pip/issues/3691 .

-Brett

...
On Sat, Jul 23, 2016, 11:37 Alex Grönholm wrote:

...
pip doesn't yet support pyproject.toml does it?

23.07.2016, 17:43, Daniel Holth kirjoitti:

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm < alex.gronholm@nextday.fi> wrote:

...
...
On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote: > I'm -1 on this because requirements.txt is not really the standard way > to list dependencies. > In the Python world, setup.py is the equivalent of Node's

...
> But as it is > Python code, it cannot so easily be programmatically modified. Packaging based on declarative metadata: http://flit.readthedocs.io/en/latest/

We have a bit of a divide. Specifying dependencies in setup.py (or flit.ini, or upcoming pyproject.toml) is the standard for library and tool packages that are intended to be published on PyPI and installed with pip. requirements.txt is generally used for applications which will be distributed or deployed by other means.

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative

23.07.2016, 17:04, Thomas Kluyver kirjoitti: package.json. metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata.

...
Thomas _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Wes Turner

6:54 a.m.

(side note: I would love to work on this but am in dire need of a job (which would ideally encourage further open source contribution)) On Sat, Jul 30, 2016 at 8:23 PM, Wes Turner wrote:

...

pbr also supports "environment markers" which we would want to preserve when round-tripping (reading in, modifying, and writing out) requirements.txt files; though IDK if environment markers are part of any Python Packaging Spec?

from http://docs.openstack.org/developer/pbr/#environment-markers :

argparse; python_version=='2.6'

On Sat, Jul 30, 2016 at 12:50 PM, Wes Turner wrote:

...
pipup has "save to a requirements.txt" functionality https://github.com/revsys/pipup

It looks like it doesn't yet handle hash-checking mode (which is from peep, IIRC):

- https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode - https://github.com/revsys/pipup/blob/master/pipup/req_files.py

I think str(req_install.InstallRequirement) could/should just work? Or maybe to_requirements_str()? - https://github.com/pypa/pip/blob/master/pip/req/req_file.py - https://github.com/pypa/pip/blob/master/pip/req/req_install.py

pip-tools probably has InstallRequirement.to_requirements_str()?

- https://github.com/nvie/pip-tools/blob/master/piptools/writer.py - https://github.com/nvie/pip-tools/blob/master/piptools/utils.py - format_requirement() - format_specifier()

Round-trip with requirements.txt files would probably be useful

On Sunday, July 24, 2016, Brett Cannon wrote:

...
On Sat, 23 Jul 2016 at 10:36 Daniel Holth wrote:

...
Not yet. Someone should fix that 😎

There is an issue tracking that if someone gets adventurous enough to write up a PR: https://github.com/pypa/pip/issues/3691 .

-Brett

...
On Sat, Jul 23, 2016, 11:37 Alex Grönholm wrote:

...
pip doesn't yet support pyproject.toml does it?

23.07.2016, 17:43, Daniel Holth kirjoitti:

Here is my attempt. The SConstruct (like a Makefile) builds the extension. The .toml file gives the static metadata. No need to put the two in the same file.

https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

https://bitbucket.org/dholth/cryptacular/src/tip/pyproject.toml

On Sat, Jul 23, 2016 at 10:11 AM Alex Grönholm < alex.gronholm@nextday.fi> wrote:

...
23.07.2016, 17:04, Thomas Kluyver kirjoitti: > On Sat, Jul 23, 2016, at 02:32 PM, Alex Grönholm wrote: >> I'm -1 on this because requirements.txt is not really the standard way >> to list dependencies. >> In the Python world, setup.py is the equivalent of Node's package.json. >> But as it is >> Python code, it cannot so easily be programmatically modified. > Packaging based on declarative metadata: > http://flit.readthedocs.io/en/latest/ > > > We have a bit of a divide. Specifying dependencies in setup.py (or > flit.ini, or upcoming pyproject.toml) is the standard for library and > tool packages that are intended to be published on PyPI and installed > with pip. requirements.txt is generally used for applications which will > be distributed or deployed by other means. > > As I understand it, in the Javascript world package.json is used in both > cases. Is that something Python should try to emulate? Is it hard to > achieve given the limitations of setup.py that you pointed out? This topic has been beaten to death. There is no way to cram the complexities of C extension compilation setup into purely declarative metadata. Distutils2 tried and failed. Just look at the setup.py files of some popular projects and imagine all that logic expressed in declarative metadata. > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Jeremy Stanley

10:24 p.m.

On 2016-07-30 20:23:14 -0500 (-0500), Wes Turner wrote:

...

pbr also supports "environment markers" which we would want to preserve when round-tripping (reading in, modifying, and writing out) requirements.txt files; though IDK if environment markers are part of any Python Packaging Spec?

from http://docs.openstack.org/developer/pbr/#environment-markers :

argparse; python_version=='2.6'

I'm assuming you didn't follow the "conditional dependencies" link from the description in that first paragraph, as it would have taken you to the corresponding section of one: https://www.python.org/dev/peps/pep-0426/#environment-markers You should probably also see: https://www.python.org/dev/peps/pep-0496/ https://www.python.org/dev/peps/pep-0508/#environment-markers Quickly skimming relevant changelogs, environment markers have been supported by Setuptools since 0.7, and directly in requirements lists since pip 6.0. -- Jeremy Stanley

Nick Coghlan

24 Jul 24 Jul

8:23 a.m.

On 24 July 2016 at 00:04, Thomas Kluyver wrote:

...

As I understand it, in the Javascript world package.json is used in both cases. Is that something Python should try to emulate? Is it hard to achieve given the limitations of setup.py that you pointed out?

There are other problems with the "one file to rule them all" approach, one of which is that libraries should aim to be as loose as is practical in declaring their dependencies, while applications should treat dependency updates like any other code change: require them to go through pre-merge CI by pinning an exact version of the dependency in requirements.txt. Donald wrote a good explanation of the distinction a few years ago: https://caremad.io/2013/07/setup-vs-requirement/ The best current toolset I know for managing the distinction in the case of an application venv or a user's ad hoc working environment is actually pip-tools: https://github.com/nvie/pip-tools#readme That splits the "explicitly declared application dependencies" out into a requirements.in file, which pip-compile then turns into a conventional (but fully pinned) requirements.txt file. The pip-sync tool then says "make this environment exactly match this requirements file". Personally, I could definitely see a feature like "pip-newdep requirements.in '<dep1>' '<dep2>'" being relevant in pip-tools as a shorthand for something like: echo '<dep1>' >> requirements.in && echo '<dep2>' >> requirements.in && pip-compile requirements.in && pip install -r requirements.txt (A full pip-sync wouldn't be appropriate though, since that may uninstall dev requirements added via a separate requirements file) I'm not sure it makes sense for pip itself though - pip's a bit more neutral than that on precisely how people do their environment management (even "pip install -r requirements.txt" is just an emergent convention). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Chris Angelico

9:20 a.m.

On Sun, Jul 24, 2016 at 12:53 PM, Nick Coghlan wrote:

...

Personally, I could definitely see a feature like "pip-newdep requirements.in '<dep1>' '<dep2>'" being relevant in pip-tools as a shorthand for something like:

echo '<dep1>' >> requirements.in && echo '<dep2>' >> requirements.in && pip-compile requirements.in && pip install -r requirements.txt

Can the file name be made implicit? If so, it would come to almost what I was originally looking for: pip-newdep sqlalchemy flask and it'd DTRT as regards building the requirements. If I understand you correctly, requirements.in would store only the names of packages required (no version info), and then requirements.txt would look like the output of 'pip freeze' but only for the packages listed in requirements.in? Because that would be _perfect_. Both files go into source control, and the task "update to the latest versions of things and check if it all works" would be straight-forward. ChrisA

Chris Angelico

9:24 a.m.

On Sun, Jul 24, 2016 at 1:50 PM, Chris Angelico wrote:

...

then requirements.txt would look like the output of 'pip freeze' but only for the packages listed in requirements.in?

Oops, should have read the readme first. It's listing the dependencies of those packages, too. Which is not a problem; the main issue I have with 'pip freeze' is that it often picks up standard library modules like argparse and wsgiref, which then cause installation issues. ChrisA

Nick Coghlan

7:53 p.m.

On 24 July 2016 at 13:50, Chris Angelico wrote:

...

On Sun, Jul 24, 2016 at 12:53 PM, Nick Coghlan wrote:

...
Personally, I could definitely see a feature like "pip-newdep requirements.in '<dep1>' '<dep2>'" being relevant in pip-tools as a shorthand for something like:

echo '<dep1>' >> requirements.in && echo '<dep2>' >> requirements.in && pip-compile requirements.in && pip install -r requirements.txt

Can the file name be made implicit?

"requirements.in" is the default in pip-compile if you don't otherwise specify a source file, so yes, you could rely on that. However, I'm not really the right person to ask - that would be Vincent Driessen, as the author of pip-tools :)

...

and it'd DTRT as regards building the requirements. If I understand you correctly, requirements.in would store only the names of packages required (no version info),

You can still put version constraints in requirements.in, but they'd generally only be for known incompatibilities (i.e. minimum versions, plus sometimes maximum versions if there's a backwards compatibility break)

...

and then requirements.txt would look like the output of 'pip freeze' but only for the packages listed in requirements.in? Because that would be _perfect_. Both files go into source control, and the task "update to the latest versions of things and check if it all works" would be straight-forward.

Yep, and if I understand the logic correctly, "pip-compile" will add new dependencies, but leave existing ones alone if they still meet the requirements, while "pip-compile --upgrade" will try to upgrade everything to the latest version. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

2825

Age (days ago)

2834

Last active (days ago)

List overview

Download

17 comments

9 participants

participants (9)

Alex Grönholm
Brett Cannon
Chris Angelico
Daniel Holth
Jeremy Stanley
Nick Coghlan
Philippe Ombredanne
Thomas Kluyver
Wes Turner

Proposal: "Install and save"

tags

participants (9)