How to specify dependencies in Python
I came across a python library which has docs, which start like this: {{{ Quickstart Include foolib in your requirements.txt file. }}} AFAIK dependencies should be specified via `install_requires` in `setup.py`. Should I talk to the maintainer of the library and create a pull-request for the docs? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
On 12 January 2017 at 22:04, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I came across a python library which has docs, which start like this:
{{{
Quickstart
Include foolib in your requirements.txt file.
}}}
AFAIK dependencies should be specified via `install_requires` in `setup.py`.
Applications and services don't necessarily have a setup.py file - setup.py is more for pip-installable libraries and frameworks (see https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on that topic). Since the section is titled "Quickstart", it seems reasonable to gloss over the fact that there are options other than requirements.txt, especially as folks that have already gone to the effort of learning how to make their software pip-installable are also likely to have worked out how that impacts where you specify your dependencies. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
On 12 January 2017 at 22:04, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I came across a python library which has docs, which start like this:
{{{
Quickstart
Include foolib in your requirements.txt file.
}}}
AFAIK dependencies should be specified via `install_requires` in `setup.py`.
Applications and services don't necessarily have a setup.py file - setup.py is more for pip-installable libraries and frameworks (see https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on that topic).
What is an application? The django project uses this definition: https://docs.djangoproject.com/en/1.10/ref/applications/ I guess you mean something else, if you talk about "application". What is an application for you? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
On 13 January 2017 at 12:23, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
On 12 January 2017 at 22:04, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I came across a python library which has docs, which start like this:
{{{
Quickstart
Include foolib in your requirements.txt file.
}}}
AFAIK dependencies should be specified via `install_requires` in `setup.py`.
Applications and services don't necessarily have a setup.py file - setup.py is more for pip-installable libraries and frameworks (see https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on that topic).
What is an application?
The django project uses this definition: https://docs.djangoproject.com/en/1.10/ref/applications/
I guess you mean something else, if you talk about "application".
What is an application for you?
IMO, an "application" in this context is a standalone program (implemented in Python). This is as opposed to a "library" which is written to be imported in other Python code. Obviously, there's some overlap - some "applications" provide a "programming API" that can be imported, and some libraries provide command line entry points. A library (something that you import) should declare in its metadata that you need to have its dependencies installed before it will work. That lets you (the user of the library) be ignorant of the dependencies of the library (which are implementation details as far as you are concerned) - pip just sorts it out for you from the metadata. To get the dependency metadata, the *library* should include its dependencies in install_requires in its setup.py. For applications on the other hand, you need to install exactly the environment you tested, so you *don't* use dependency metadata. Rather, you create a requirements.txt file that contains the exact versions of everything (direct dependencies) your application needs, then deploy your application script to a Python environment where you've run "pip install -r requirements.txt" to set it up correctly. As I say, there are lots of grey areas here. But from your description, the library you found sounds like it should have dependency metadata, and not need you to use a requirements file. On the other hand, the "Quickstart" may be trying to tell people how to use foolib in their *application*, in which case it's correct. (It's oversimplified, as if you're writing a library that depends on foolib you'd use install_requires, but simply changing the text to just say "put foolib in your install_requires" is no better, as then you'd have ignored the application use case...) If you can point to the actual library you're referring to, it would be easier to be specific :-) Paul
It's probably a good idea here to explicitly distinguish between the kind of application that's a web app you deploy to sort of managed server environment, and the kind of application that's a command line or GUI tool that people download and run. The same word means radically different things to different people :-). I believe Paul's advice below is specifically addressing the former case, not the latter. On Jan 13, 2017 5:07 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 13 January 2017 at 12:23, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
On 12 January 2017 at 22:04, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I came across a python library which has docs, which start like this:
{{{
Quickstart
Include foolib in your requirements.txt file.
}}}
AFAIK dependencies should be specified via `install_requires` in `setup.py`.
Applications and services don't necessarily have a setup.py file - setup.py is more for pip-installable libraries and frameworks (see https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on that topic).
What is an application?
The django project uses this definition: https://docs.djangoproject.com/en/1.10/ref/applications/
I guess you mean something else, if you talk about "application".
What is an application for you?
IMO, an "application" in this context is a standalone program (implemented in Python). This is as opposed to a "library" which is written to be imported in other Python code. Obviously, there's some overlap - some "applications" provide a "programming API" that can be imported, and some libraries provide command line entry points.
A library (something that you import) should declare in its metadata that you need to have its dependencies installed before it will work. That lets you (the user of the library) be ignorant of the dependencies of the library (which are implementation details as far as you are concerned) - pip just sorts it out for you from the metadata. To get the dependency metadata, the *library* should include its dependencies in install_requires in its setup.py.
For applications on the other hand, you need to install exactly the environment you tested, so you *don't* use dependency metadata. Rather, you create a requirements.txt file that contains the exact versions of everything (direct dependencies) your application needs, then deploy your application script to a Python environment where you've run "pip install -r requirements.txt" to set it up correctly.
As I say, there are lots of grey areas here. But from your description, the library you found sounds like it should have dependency metadata, and not need you to use a requirements file. On the other hand, the "Quickstart" may be trying to tell people how to use foolib in their *application*, in which case it's correct. (It's oversimplified, as if you're writing a library that depends on foolib you'd use install_requires, but simply changing the text to just say "put foolib in your install_requires" is no better, as then you'd have ignored the application use case...)
If you can point to the actual library you're referring to, it would be easier to be specific :-)
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Am 13.01.2017 um 14:07 schrieb Paul Moore:
On 13 January 2017 at 12:23, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
On 12 January 2017 at 22:04, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I came across a python library which has docs, which start like this:
{{{
Quickstart
Include foolib in your requirements.txt file.
}}}
AFAIK dependencies should be specified via `install_requires` in `setup.py`.
Applications and services don't necessarily have a setup.py file - setup.py is more for pip-installable libraries and frameworks (see https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on that topic).
What is an application?
The django project uses this definition: https://docs.djangoproject.com/en/1.10/ref/applications/
I guess you mean something else, if you talk about "application".
What is an application for you?
IMO, an "application" in this context is a standalone program (implemented in Python). This is as opposed to a "library" which is written to be imported in other Python code. Obviously, there's some overlap - some "applications" provide a "programming API" that can be imported, and some libraries provide command line entry points.
Hi Paul, yes, I think the same way. There is some overlap.
A library (something that you import) should declare in its metadata that you need to have its dependencies installed before it will work. That lets you (the user of the library) be ignorant of the dependencies of the library (which are implementation details as far as you are concerned) - pip just sorts it out for you from the metadata. To get the dependency metadata, the *library* should include its dependencies in install_requires in its setup.py.
Thank you for your explanation. For me this means, I should tell the maintainer that a library should specify it's dependencies via install_requires.
For applications on the other hand, you need to install exactly the environment you tested, so you *don't* use dependency metadata. Rather, you create a requirements.txt file that contains the exact versions of everything (direct dependencies) your application needs, then deploy your application script to a Python environment where you've run "pip install -r requirements.txt" to set it up correctly.
I think requirements.txt should be the result of some kind of Continous-Integration run. If all tests are successful, then requirements.txt should be created with "pip freeze". This means, that the Continous-Integration run does not use requirements.txt to build its environment. Next question: Where is the best place to store requirements.txt? I think it should not be in the repo of a library. It should be somehow outside. Up to now I a missing the right english term for it.
As I say, there are lots of grey areas here. But from your description, the library you found sounds like it should have dependency metadata, and not need you to use a requirements file. On the other hand, the "Quickstart" may be trying to tell people how to use foolib in their *application*, in which case it's correct. (It's oversimplified, as if you're writing a library that depends on foolib you'd use install_requires, but simply changing the text to just say "put foolib in your install_requires" is no better, as then you'd have ignored the application use case...)
If you can point to the actual library you're referring to, it would be easier to be specific :-)
I did not tell the name here, since I don't want to offend the library maintainer. I think he does not like it, when people talk behind his back about his code. I will do so after we found a solid solution here. Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
On 16 January 2017 at 15:59, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
Thank you for your explanation. For me this means, I should tell the maintainer that a library should specify it's dependencies via install_requires.
Well yes, but that doesn't mean the quoted comment from Quickstart is wrong. It's not clear without context whether the comment is intended to refer to applications using this library, or other libraries using this library. It's even possible that the project author doesn't fully understand the subtleties here.
I think requirements.txt should be the result of some kind of Continous-Integration run. If all tests are successful, then requirements.txt should be created with "pip freeze".
This means, that the Continous-Integration run does not use requirements.txt to build its environment.
Next question: Where is the best place to store requirements.txt?
I think it should not be in the repo of a library. It should be somehow outside.
Up to now I a missing the right english term for it.
I'm not at all sure how true that is, but as I don't write this sort of application, and certainly don't publish such things, I can't really comment.
If you can point to the actual library you're referring to, it would be easier to be specific :-)
I did not tell the name here, since I don't want to offend the library maintainer. I think he does not like it, when people talk behind his back about his code.
I will do so after we found a solid solution here.
OK, but I don't think we can find anything more concrete without more information, and at some point you're going to end up having to give enough information that we might as well just look at the project. After all, it's (presumably) open source, so we can always look at the code and submit PRs/issues anyway. It seems unlikely that the author is going to be offended - if there's a risk, why not simply invite him/her to participate in this thread? Paul
On Jan 16, 2017, at 7:59 AM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I think requirements.txt should be the result of some kind of Continous-Integration run. If all tests are successful, then requirements.txt should be created with "pip freeze".
This means, that the Continous-Integration run does not use requirements.txt to build its environment.
Next question: Where is the best place to store requirements.txt?
I think it should not be in the repo of a library. It should be somehow outside.
I think I understand what you're trying to say here, but I think you have it backwards. The best example I have made of the "right" way to do this is in this project: https://github.com/rackerlabs/mimic/ <https://github.com/rackerlabs/mimic/> The project has both a setup.py and (several versions of) requirements.txt. setup.py gives the abstract requirements for the project. This is what should work, but has not necessarily been exactly tested to. requirements.txt is the concrete requirements. This is generated (more or less) by freezing an environment where one does `pip install .` to trigger the setup.py. However, all continuous integration tests are run against the versions listed in requirements.txt (the details are here: https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f2550... <https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f2550...> and here https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f2550... <https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f2550...>) to ensure that new contributors always have a stable set of versions to test against, and their PR won't end up randomly having to fix some upgrade issue. Finally, we use https://requires.io <https://requires.io/> to submit PRs every time one of the dependencies upgrades. On a regular basis (every 6 months or so), these do cause errors; when they do, development unrelated to the version upgrade can continue on unimpeded, and the version-compatibility fix lives neatly in its own PR that solves just that problem. It's a bit subtle to understand the distinction between setup.py and requirements.txt (https://caremad.io/posts/2013/07/setup-vs-requirement/ <https://caremad.io/posts/2013/07/setup-vs-requirement/> is a good attempt, but I think stumbles over explaining some nuances that are obvious to dstufft and not to anyone else), but if you get them lined up correctly then a bunch of edge-cases work very nicely, and reasoning about deployment is much easier. -glyph
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
What is an application for you?
Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries). It's important for components to be flexible, so they're typically very flexible about versions of their dependencies. For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive). Jim -- Jim Fulton http://jimfulton.info
Am 13.01.2017 um 16:25 schrieb Jim Fulton:
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
What is an application for you?
Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries).
It's important for components to be flexible, so they're typically very flexible about versions of their dependencies.
For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive).
Yes, install_requires in setup.py should define flexible dependencies, but requirements.txt should define fixed dependencies via fixed version. Do you agree with my sentence from above? -- Thomas Guettler http://www.thomas-guettler.de/
On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
Am 13.01.2017 um 16:25 schrieb Jim Fulton:
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
What is an application for you?
Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries).
It's important for components to be flexible, so they're typically very flexible about versions of their dependencies.
For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive).
Yes, install_requires in setup.py should define flexible dependencies, but requirements.txt should define fixed dependencies via fixed version.
Do you agree with my sentence from above?
Are you speaking of a component/library or whole system? Jim -- Jim Fulton http://jimfulton.info
Am 16.01.2017 um 18:06 schrieb Jim Fulton:
On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Am 13.01.2017 um 16:25 schrieb Jim Fulton:
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler <guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de> <mailto:guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>>> wrote:
What is an application for you?
Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries).
It's important for components to be flexible, so they're typically very flexible about versions of their dependencies.
For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive).
Yes, install_requires in setup.py should define flexible dependencies, but requirements.txt should define fixed dependencies via fixed version.
Do you agree with my sentence from above?
Are you speaking of a component/library or whole system?
I am speaking of both. And: I think requirements.txt is optional. -- Thomas Guettler http://www.thomas-guettler.de/
On Tue, Jan 17, 2017 at 11:34 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
Am 16.01.2017 um 18:06 schrieb Jim Fulton:
On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler < guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote:
Am 13.01.2017 um 16:25 schrieb Jim Fulton:
On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler < guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de> <mailto: guettliml@thomas-guettler.de <mailto:guettliml@thomas-guettler.de>>> wrote:
What is an application for you?
Another way to think about this, FWIW, is to distinguish between the "whole system" (for which "Application" is often a useful shorthand), as opposed to components (aka libraries).
It's important for components to be flexible, so they're typically very flexible about versions of their dependencies.
For whole systems, OTOH, it's important that once a configuration is tested, that the same configuration is used in production, so systems typically pin all of their dependencies, ideally extending to their environments (which is a reason why container technology is attractive).
Yes, install_requires in setup.py should define flexible dependencies, but requirements.txt should define fixed dependencies via fixed version.
Do you agree with my sentence from above?
Are you speaking of a component/library or whole system?
I am speaking of both. And: I think requirements.txt is optional.
Then I disagree with your statement. :) I should stop, but I'll take one more stab at this. It matters whether you're talking about components(/libraries) or whole systems (/applications). For components: Consumers of a component need to be able to to determine the component's dependencies. The component uses install_required (and extras_require) for this. The version specifications in these dependencies should be as flexible as possible, to allow reuse in as many whole systems as possible. Developers of a component will use tools like pip and buildout to automate their development activities. For pip, that will usually entail one or more requirements.txt files. For buildout, that will entail one or more (for different development activities) buildout configs and a single versions.cfg. For whole systems: Many whole systems only assemble components. For these systems, there is no setup.py file (no python project). If a whole system includes a Python project (that isn't distributed separately), it's a matter of taste how much information is included in setup.py. Personally, I would treat the Python project like a component project and include its direct dependencies and minimal version constraints. Typically a whole system managed with pip will use a requirements.txt file (or possibly multiple) and a system developed with buildout will have a buildout config and a versions config. A whole system could fix all of its dependent versions in install_requires in a setup script, but that would be cumbersome. By using requirements.txt or a buildout versions config, a developer can avail themselves of automation to help maintain the files. Jim -- Jim Fulton http://jimfulton.info
On 17 January 2017 at 17:40, Jim Fulton <jim@jimfulton.info> wrote:
I am speaking of both. And: I think requirements.txt is optional.
Then I disagree with your statement. :)
I should stop, but I'll take one more stab at this.
It matters whether you're talking about components(/libraries) or whole systems (/applications).
FWIW, that was my point as well. Thomas, if you're not understanding that instructions need to be different depending on whether you're talking about components/libraries or systems/applications, then we're not going to reach an agreement. IMO, the "Quickstart" you quoted that triggered this is pretty much OK as long as it's talking about developing applications. It's not clear without context whether that's the case, but unless *you* understand the distinction Jim and I are trying to make, then I doubt we'd agree with any suggested rewording of that section that you might propose. Paul
participants (6)
-
Glyph Lefkowitz -
Jim Fulton -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Thomas Güttler