On 01/10, 2018, at 00:47, Dan Ryan <dan@danryan.co> wrote:
Uses Pipfile as a project marker instead of pyproject.toml.
See above. pyproject.toml wasn't standardized yet when pipenv was released (and still isn't, beyond that it is a file that could exist and store information). Pipfile was intended to replace requirements.txt per some previous thread on the topic, and pipenv was an experimental implementation of the separation between the two different ways that people currently use requirements.txt in the wild -- one as a kind of abstract, unpinned dependency list (Pipfile), and the other as a transitive closure (Pipfile.lock). Since neither is standardized _for applications_, I'm not totally sure this is an actual sticking point.
In either case, this seems super minor…
I feel this would need to be extensively discussed either way before the community can jump into a decision. The discussion I’ve seen has been quite split on whether we should use one file or the other, but nothing very explaining why outside of “one file is better than two”. To me, there are two main things one would want to specify dependencies for: a thing you want to import (as a Python library from another Python source file), and a thing you want to run (as a command, a webapp, etc.). These setups require inherently different ways to specify dependencies; sometime they match, but not always, and a tool would (eventually) need to provide a solution when they diverge (i.e. when a project needs to be both importable and runnable, and has different dependency sets for them). Of course, the solution to this may as well be to always use pyproject.toml, and to design it to fit both scenarios. In this case, however, the fields need to be designed carefully to make sure all areas are taken care of. NPM, for example, requires you to specify a (package) name for your project, even if it never needs it (e.g. a website backend), which to me is a sign of designing too much toward package distribution, not standalone runnable. Other tools using a one-file-of-truth configuration has a similar problem to a degree, as far as I can tell. Another example would be Rust’s Cargo, which cannot specify a binary-only dependency. A duel-file configuration (e.g. Pipfile and pyproject.toml) would essentially punt this design decision; it is probably not the most purest solution, but at least keeps things open enough they don’t collide. Maybe another “solution” would be to have multiple (two?) files for each use case, but have a tool for syncing them automatically if desired. This is how Bundler works if you use it to package a Ruby Gem, but I am not sure if that is by design or an artefact of circumstances (Gem specification predates Bundler like how Python packages appear before all-in-one project management tools start to appear). I don’t have much to provide at the current time regarding what the best design should look like, but want to voice my concerns before it is too late. TP
Not shipped with Python. (Obviously not pipenv's fault, but nonetheless.) (Not sure it should be)
Environments should be stored in project directory, not off in $HOME somewhere. (Not sure what this is about, but some of the folks present were quite insistent.)
They used to be by default stored in $PROJECT/.venv but user feedback led us to use $WORKON_HOME by default. This is configurable by environment variable ($PIPENV_VENV_IN_PROJECT) or if you simply have a virtualenv in the .venv folder in your project directory.
Environments should be relocatable.
And that will be possible whenever we can use venv across platforms and python versions. Currently that isn't possible, and we are forced to use virtualenv for compatibility.
Hardcoded to only support "default" and "dev" environments, which is insufficient.
Why? I mean, if you are planning to replace setuptools / flit / other build systems with pipenv and use pipfile as your new specification for declaring extras, I guess, but that's not how it's designed currently. Beyond that, I think we need some actual information on this one -- adding more complexity to any tool (including this kind of complexity) is going to ask more of the user in terms of frontloaded knowledge. This constraint limits the space a bit and for applications, I've very rarely seen actual limitations of this setup (but am interested, we have obviously had this feedback before but are not eager to add knobs in this specific area).
No mechanism for sharing prespecified commands like "run tests" or "reformat".
There is, but the documentation on the topic is not very thorough: https://pipenv.readthedocs.io/en/latest/advanced/#custom-script-shortcuts See also: https://github.com/sarugaku/requirementslib/blob/master/Pipfile#L26
For an example for the specific cases you mentioned, the Pipfile entry in that project looks like this: [scripts] black = 'black src/requirementslib/ --exclude "/(\.git|\.hg|\.mypy_cache|\.tox|\.venv|_build|buck-out|build|dist)/"' tests = "pytest -v --ignore=src/requirementslib/_vendor/ tests"
Can't install Python. (There's... really no reason we *couldn't* distribute pre-built Python interpreters on PyPI? between the python.org installers and the manylinux image, we're already building redistributable run-anywhere binaries for the most popular platforms on every Python release; we just aren't zipping them up and putting them on PyPI.)
Erm, well actually you can install python currently via pyenv on linux, and Tzu-ping is the maintainer of a pyenv-clone on windows which we've just never really got around to integrating fully. I've spoken to some of the folks over at Anaconda and I know they are interested in this as well especially given that it's pretty straightforward. It hasn't been a primary focus lately, but the tooling does exist (I don't think I've ever used it personally though)
Anyway, this is all a good discussion to have and I really appreciate you kicking it off. I've been following the __pypackages__ conversation a bit since pycon and I honestly don't have much opinion about where we want to put stuff, but I'm not sure that the impact of the folder is going to be as great to the user as people might imagine -- the tooling is already being built, so maybe it's just a matter of agreeing on that as the place to put stuff, which schema to follow, and honestly working with some new users. I do this quite a bit but I haven't done any formal information gathering. Anecdotally I'll always tell you I'm right, but if we had some user data on specific pain points / usability issues I'd definitely be prepared to change my mind.
Dan Ryan gh: @techalchemy // e: dan@danryan.co
-----Original Message----- From: Nathaniel Smith [mailto:njs@pobox.com] Sent: Sunday, September 30, 2018 6:42 AM To: distutils-sig Subject: [Distutils] Notes from python core sprint on workflow tooling
Now that the basic wheels/pip/PyPI infrastructure is mostly functional, there's been a lot of interest in improving higher-level project workflow. We have a lot of powerful tools for this – virtualenv, pyenv, conda, tox, pipenv, poetry, ... – and more in development, like PEP 582 [1], which adds a support for project-local packages directories (`__pypackages__/`) directly to the interpreter.
But to me it feels like right now, Python workflow tools are like the blind men and the elephant [2]. Each group sees one part of the problem, and so we end up with one set of people building legs, another a trunk, a third some ears... and there's no overall plan for how they can fit together.
For example, PEP 582 is trying to solve the problem that virtualenv is really hard to use for beginners just starting out [3]. This is a serious problem! But I don't want a solution that *only* works for beginners starting out, so that once they get a little more sophisticated they have to throw it out and learn something new from scratch.
So I think now might be a time for a bit of top-down design. **I want a picture of the elephant.** If we had that, maybe we could see how all these different ideas could be put together into a coherent whole. So at the Python core sprint a few weeks ago, I dragged some interested parties [4] into a room with a whiteboard [5], and we made a start at it. And now I'm writing it up to share with you all.
This is very much a draft, intended as a seed for discussion, not a conclusion.
[1] https://www.python.org/dev/peps/pep-0582/ [2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant [3] https://www.python.org/dev/peps/pep-0582/#motivation [4] I won't try to list names, because I know I'll forget someone, and I don't know if everyone would agree with everything I wrote there. But thank you all! [5] https://photos.app.goo.gl/4HfY8P3ESPNi9oLMA, including special guest appearance by Kushal's elbow
# The idealized lifecycle of a Python project
## 1. Beginner
Everyone starts out as a rank beginner. This may be the first time they have programmed at all. At this stage, users want to:
- install *one* thing to get started (e.g. python itself) - write and run simple scripts (standalone .py files) - run a REPL - install and use PyPI packages like requests or numpy - install and use tools like jupyter - their IDE should also be able to find these packages/tools
Over time, they'll probably end up with multiple scripts, and maybe want to organize them into subdirectories. The above should all work from subdirectories.
## 2. Sharing with others
Now we have a neat little script. Or maybe we've made a pretty jupyter notebook that computes some crucial business analytics. We want to share it with our friends or coworkers. We still need the features above; and now we also care about:
- version control - some way for our friend to reconstruct, on their computer: - the same PyPI packages that we were using - the same tools that we were using - the ways we invoked those tools
This last point is important: as projects grow in complexity, and are used by a wider audience, they often end up with fairly complex tool specifications that have to be shared among a team. For example:
- to run tests: in an environment that has pytest, pytest-cov, and pytest-trio installed, and with our project working directory on PYTHONPATH, run `pytest -Werror --cov ...` - to format code: in an environment using python 3.6 or later, that has black installed, run `black -l 79 *.py my-util-directory/*.py`
This kind of tool specification also puts us in a good position to set up CI when we reach that point.
At this point our project can grow in a few different directions.
## 3a. Deployable webapp
This adds the requirement to "deploy". I think this is mostly covered by the set-up-an-environment-to-run-a-command functionality already described? I'm not super familiar with this, but it's pipenv's core target, and pipenv doesn't have much more than that, so I assume that's about right...
## 3b. Reusable library
For this we also need to:
- Build sdists and wheels - Which means: pyproject.toml, and some way to invoke it - Install our library into our environments - Including dependency locking (best practice is to not pin dependencies in wheel metadata, but to pin all dependencies in CI; so there needs to be some way to track those separately, but integrated enough that it's not a huge ceremony to add or change a dependency)
## 3c. Reusable standalone app
I think this is pretty much like the "Reusable library", except that it'd be nice to have better tools to build/distribute standalone applications. But if we had them, we could invoke them the same way as we invoke other build systems?
# How do existing tools/proposals fit into this picture?
pyenv, virtualenv, and conda all solve parts of the "create an environment" problem, but consider the other aspects out-of-scope.
tox solves the problem of keeping a shared record of how to run a bunch of different tools in the appropriate environments, but doesn't handle pinning or procuring appropriate python versions, and requires a separate bootstrapping step to install tox.
`__pypackages__` (if implemented) makes it very easy for beginners to use PyPI packages in their own scripts and from the REPL; in particular, it would be part of python, so it meets the "install *one* thing" criterion. But, it doesn't provide any way to run tools. (There's no way to put `__pypackages__/bin` on PATH.) It doesn't allow scripts to be organized into subdirectories. (For security reasons, we can't have the python interpreter going off walking the filesystem looking for `__pypackages__/`, so the PEP specifies that `__pypackages__/` has to be in the same directory as the script that uses it.) There's no way to share your `__pypackages__` environment with a friend. So... it seems like a something that people would outgrow very quickly.
pipenv and poetry are interesting. Their basic strategy is to say, there is a top-level command that acts as your entry point to performing workflow actions on on a python project (`pipenv` or `poetry`, respectively). And this strategy at least in principle can solve the problems that `__pypackages__/` runs into. In particular, it doesn't rely on `$PATH`, so it can run tools; and because it's a dedicated project management tool, it can go looking for the project marker file.
# A fantastic elephant
So if our idealized user had an idealized tool, what would that look like?
They'll be interacting with Python through a dedicated tool, similar to pipenv or poetry. In my little fantasy here I'll call it `pyp`, because (a) I want to be neutral, (b) 6 characters is too long.
To get this tool, either they install Python (via python.org download, apt, homebrew, whatever), and the tool is automatically included. Or else, they install the tool directly, and it has the ability to install Python interpreters when needed.
Once they have the tool, they start by making a new directory for their project (this way they're ready to switch to version control later).
Then they somehow mark this directory as being a "python project root". I guess the UI would be something like `pyp new <name>` and it just does it for you, but we have to figure out what this creates on disk. We need some sort of marker file. Files that currently serve this kind of role include tox.ini, Pipfile, pyproject.toml, __pypackages__, ... But only one of these is a standard thing we're already committed to sticking with, so, pyproject.toml it is. Let's make it the marker for any python project, not just redistributable libraries. (And if we do grow up into a redistributable library, then we're already prepared.)
In the initial default configuration, there's a single default environment. You can install things with `pyp install ...` or `pyp uninstall ...`, and it tracks the requested packages in some standardized way in pyproject.toml, and also pins specific versions somewhere (could be pyproject.toml again I guess, or poetry's pyproject.lock would work too). This way when we decide to share our project later, our friends can recreate our environment on their system.
However, there's also the capability to configure multiple custom execution environments, including python version and installed packages. And the capability to configure new aliases like `pyp test` or `pyp reformat`, which run some specified command in a specified environment.
Since the install/locking metadata is all standardized, you can even switch between competing tools, and integrate with third-party tools like pyup.io.
For redistributable libraries, we also need some way to get the wheel metadata and the workflow metadata to play nicely together. Maybe this means that we need a standardized install-requires field in pyproject.toml, so that build backends and workflow tools have a shared source of truth?
# What's wrong with pipenv?
Since pipenv is the tool that those of us in the room were most familiar with, that comes closest to matching this vision, we brainstormed a list of complaints about it. Some of these are more reasonable than others.
- Not ambitious enough. This is a fuzzy sort of thing, but perception matters, and it's right there in the name: it's a tool to use pip, to manage an environment. If we're reconceiving this as the grand unified entryway to all of Python, then the name starts to feel pretty weird. The whole thing where it's only intended to work for webapp-style projects would have to change.
- Uses Pipfile as a project marker instead of pyproject.toml.
- Not shipped with Python. (Obviously not pipenv's fault, but nonetheless.)
- Environments should be stored in project directory, not off in $HOME somewhere. (Not sure what this is about, but some of the folks present were quite insistent.)
- Environments should be relocatable.
- Hardcoded to only support "default" and "dev" environments, which is insufficient.
- No mechanism for sharing prespecified commands like "run tests" or "reformat".
- Can't install Python. (There's... really no reason we *couldn't* distribute pre-built Python interpreters on PyPI? between the python.org installers and the manylinux image, we're already building redistributable run-anywhere binaries for the most popular platforms on every Python release; we just aren't zipping them up and putting them on PyPI.)
-n
-- Nathaniel J. Smith -- https://vorpus.org -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils- sig@python.org/message/YFJITQB37MZOPOFJJF3OAQOY4TOAFXYM/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/O...