On Sun, 30 Sep 2018 at 11:48, Nathaniel Smith
Now that the basic wheels/pip/PyPI infrastructure is mostly functional, there's been a lot of interest in improving higher-level project workflow.
[...]
This is very much a draft, intended as a seed for discussion, not a conclusion. [...]
Interesting ideas - thanks for writing them up, I wish I could have been there! One workflow that I see a lot (and I *use* a lot myself!) is the "directory full of stuff" pattern. Describing it that way makes it sound disorganised, but it often really isn't. Essentially, it's where the user has a directory (maybe something as generic as "Work", maybe something like "Customer 1" or "Research on topic X") which contains a lot of scripts, some in Python, some not, plus support programs, data, working notes, etc. It's emphatically *not* suitable for checking into VCS, or bundling up, but it does potentially require things like dependency management. At the moment, when using such a workflow, it's more or less essential to either use nothing but the Python stdlib, remember details like "I need to activate virtualenv X to run this script", or install dependencies into the system Python (or maybe user site). Sharing scripts like this (unless they take the "stdlib only" approach) is really difficult. It's very easy to dismiss this sort of approach as something we don't want to (or can't) support, but it's very much the way beginners (and non-beginner part time users) approach *any* language. And when it fails for Python, they see that as a failing of Python, not of the approach. In my workplace, this sort of approach is standard for shell scripts, batch files, SQL scripts, etc. The fact that it doesn't work for non-trivial Python scripts makes it nearly impossible for me to promote Python as an automation solution (I usually end up suggesting Groovy instead, because Java is everywhere, and JVM code can be bundled up with its dependencies into a standalone jar). Essentially, this is your "Beginner" level (or more accurately "Sharing with others" but without infrastructure like VCS), but I think that term ignores just how far some organisations push that model - way beyond anything that an *actual* "beginner" would use. It's not impossible to argue that doing so isn't a model that we want to support, but in doing that we're pretty much abandoning the idea of Python for "adhoc scripting" and I don't think that's a good idea. As someone very much in that situation, I really need some sort of solution. But I've tried pipenv and similar, and uniformly their assumptions on how I structure my work, and what flexibility I have in my environment, are wrong making them useless for me. At the moment, I'm not sure what else to add to your summary, apart from "yes, you're right". Getting the various projects in this area talking and sharing ideas/resources would be great.
- Environments should be stored in project directory, not off in $HOME somewhere. (Not sure what this is about, but some of the folks present were quite insistent.)
For me, it's about being able to copy/relocate a project, and about housekeeping. If I rename my project directory, I'd rather not have to run a rebuild step (no matter how easy that is). Also, I'd rather not have to remember that I once had a directory called "secret_project" and hunt out and remove the clutter in $HOME that's linked to it, now that I've officially named the project "pip2" <wink>. It's not fundamental to "having stuff in $HOME" so much as how pipenv does (or doesn't) maintain a record of where all the bits that make up a "project" are kept. At least to me (and I'd have been insistent if I were present). The problem with high-level management tools for workflow (and especially opinionated ones) is that unless you're very careful to survey people's requirements and specify your scope, you're always going to end up with people who need to do certain things *not* being served by your tool. So it's almost impossible to be "the one official tool". Paul