On Mon, 10 Sep 2018 at 10:13, Melvyn Sopacua m.r.sopacua@gmail.com wrote:
First post, so short introduction:
Hi, and welcome.
I'm mostly working on Django projects, currently for 3yourminD in Berlin. Working with python / Django for 6 going on 7 years. Also experienced in PHP, some Js and build systems like docker. We're currently migrating everything to docker and pip / requirements.txt isn't cutting it anymore and I've tasked myself with looking for or building an alternative tool.
This weekend I did quite a bit of research into the packaging tools and came accross PEP 518. What it doesn't address is that there now is a file resvered that declares very little. It does not declare and define sections to serve as a deployment file for a project and apparently it is unclear to tool builders what the one section is does define is supposed to be used for.
It is true, PEP 518 doesn't define a lot by itself. But as it notes, "Tables not specified in this PEP are reserved for future use by other PEPs" - in other words, it's possible to add further details, but they should be backed by a formal specification, so that tools have a common understanding of what to do with the data.
Case in point: poetry doesn't add itself to the build-system table when writing the py-project.toml file. Which, as I read it, it should. Ideally, every tool.${name} should be in the build system?
Well, poetry isn't (as I understand it) a build system. It doesn't define *how* to build a wheel from project sources, rather it orchestrates the process of building an application from a set of projects (as I understand it from a brief read of the project page). That puts it in the same area as pipenv, I guess? That process (building *applications*) isn't something that's been really looked at yet in terms of standardisation, so yes, it'd still very much fair game for something to be discussed and agreed.
One reason this hasn't yet been looked at, as I understand things, is that there are a lot of very different scenarios to consider - web applications (which I guess are your area of interest), standalone command line utilities like Mercurial or the Leo editor, orchestration tools like Ansible, background OS service processes, etc. And it's unlikely that a "one size fits all" approach would work, so there's a whole question of how any standard would establish its scope and boundaries. One note of caution - the attempts I've seen in the past to define "build processes" tend to promote a certain amount of confusion by assuming all projects are similar, and then getting bogged down when people with very different requirements come along with different assumptions and mistaken expectations.
So, can we improve the file specification and introduce sections that allows a project (not a library) to specify how it should be built? Right now, this is all off-loaded to the tools that use the file and one cannot rely on anything. It also makes migration from one tool to another difficult and cooperation between tools hard. Or is this intentional: is this supposed to be tied to specific tools as build system are generally tightly coupled? And should we not change that?
You say "a project (not a library)". I presume by that you mean something like the application building process I described above? Yes, there's not been any standardisation effort in this area yet, so the situation is as you describe. It's not intentional as such, simply that no-one has really stepped up to propose anything yet. And maybe because there aren't that many formal tools in this area - a lot of what goes on appears to be simply workflow and local standards. But certainly it's something that might be worth exploring.
It would help if the PEP had some explanation of the choices made to reserve a file that partially sets out to replace (all?) other files, but does not do anything to accomplish that.
PEP 518 defines a means of "Specifying Minimum Build System Requirements for Python Projects". It defines a new, flexible, file format to hold that information, and defers further definition of the file contents to future PEPs, but it was never (to my knowledge) a goal of that PEP to look at "all other files" and fit them into the pyproject.toml format. It did reserve a namespace "tool.XXX" for projects that choose to consider themselves as "build tools" - and a number of projects (including poetry, I believe) have made use of that. (And as you can see, this opens up some confusion over what constitutes a "build tool", in contrast to my fairly strict definition above).
PEP 517 extended the pyproject.toml format to add data for projects to define which build system they want to be used when they are built.
There's nothing yet for "building a project (not a library)". You could start with getting agreement on some terminology explaining what "a project (not a library)" is, and should be called :-) Would it cover pip? What about Mercurial? Ansible? A Django site? A blogging application backed by Django (or Flask, or whatever)? Maybe we need multiple terms...
I realize that PEP-517 defines the API and leaves the implementation up to the tool, but that doesn't necessarily mean the py- project file should not name / reserve tables that the PEP 517 API should use.
It does - tools.flit and tools.setuptools, and similar, for other build tools.
Yes, there are places where further standardisation would be good (for example, config_options has no defined semantics, which means tools like pip can do little beyond pass unmodified key/value pairs to the backend), but I don't think that's what you're talking about here. I don't think PEP 517 is relevant to your proposal, though (unless you expect a tool like poetry to add a PEP 517 backend?)
I'm considering writing a tool that focuses solely on the deployment and dependency maintenance of a project and would like to not "vendor lock-in" my users, with even those dependency lists, because I have to use the tool namespace and can fill it every way I see fit and so will others.
So ideally, we would reserve dependency tables: [dependencies.$name]
Reserved names: standard, dev, test
Tools can add more names which they can then interpret themselves: [dependencies.stage] django-anonymizer = "^0.5"
And test tables and probably one or two more that come to mind. Is there any interest from the team to improve the file format and lock down such tables? I'd love to do the initial proposal if that helps move things forward.
For those tables to *not* lock people into your tool, you'd need other tools to agree to use the same names (I assume there *are* other tools doing the same as you propose, otherwise there's no potential for "lock in" being an issue at this stage ;-)), and for that you'd need a standard. That's why PEP 518 has such keys "reserved for future PEPs". It's not to block suggestions like yours, but rather to enable them (in a way that everyone gets to buy into, rather than the most popular tool dictating implementation defined standards).
Paul