One minor clarification—Poetry os both an application and a library packaging tool.
It however relies on the old, non-isolated Setuptools approach to do the building part
under the hood, so (from my own understanding)

    Poetry ≒ Pipenv + Flit - PEP 517 - PEP 518

Whether this is a good approach is another topic (well, I guess my position is implied as
a Pipenv maintainer), and Paul’s explanation is still complete valid.


On 10/9, 2018, at 18:47, Paul Moore <p.f.moore@gmail.com> wrote:

On Mon, 10 Sep 2018 at 10:13, Melvyn Sopacua <m.r.sopacua@gmail.com> wrote:

First post, so short introduction:

Hi, and welcome.

I'm mostly working on Django projects, currently for 3yourminD in Berlin.
Working with python / Django for 6 going on 7 years. Also experienced in PHP,
some Js and build systems like docker.
We're currently migrating everything to docker and pip / requirements.txt
isn't cutting it anymore and I've tasked myself with looking for or building
an alternative tool.

This weekend I did quite a bit of research into the packaging tools and came
accross PEP 518. What it doesn't address is that there now is a file resvered
that declares very little. It does not declare and define sections to serve as
a deployment file for a project and apparently it is unclear to tool builders
what the one section is does define is supposed to be used for.

It is true, PEP 518 doesn't define a lot by itself. But as it notes,
"Tables not specified in this PEP are reserved for future use by other
PEPs" - in other words, it's possible to add further details, but they
should be backed by a formal specification, so that tools have a
common understanding of what to do with the data.

Case in point: poetry doesn't add itself to the build-system table when
writing the py-project.toml file. Which, as I read it, it should. Ideally,
every tool.${name} should be in the build system?

Well, poetry isn't (as I understand it) a build system. It doesn't
define *how* to build a wheel from project sources, rather it
orchestrates the process of building an application from a set of
projects (as I understand it from a brief read of the project page).
That puts it in the same area as pipenv, I guess? That process
(building *applications*) isn't something that's been really looked at
yet in terms of standardisation, so yes, it'd still very much fair
game for something to be discussed and agreed.

One reason this hasn't yet been looked at, as I understand things, is
that there are a lot of very different scenarios to consider - web
applications (which I guess are your area of interest), standalone
command line utilities like Mercurial or the Leo editor, orchestration
tools like Ansible, background OS service processes, etc. And it's
unlikely that a "one size fits all" approach would work, so there's a
whole question of how any standard would establish its scope and
boundaries. One note of caution - the attempts I've seen in the past
to define "build processes" tend to promote a certain amount of
confusion by assuming all projects are similar, and then getting
bogged down when people with very different requirements come along
with different assumptions and mistaken expectations.

So, can we improve the file specification and introduce sections that allows a
project (not a library) to specify how it should be built? Right now, this is
all off-loaded to the tools that use the file and one cannot rely on anything.
It also makes migration from one tool to another difficult and cooperation
between tools hard. Or is this intentional: is this supposed to be tied to
specific tools as build system are generally tightly coupled? And should we
not change that?

You say "a project (not a library)". I presume by that you mean
something like the application building process I described above?
Yes, there's not been any standardisation effort in this area yet, so
the situation is as you describe. It's not intentional as such, simply
that no-one has really stepped up to propose anything yet. And maybe
because there aren't that many formal tools in this area - a lot of
what goes on appears to be simply workflow and local standards. But
certainly it's something that might be worth exploring.

It would help if the PEP had some explanation of the choices made to reserve a
file that partially sets out to replace (all?) other files, but does not do
anything to accomplish that.

PEP 518 defines a means of "Specifying Minimum Build System
Requirements for Python Projects". It defines a new, flexible, file
format to hold that information, and defers further definition of the
file contents to future PEPs, but it was never (to my knowledge) a
goal of that PEP to look at "all other files" and fit them into the
pyproject.toml format. It did reserve a namespace "tool.XXX" for
projects that choose to consider themselves as "build tools" - and a
number of projects (including poetry, I believe) have made use of
that. (And as you can see, this opens up some confusion over what
constitutes a "build tool", in contrast to my fairly strict definition
above).

PEP 517 extended the pyproject.toml format to add data for projects to
define which build system they want to be used when they are built.

There's nothing yet for "building a project (not a library)". You
could start with getting agreement on some terminology explaining what
"a project (not a library)" is, and should be called :-) Would it
cover pip? What about Mercurial? Ansible? A Django site? A blogging
application backed by Django (or Flask, or whatever)? Maybe we need
multiple terms...

I realize that PEP-517 defines the API and leaves
the implementation up to the tool, but that doesn't necessarily mean the py-
project file should not name / reserve tables that the PEP 517 API should use.

It does - tools.flit and tools.setuptools, and similar, for other build tools.

Yes, there are places where further standardisation would be good (for
example, config_options has no defined semantics, which means tools
like pip can do little beyond pass unmodified key/value pairs to the
backend), but I don't think that's what you're talking about here. I
don't think PEP 517 is relevant to your proposal, though (unless you
expect a tool like poetry to add a PEP 517 backend?)

I'm considering writing a tool that focuses solely on the deployment and
dependency maintenance of a project and would like to not "vendor lock-in" my
users, with even those dependency lists, because I have to use the tool
namespace and can fill it every way I see fit and so will others.

So ideally, we would reserve dependency tables:
[dependencies.$name]

Reserved names: standard, dev, test

Tools can add more names which they can then interpret themselves:
[dependencies.stage]
django-anonymizer = "^0.5"

And test tables and probably one or two more that come to mind. Is there any
interest from the team to improve the file format and lock down such tables?
I'd love to do the initial proposal if that helps move things forward.

For those tables to *not* lock people into your tool, you'd need other
tools to agree to use the same names (I assume there *are* other tools
doing the same as you propose, otherwise there's no potential for
"lock in" being an issue at this stage ;-)), and for that you'd need a
standard. That's why PEP 518 has such keys "reserved for future PEPs".
It's not to block suggestions like yours, but rather to enable them
(in a way that everyone gets to buy into, rather than the most popular
tool dictating implementation defined standards).

Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-leave@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/EOFZVUYQAWFEUI653UGKVTBJWVF3BXHP/