<div dir="ltr">The drawback of .zip is file size since it compresses each file individually rather than giving the compression algorithm a larger input, it's a great format otherwise. Ubiquitous including Apple iOS packages, Java, word processor file formats. And most Python packages are small.<div><br></div><div>We must do the hard work to support Unicode file names, and spaces and accent marks in home directory names (historically a problem on Windows), in our packaging system. It is the right thing to do. It is not the publisher's fault that your system has broken Unicode.</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 27, 2015 at 6:43 AM Paul Moore <<a href="mailto:p.f.moore@gmail.com">p.f.moore@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 26 October 2015 at 06:04, Nathaniel Smith <<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>> wrote:<br>
> Here's a second round of text towards making a build-system<br>
> independent interface between pip and source trees/sdists. My idea<br>
> this time is to take a divide-and-conquer approach: this text tries to<br>
> summarize all the stuff that it seemed like we had mostly reached<br>
> consensus on in the previous thread + call, with blank chunks marked<br>
> "TBD" where there are specific points that still need To Be<br>
> Determined. So my hope is that everyone will read what's here and<br>
> agree that it's great as far as it goes, and then we can go through<br>
> and fill in each missing piece one at a time.<br>
<br>
I'll comment on what's here, but ignore the TBD items - I'd rather (as<br>
you suggest) leave discussion of those details till the basic idea is<br>
agreed.<br>
<br>
> Abstract<br>
> ========<br>
><br>
> Distutils delenda est.<br>
<br>
While this makes a nice tagline, I'd rather something less negative.<br>
Distutils does not "need" to be destroyed. It's perfectly adequate<br>
(although hardly user friendly) for a lot of cases - I'd be willing to<br>
suggest *most* users can work just fine with distutils.<br>
<br>
I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric<br>
limited - as Nick pointed out this whole area is as much a political<br>
issue as a technical one.<br>
<br>
> Extended abstract<br>
> =================<br>
><br>
> While ``distutils`` / ``setuptools`` have taken us a long way, they<br>
> suffer from three serious problems: (a) they're missing important<br>
> features like autoconfiguration and usable build-time dependency<br>
> declaration, (b) extending them is quirky, complicated, and fragile,<br>
> (c) it's very difficult to use anything else, because they provide the<br>
> standard interface for installing python packages expected by both<br>
> users and installation tools like ``pip``.<br>
<br>
Again, this is overstated. You very nearly lost me right here - people<br>
won't read the details of the proposal if they disagree with the<br>
abstract(s). Specifically:<br>
<br>
* The features in (a) are only important to *some* parts of the<br>
community. The scientific community is the major one, and is a huge<br>
influence over the direction we want to go in, but again, not crucial<br>
to many people. And even where they might be useful (e.g., Windows<br>
users building pyyaml, lxml, pillow, ...) the description implies<br>
"working out what's there" rather than "allowing users to easily<br>
manage non-Python dependencies", which gives the wrong impression.<br>
<br>
* The features in (b) are highly specialised. Very few people extend<br>
setuptools/distutils. And those who do, have often invested a lot of<br>
effort in doing so. Sure, they'd rather not have needed to, but now<br>
that they have, a replacement system simply means that work is lost.<br>
Arguably, fixing (b) is only useful for people (like the scientific<br>
community) who have needed to extend setuptools and have been unable<br>
to achieve their goals that way. That's an even smaller part of the<br>
community.<br>
<br>
> Previous efforts (e.g. distutils2 or setuptools itself) have attempted<br>
> to solve problems (a) and/or (b). We propose to solve (c).<br>
<br>
Agreed - this is a good approach. But it's at odds with your abstract,<br>
which says distutils must die. Here you're saying you want to allow<br>
people to keep using distutils but allow people with specialised needs<br>
to choose an alternative. Or are you offering an alternative to people<br>
who use distutils?<br>
<br>
The whole of the above is confusing on the face of it. The details<br>
below clarify a lot, as does knowing how the previous discussions have<br>
gone. But it would help a lot if the introduction to this PEP were<br>
clearer.<br>
<br>
> The goal of this PEP is get distutils-sig out of the business of being<br>
> a gatekeeper for Python build systems. If you want to use distutils,<br>
> great; if you want to use something else, then that should be easy to<br>
> do using standardized methods. The difficulty of interfacing with<br>
> distutils means that there aren't many such systems right now, but to<br>
> give a sense of what we're thinking about see `flit<br>
> <<a href="https://github.com/takluyver/flit" rel="noreferrer" target="_blank">https://github.com/takluyver/flit</a>>`_ or `bento<br>
> <<a href="https://cournape.github.io/Bento/" rel="noreferrer" target="_blank">https://cournape.github.io/Bento/</a>>`_. Fortunately, wheels have now<br>
> solved many of the hard problems here -- e.g. it's no longer necessary<br>
> that a build system also know about every possible installation<br>
> configuration -- so pretty much all we really need from a build system<br>
> is that it have some way to spit out standard-compliant wheels.<br>
<br>
OK. Although I see a risk here that if I want to build package FOO, I<br>
now have to worry whether FOO's build system supports Windows, as well<br>
as worrying whether FOO itself supports Windows.<br>
<br>
There's still a role for some "gatekeeper" (not a good word IMO, maybe<br>
"coordinator") to provide a certain level of support or review of<br>
build systems, and a point of contact for users with build issues (the<br>
point of this proposal is to some extent that people don't need to<br>
*know* what build system a project uses, so suggesting everyone has to<br>
direct issues to the correct build system support forum isn't<br>
necessarily practical).<br>
<br>
> We therefore propose a new, relatively minimal interface for<br>
> installation tools like ``pip`` to interact with package source trees<br>
> and source distributions.<br>
><br>
> In addition, we propose a wheel-inspired static metadata format for<br>
> sdists, suitable for tools like PyPI and pip's resolver.<br>
><br>
><br>
> Terminology and goals<br>
> =====================<br>
><br>
> A *source tree* is something like a VCS checkout. We need a standard<br>
> interface for installing from this format, to support usages like<br>
> ``pip install some-directory/``.<br>
><br>
> A *source distribution* is a static snapshot representing a particular<br>
> release of some source code, like ``lxml-3.4.4.zip``. Source<br>
> distributions serve many purposes: they form an archival record of<br>
> releases, they provide a stupid-simple de facto standard for tools<br>
> that want to ingest and process large corpora of code, possibly<br>
> written in many languages (e.g. code search), they act as the input to<br>
> downstream packaging systems like Debian/Fedora/Conda/..., and so<br>
> forth. In the Python ecosystem they additionally have a particularly<br>
> important role to play, because packaging tools like ``pip`` are able<br>
> to use source distributions to fulfill binary dependencies, e.g. if<br>
> there is a distribution ``foo.whl`` which declares a dependency on<br>
> ``bar``, then we need to support the case where ``pip install bar`` or<br>
> ``pip install foo`` automatically locates the sdist for ``bar``,<br>
> downloads it, builds it, and installs the resulting package.<br>
<br>
This is somewhat misleading, given that you go on to specify the<br>
format below, but maybe that's only an issue for someone like me who<br>
saw the previous debate over "source distribution" (as a bundled up<br>
source tree) vs "sdist" as a specified format. If I understand, you've<br>
now discarded the former sense of source distribution, and are<br>
sticking with the latter (specified format) definition.<br>
<br>
> Source distributions are also known as "sdists" for short.<br>
><br>
><br>
> Source trees<br>
> ============<br>
><br>
> We retroactively declare the legacy source tree format involving<br>
> ``setup.py`` to be "version 0". We don't try to specify it further;<br>
> its de facto specification is encoded in the source code and<br>
> documentation of ``distutils``, ``setuptools``, ``pip``, and other<br>
> tools.<br>
><br>
> A "version 1" (or greater) source tree is any directory which contains<br>
> a file named ``pypackage.cfg``, which will -- in some manner whose<br>
> details are TBD -- describe the package's build dependencies and how<br>
> to invoke the build system. This mechanism:<br>
><br>
> - Will allow for both static and dynamic specification of build dependencies<br>
><br>
> - Will have some degree of isolation of different builds from each<br>
> other, so that it will be possible for a single run of pip to install<br>
> one package that build-depends on ``foo = 1.1`` and another package<br>
> that build-depends on ``foo = 1.2``.<br>
<br>
All good so far.<br>
<br>
> - Will leave the actual installation of the package in the hands of<br>
> the build/installation tool (i.e. individual package build systems<br>
> will not need to know about things like --user versus --global or make<br>
> decisions about when and how to modify .pth files)<br>
<br>
This seems completely backwards to me. It's pip's job to do the actual<br>
install. The build tool should *only* focus on generating standard<br>
conforming binary wheels - otherwise what's the point of the<br>
separation of concerns that wheels provide?<br>
<br>
Or maybe I'm confused by the term "build/installation tool" - by that<br>
did you actually mean pip, rather than the build system?<br>
<br>
(TBDs omitted)<br>
<br>
> Source distributions<br>
> ====================<br>
><br>
> [possibly this should get split off into a separate PEP, but I'll keep<br>
> it together for now for ease of discussion]<br>
><br>
> A "version 1" (or greater) source distribution is a file meeting the<br>
> following criteria:<br>
><br>
> - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where<br>
> {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant<br>
> version number, and {EXT} is a compliant archive format.<br>
><br>
> The set of compliant archive formats is: zip, [TBD]<br>
><br>
> [QUESTION: should we continue to allow .tar.gz and friends? In<br>
> practice by "allow" I mean something like "accept new-style sdists on<br>
> PyPI in this format". I'm inclined not to -- zip is the most<br>
> universally supported format around, it allows file-based random<br>
> access (unlike tar-based things) which is useful for pulling out<br>
> metadata without decompressing the whole thing, and standardizing on<br>
> one format dodges distracting and pointless discussions about which<br>
> format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to<br>
> continue to support other archive formats when passed explicitly on<br>
> the command line. Any objections?]<br>
<br>
+1 on having a single archive format, and zip seems like the best choice.<br>
<br>
> Similar to wheels, the archive is Unicode, and the filenames inside<br>
> the archive are encoded in UTF-8.<br>
<br>
This isn't the job of the sdist format to specify. It should be<br>
implicit in the choice of archive format.<br>
<br>
Having said that, I'd go with<br>
<br>
1. The sdist filename MUST support the full range of package names as<br>
specified in PEP 426 (<a href="https://www.python.org/dev/peps/pep-0426/#name" rel="noreferrer" target="_blank">https://www.python.org/dev/peps/pep-0426/#name</a>)<br>
and versions as in PEP 440<br>
(<a href="https://www.python.org/dev/peps/pep-0440/" rel="noreferrer" target="_blank">https://www.python.org/dev/peps/pep-0440/</a>). That's actually far less<br>
than full Unicode.<br>
2. The archive format MUST support arbitrary Unicode filenames. That<br>
means zip is OK, but tar.gz isn't unless you specify UTF-8 is used<br>
(the tar format doesn't allow for an encoding declaration - see<br>
<a href="https://docs.python.org/3.5/library/tarfile.html#tar-unicode" rel="noreferrer" target="_blank">https://docs.python.org/3.5/library/tarfile.html#tar-unicode</a> for<br>
details on Unicode issues in the tar format).<br>
<br>
Having said that I'd also go with "filenames in the archive SHOULD be<br>
limited to ASCII" - because we have had issues with pip where test<br>
files have Unicode filenames, and builds break because they get<br>
mangled on systems with weird encoding setups... IIRC, these are<br>
typically related to .tar.gz sdists, which (due to the lack of<br>
encoding support) result in files being unpacked with the wrong names.<br>
So maybe if we enforce zip format we don't need to add this<br>
limitation.<br>
<br>
> - When unpacked, it MUST contain a single directory directory tree<br>
> named ``{PACKAGE}-{VERSION}``.<br>
><br>
> - This directory tree MUST be a valid version 1 (or greater) source<br>
> tree as defined above.<br>
><br>
> - It MUST additionally contain a directory named<br>
> ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the<br>
> following contents:<br>
><br>
> - ``SDIST``: Mandatory. Same record-oriented format as a wheel's<br>
> ``WHEEL`` file, but with different fields::<br>
><br>
> SDist-Version: 1.0<br>
> Generator: setuptools sdist 20.1<br>
><br>
> ``SDist-Version`` is the version number of this specification.<br>
> Software that processes sdists should warn if ``SDist-Version`` is<br>
> greater than the version it supports, and must fail if<br>
> ``SDist-Version`` has a greater major version than the version it<br>
> supports.<br>
><br>
> ``Generator`` is the name and optionally the version of the<br>
> software that produced the archive.<br>
><br>
> - ``RECORD``: Mandatory. A list of all files contained in the sdist<br>
> (except for the RECORD file itself and any signature files) together<br>
> with their hashes, as specified in PEP 427.<br>
><br>
> - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as<br>
> specified in PEP 427.<br>
><br>
> - ``METADATA``: Mandatory. Metadata version 1.1 or greater format<br>
> metadata, with an additional rule that fields may contain the special<br>
> sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value<br>
> of this field cannot be determined until build time. If a "multiple<br>
> use field" is present with the value ``__SDIST_DYNAMIC__``, then this<br>
> field MUST occur exactly once, e.g.::<br>
><br>
> # Okay:<br>
> Requires-Dist: lxml (> 3.3)<br>
> Requires-Dist: requests<br>
><br>
> # no Requires-Dist lines at all is okay<br>
> # (meaning: this package's requirements are the empty set)<br>
><br>
> # Okay, requirements will be determined at build time:<br>
> Requires-Dist: __SDIST_DYNAMIC__<br>
><br>
> # NOT okay:<br>
> Requires-Dist: lxml (> 3.3)<br>
> Requires-Dist: __SDIST_DYNAMIC__<br>
><br>
> (The use of a special token allows us to distinguish between<br>
> multiple use fields whose value is statically the empty list versus<br>
> one whose value is dynamic; it also allows us to distinguish between<br>
> optional fields which are statically not present versus ones whose<br>
> value is dynamic.)<br>
><br>
> When this sdist is built, the resulting wheel MUST have metadata<br>
> which is identical to the metadata present in this file, except that<br>
> any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have<br>
> arbitrary values in the wheel.<br>
><br>
> A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for<br>
> the package name or version (i.e., these must be given statically),<br>
> and these MUST match the {PACKAGE} and {VERSION} of the sdist as<br>
> described above.<br>
<br>
This seems pretty good at first reading.<br>
<br>
> [TBD: do we want to forbid the use of dynamic metadata for any<br>
> other fields? I assume PyPI will enforce some stricter rules at least,<br>
> but I don't know if we want to make that part of the spec, or just<br>
> part of PyPI's administrative rules.]<br>
<br>
This covers the main point of contention. It would be bad if build<br>
systems started using __SDIST_DYNAMIC__ just because "it's easier".<br>
<br>
Maybe add<br>
<br>
* A valid sdist SHOULD NOT use the __SDIST_DYNAMIC__ mechanism any<br>
more than necessary (i.e., if the metadata is the same in all<br>
generated wheels, it does not need to use the __SDIST_DYNAMIC__<br>
mechanism, and so should not do so).<br>
<br>
> This is intentionally a close analogue of a wheel's ``.dist-info``<br>
> directory; intention is that as future metadata standards are defined,<br>
> the specifications for the ``.sdist-info`` and ``.dist-info``<br>
> directories will evolve in synchrony.<br>
><br>
><br>
> Evolutionary notes<br>
> ==================<br>
><br>
> A goal here is to make it as simple as possible to convert old-style<br>
> sdists to new-style sdists. (E.g., this is one motivation for<br>
> supporting dynamic build requirements.) The ideal would be that there<br>
> would be a single static pypackage.cfg that could be dropped into any<br>
> "version 0" VCS checkout to convert it to the new shiny. This is<br>
> probably not 100% possible, but we can get close, and it's important<br>
> to keep track of how close we are... hence this section.<br>
><br>
> A rough plan would be: Create a build system package<br>
> (``setuptools_pypackage`` or whatever) that knows how to speak<br>
> whatever hook language we come up with, and convert them into<br>
> setuptools calls. This will probably require some sort of hooking or<br>
> monkeypatching to setuptools to provide a way to extract the<br>
> ``setup_requires=`` argument when needed, and to provide a new version<br>
> of the sdist command that generates the new-style format. This all<br>
> seems doable and sufficient for a large proportion of packages (though<br>
> obviously we'll want to prototype such a system before we finalize<br>
> anything here). (Alternatively, these changes could be made to<br>
> setuptools itself rather than going into a separate package.)<br>
><br>
> But there remain two obstacles that mean we probably won't be able to<br>
> automatically upgrade packages to the new format:<br>
><br>
> 1) There currently exist packages which insist on particular packages<br>
> being available in their environment before setup.py is executed. This<br>
> means that if we decide to execute build scripts in an isolated<br>
> virtualenv-like environment, then projects will need to check whether<br>
> they do this, and if so then when upgrading to the new system they<br>
> will have to start explicitly declaring these dependencies (either via<br>
> ``setup_requires=`` or via static declaration in ``pypackage.cfg``).<br>
><br>
> 2) There currently exist packages which do not declare consistent<br>
> metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different<br>
> ``install_requires=``). When upgrading to the new system, projects<br>
> will have to evaluate whether this applies to them, and if so they<br>
> will need to either stop doing that, or else add ``__SDIST_DYNAMIC__``<br>
> annotations at appropriate places.<br>
><br>
> We'll also presumably need some API for packages to describe which<br>
> parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for<br>
> the packages that need it (a new argument to ``setup()`` or some<br>
> setting in ``setup.cfg`` or something).<br>
<br>
I'm confused here. And it's just now become clear *why* I'm confused.<br>
<br>
The sdist format MUST be a generated format - i.e., we should insist<br>
(in principle at least) that it's only ever generated by tools.<br>
Otherwise it's way too easy for people to just zip up their source<br>
tree, hand craft something generic (that over-uses __SDIST_DYNAMIC__)<br>
and say "here's an sdist". Obviously, people always *can* manually<br>
create an sdist but we need to pin down the spec tightly, or we've not<br>
improved things.<br>
<br>
That's why I'm concerned about __SDIST_DYNAMIC__ and it's also what<br>
confuses me about the above transition plan.<br>
<br>
For people using setuptools currently, the transition should be simply<br>
that they upgrade setuptools, and the "setup.py sdist" command in the<br>
new setuptools generates the new sdist format. By default, the<br>
setuptools sdist process assumes everything is static and requires the<br>
user to modify the setup.py to explicitly mark which metadata they<br>
want to be left to build time. That way, we get a relatively<br>
transparent transition, while avoiding overuse of dynamic metadata.<br>
<br>
If setup.py has to explicitly mark dynamic metadata, that also allows<br>
us to reject attempts to make name and version dynamic. Which is good.<br>
<br>
Paul<br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</blockquote></div>