On September 30, 2014 at 5:07:06 AM, M.-A. Lemburg (mal@egenix.com) wrote:
Thanks for the confirmation that my interpretation was wrong. This makes things a lot better :-)
More below...
On 29.09.2014 11:39, Nick Coghlan wrote:
On 29 Sep 2014 18:49, "M.-A. Lemburg" wrote:
You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files.
Such changes often do not affect the software itself, and so are not detected by software tests.
Fixing such packaging errors is the primary intended use case of the "post" field in PEP 440. Alternatively, many projects will just spin a new release that addresses those issues, just as they would for any other bug.
Both of those approaches have the advantage of letting users (especially those operating mirrors, or downloading tarballs and feeding them into a separate redistribution system) easily tell whether or not they have the fixed version.
I don't see how that would help. AFAIU, PyPI would regard a "3.1.4.post1" as new release and so invalidate all other already uploaded distribution files rather than just regard the fixed upload as update to the "3.1.4" release.
If we could get a widely adopted notion of build numbers into the tools that would help a lot.
Installers and PyPI would then regard "3.1.4-1" as belonging to release "3.1.4", but being a more current build as a distribution file carrying "3.1.4" in its file name.
This would also have to work for all files uploaded for a release, not only for some distribution formats, of course, including source release files, Windows installers, egg files, etc.
I'd have to run some experiments, but don't believe that setuptools and associated tools support this at the moment.
I don’t personally believe it makes sense for a source distribution to have a build number. Yes it would invalidate all other uploads because it should, They take the source distribution as an input, if you change the input then in all likelihood you change the output. This won’t be true in every case but to be able to determine when and where it won’t be true requires intimate knowledge of the package formats as well as the project in question. On the other hand Wheel files already support the concept of build numbers, I don’t see any technical reason why Windows installers can’t support them given that nothing automated pulls them down. Getting Egg files to support them may be a lost cause though since I’m not sure there’s a way to add that information to the Egg spec without mandating a new setuptools.
If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM.
No, deletion remains supported. The only capability being removed is silent substitution of hosted files with different ones bearing the same name.
So if, for example, release "3.1.4" had a packaging error, then deleting it would still be possible, but the replacement would need to be something like "3.1.4.post1" or "3.1.5", rather than being permitted to reuse the "3.1.4" name.
So just to summarize: the proposal is to turn PyPI into a WORM, but at least it's still possible to remove distribution files.
The external hosting support is then the mechanism by which authors can retain complete and total control over their release process. That approach avoids all licensing concerns (including those related to US export controls), as well as ensuring they have the ability to silently change the contents of previously released files if they choose to do so (although, as noted above, actually doing so may break installation for anyone installing with peep, which checks file hashes based on the first version downloaded).
You're regularly bringing up this argument.
Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment.
pip is unfortunately using the same kind of --no-one-will-want-to-use-this-option-because-its-too-long approach as setuptools/easy_install has done in the past to force people into installing packages as eggs rather than installing the packages in the standard write to site-packages dir way.
The end result is the same: users will not want to go through those extra hoops and thus packages not hosted on PyPI itself will be regarded as broken (because they don't install using the standard method; not because they are really broken).
This is what I'm trying to address in discussions like these all along. PyPI has a responsibility not only for the part consuming part of the Python community, but also for the creating part of it.
While PyPI is great for indexing packages, it's not necessarily the best answer for hosting the distribution files and I believe we should open up some more to allow for making it possible to offer the same kind of user experience while not making pypi.python.org the only source of distribution files.
In the Linux world, this already works great by having multiple repos which you can switch on/off easily.
This is already 100% possible in the same fashion as the Linux repos already do. You can add extra indexes to pip and have been anble to for a long time. This almost exactly matches the behavior of the Linux repositories where ``apt-get install thing-that-isnt-in-a-repo-i-have-added`` throws an error that it can't find something. However you can then add an external index either via command line flag, environment variable, or config file (currently only per user, pip 6.0 will have per machine and per virtual env files as well). In fact the only difference between not using PyPI as the package repository and using PyPI as the package repository is that pip and setuptools ship with the PypI URL out of the box. Then PEP 470, which Paul mentioned, does one better and enables a discovery mechanism so that if you register your external index with PyPI for your project, that someone doing ``pip install a-thing-which-requires-an-external-index`` it'll suggest to them to add the registered external index. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA