[Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 15, 2016, at 7:12 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> 
>> On Aug 15, 2016, at 1:56 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> My main thought regarding this is that bdist_dmg != all dmg files (similarly for msi and wininst). These are specific files created by distutils without a standard or without the needed work to make them truly what users should be using. I also think they are a different class of upload, the general use case for PyPI's current file uploads are for automated installs (as evidenced by the simple API and mirroring).
> 
> I guess I'm just a little confused - are we talking about just hiding them from some parts of the API or disallowing their upload entirely?
> 
> If we're talking about the literal output of bdist_dmg and bdist_rpm I probably agree that they're almost useless.

Right now we’re basically talking about the literal output of bdist_dmg and bdist_rpm, because their outputs are the only thing we actually support uploading. Our current checks aren’t very stringent (or in some cases, exist at all) so it’s *possible* someone is constructing an actual user friendly .dmg and uploading it pretending to be a “bdist_dmg”, that’s not really a supported operation.

Ideally I’d like to disallow their upload completely and hide them from the API, but if the current use case of bdist_dmg, bdist_rpm, bdist_wininst, etc is useful enough to keep around, then I’d like to just hide them from the API.

> 
>> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety for automated tooling, then we could do something like "related files" people can upload which don't get mirrored for pip and which don't show up in the repo API. Since they will be classified differently we could also do better work around the ux of discovering them and separate them from the 50 wheels that some projects end up uploading and make them more obviously visible. I don't know if pypi as a distribution for _end user_ (vs developer/power user) software makes sense or not, but if it does we should support it better than accidentally via distutils. 
> 
> My concern here is that if someone has a hacky workaround working with the current system, it might be better to add support for the new thing ("related files") before killing the old thing.  If the plan is to do them both anyway, wouldn't it be better to do it in that order?  As a community (and I mean the broader open source community here, not distutils-sig; if anything distutils is way better about this) we have an unfortunate habit of killing potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and then only adding the better thing after the fact.

I don’t know if it makes sense to add the hypothetical new thing. Is PyPI the right place to distribute a .dmg that say, ships Python itself, some C libraries, some Python libraries, some bash scripts, and some static files? I don’t know. Currently I think the answer is “no”, PyPI is a repository of software _for_ Python, not a repository of software that just happens to be written in Python. If the ultimate end goal of a .dmg is that people no longer need to worry about what language the thing they are installing is written in, it seems weird for them to go to PyPI for a .dmg for one app, npmjs.org for a .dmg for another app, and so on.

Now, if it *does* make sense to support generalized uploads for applications written in Python with some sort of system level packaging (dmg, deb, conda, whatever) then we should figure out a way to actually support that, and support it well. As it is, I don’t have any evidence that the files that are currently being uploaded to PyPI are anything but “wheels, but in dmg format” (e.g. binary packages containing a single library). I don’t want to put a bunch of effort and work into making this well supported if there isn’t some evidence that folks will actually use it (I suspect a minimum we’d need some buy in from the authors of tooling to make these self contained packages).


—
Donald Stufft