new PyPI: a rant from a package maintainer
I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened. Disclaimers: - Any and all feedback is welcome, but please be charitable. - I'm angry *at* things, I hope not to offend anybody. - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet. For formatting reasons the text is here: https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 Thank you for your attention
Since I have less direct personal involvement, I'll say this: Yes, there are problems. A lot of them are around documentation, and timing of when things switch over (the whole Python packaging ecosystem has been in a state of flux/rewrite for a while now; this is unfortunate, but also necessary). Nobody *likes* this. Nobody *wants* this. Nobody *defends* this. I get that this is really frustrating to someone who's brand new to trying to use the tools and services. At times it's frustrating to me, and I've been building and distributing Python packages through PyPI for something like a decade now. But I think what you posted is more about lashing out in anger than anything else, and while it feels good to yell and curse and rant, it also doesn't really help anything to do so. The problems are known, and they're being worked on, and the real root problem is a chronic lack of people with time to keep plugging away at it and fixing problems and implementing the improved solutions. The only way it gets better is by supporting the people who are doing that, or becoming one of the people who are doing that. I'm sorry that you're frustrated. You're not the only person who's frustrated, and in fact I'd bet a lot of the people who are trying to get this stuff to work seamlessly and perfectly for you are at least as frustrated as you are. They don't need more people yelling at them. They know there's pain. They've had to face it every day for a long time.
On Fri, Aug 4, 2017 at 2:29 PM, James Bennett <ubernostrum@gmail.com> wrote:
I'm sorry that you're frustrated. You're not the only person who's frustrated, and in fact I'd bet a lot of the people who are trying to get this stuff to work seamlessly and perfectly for you are at least as frustrated as you are. They don't need more people yelling at them. They know there's pain. They've had to face it every day for a long time.
I for one am actually very encouraged by the amount of activity around improving packaging. There are lots of people that care deeply about this stuff and are trying to get it right and improve things. We should be supportive of them and help where we can. If something isn't working like you think it should, I would always try asking first. In my experience, people on this list are very helpful in that regard, perhaps in part because they know certain things are in flux. --Chris
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Aug 4, 2017, at 4:43 PM, Lucas Boppre Niehues <lucasboppre@gmail.com> wrote:
I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened.
Disclaimers: - Any and all feedback is welcome, but please be charitable. - I'm angry *at* things, I hope not to offend anybody. - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet.
For formatting reasons the text is here:
https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 <https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21>
Thank you for your attention _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Going through your text, there’s a mixture of things that sort of combined here, I’m kind of just going to stream of conscious reply to them here, because I can’t reply inline.
error: No dist file created in earlier command
Yes, ``setup.py upload`` *requires* you to create the files it is uploading within the same command during the upload. This is one of the reasons that twine as a tool exists and is generally recommended.
It's the exact same file. You could add a check for it and ignore, but no, it had to be an error.
This seems like a reasonable feature request, please file an issue with Warehouse asking for it so I don’t forget it.
It seems that long_description in setup.py was missed entirely.
I’m not sure what happened here. If you share the package I can probably figure it out though. I’m not currently aware of any bugs with how long_description is handled.
By the way, why was this feature included in the register command?
This was not an explicit feature of the register command, and more of an accident in the way things got implemented in PyPI’s decade+ history.
[Stuff about editing the description]
You located the issue on Warehouse. The fundamental issue here is trying to find the right balance between constraining authors so that end users can have a consistent behavior between packages and giving authors power to best manage their own projects. Discussion about editing the description is probably best handled on the warehouse issue though (Which I forgot to update with my thoughts!).
My setup.py has been communicating with https://upload.pypi.org/legacy/ by default. Note the legacy part.
This is confusing, and it is essentially due to the fact there are two things here which are considered “legacy”. One is the legacy codebase/deployment that currently powers pypi.python.org which are are slowly replacing and migrating things over to the new, modern code base that powers pypi.org <http://pypi.org/>. The other thing is the actual upload API itself, which has stayed the same currently, but which I plan to replace at some point in the future. This API is also considered “legacy” (it just doesn’t have a replacement yet). So the legacy in https://upload.pypi.org/legacy/ <https://upload.pypi.org/legacy/> has to do with the API rather than the location/deployment. I probably should have named it something other than /legacy/, my goal was mostly that changing URLs is hard (it requires a bunch of documentation/updates in different packages and N years for that to percolate out) so since we were forcing people to change the URLs with the move from pypi.python.org <http://pypi.python.org/> to pypi.org <http://pypi.org/>, might as well get it all done at once. It might be reasonable to name it something else now, and just keep the /legacy/ around as an alias. I’m not sure if that adds or subtracts from the confusion, but if you think that would have helped you, please open a new issue on Warehouse.
The "new APIs" seem to be missing at last one important feature
Some of this is purposeful as we attempt to rein in some of the more “random” features that PyPI has grown over time. Some of it is accidental as there’s not really any documentation about what PyPI provides, and people have, over the decade of it’s existence, depended on all sorts of random bits and pieces of PyPI which makes it hard to tell what people are even using.
I must have missed the deprecation period that preceded the complete removal of the old APIs. A friendly email would have been nice if you expect me to migrate to a new solution.
This is a hard balance to strike. When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. Particularly for changes like this where there belief is that for most people, the impact should be minimal, if there is any at all. I do try to at least tweet and post to distutils-sig about possibly disruptive changes like this. For this specific change, that thread was at https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html <https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html>.
The new website is "pre-production”.
This is mostly just a warning at this point that the UI might return funky results, some things might not be great yet, and to not start throwing the full force of people’s production traffic at it. The current hosting arrangement can’t handle the full amount of traffic we get from ``pip install …`` (but uploads are fine!) and we’re working on building on the new infrastructure with our future hosting arrangement that can handle the full production load.
HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/ <https://upload.pypi.org/legacy/>
The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/ <https://www.python.org/dev/peps/pep-0527/>.
Maybe the tutorial is outdated, and TestPyPI supports auto-registration now?
I don’t think anyone has kept the tutorial on wiki.python.org <http://wiki.python.org/> up to date. To be frank, I don’t even know how to update wiki.python.org <http://wiki.python.org/>. Generally we’ve been trying to consolidate packaging documentation onto packaging.python.org — although I don’t think we have one for TestPyPI yet. TestPyPI itself is also severely under maintained. I don’t use it so I rarely even think about it and my time is stretched thin to an extreme degree. I plan on getting rid of TestPyPI at some point and replacing it with something better, You can see discussion about that at https://github.com/pypa/warehouse/issues/726 <https://github.com/pypa/warehouse/issues/726>.
I try python setup.py bdist upload
As I mentioned above, we deprecated/removed everything but sdist, wheels, and eggs as per PEP 527. We could probably do better error messages here though.
I can't even specify the description of a package, not even during registration or upload.
Tell me your package name privately or publicly and I’ll figure out what went wrong.
I cannot even report the issues. [ ... ] because there's no clear location to report them.
This is a problem, and we don’t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends up getting filed in the incorrect place, we just redirect people tot he correct place.
I am trying to donate my precious free time to the Python community by offering free software, receiving absolutely no compensation whatsoever, and every step of the experience has been shitty.
I/we understand that, and we’re generally trying to do the best we can with what limited resources we have. The flip side of this is that we’re *also* (for the most part) donating our precious free time to the Python community to run PyPI (and the other PSF services too!). For the record, in July PyPI used something just under 1 Petabyte of bandwidth and served 38 billion HTTP requests. People working on that include myself with 16h/week of paid time (split between *all* of the packaging related tools I work on) plus something like 20-30h/week of volunteer time (on top of the additional 24h/week of time I spent working on my employers projects). On the ops side there is another individual (who I won’t name here to prevent him from getting drug into this [1]) who donates his spare time as well (though I don’t know offhand how much that is). As far as keeping PyPI running as well as getting the new code base developed and deployed… that’s about it [2]. This is a service used by ~everyone in the Python community without even a single full time person on it. We’re trying to slowly migrate to a new code base which is (A) easier to maintain/scale taking up fewer hours (B) easier to actually work on to develop new features, which will hopefully help attract and retain new contributors and (C) has removed decades of cruft that has built up and either explicitly made something supported or explicit decided it is no longer supported, instead of a quasi supported “well it’s what PyPI does” scenario we have now. This probably won’t be the last less than perfect phase of the migration to the new code base. My best recommendation if you don’t want to discover these things JIT (even though we *do* try to minimize them) is to subscribe to and follow distutils-sig. Alternatively if you want to get more involved, one thing we’re severely lacking is a better mechanism for communicating these things to users, and if you wanted to propose/get sign off/implement/etc something for that, then that is another possibility. Finally if you wanted you could also get involved in actually shaping the direction of these tools, either by discussing on distutils-sig and in their respective issue trackers, or by becoming an active contributor. That’s the best way to ensure that your relevant use cases are considered during the design or execution of things. [1] Folks can yell at me if they want, don’t go yelling at other contributors. Generally the buck stops with me anyways. [2] There is a designer working on the new code base, and there have been some other contributors, but since this is about the APIs and maintenance I left them out. — Donald Stufft
On Fri, Aug 4, 2017 at 2:51 PM, Donald Stufft <donald@stufft.io> wrote:
This API is also considered “legacy” (it just doesn’t have a replacement yet). So the legacy in https://upload.pypi.org/legacy/ has to do with the API rather than the location/deployment.
I probably should have named it something other than /legacy/, my goal was mostly that changing URLs is hard (it requires a bunch of documentation/updates in different packages and N years for that to percolate out) so since we were forcing people to change the URLs with the move from pypi.python.org to pypi.org, might as well get it all done at once. It might be reasonable to name it something else now, and just keep the /legacy/ around as an alias. I’m not sure if that adds or subtracts from the confusion, but if you think that would have helped you, please open a new issue on Warehouse.
I guess calling it /v1 would avoid these issues and also make life easier when v3 comes around... -n -- Nathaniel J. Smith -- https://vorpus.org
Thank you for the kind responses. I hope I haven't offended anybody, and please know that I'm incredibly grateful for all the sweat and blood that has been poured into these projects. I'm complaining about the current situation to show the largest pain points from my perspective. That's also why I'm posting here and not somewhere more visible. James, please see this is not just me lashing out to vent. This is the first time I ever post a rant. I'm pointing out the problems I experienced in the hopes that the people responsible notice the patterns, and derive solutions from them. I'm sorry for the flowery language, but be sure that the original events contained a lot more expletives, and I think people's reactions are important to prioritization. I've changed development focus before because someone emailed me complaining about a specific part of a project. Donald, thank you very much for the reasoned, helpful and throughout response. Here's my reply to the most important points:
The fundamental issue here is trying to find the right balance between constraining authors so that end users can have a consistent behavior between packages and giving authors power to best manage their own projects. [...] Some of this is purposeful as we attempt to rein in some of the more “random” features that PyPI has grown over time
I'm very glad to hear that. Speaking for myself, I wouldn't mind if PyPI became an API-only service that hosts immutable packages. Even if it makes my life a bit harder and I have to re-invent my release process, I would rather the platform be more solid.
The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/.
That's... interesting. Thank you for the link, this is the first time I'm seeing it. I really liked the Windows installers, but I understand the need to slim down the infrastructure. Also, I now see some of my confusion was because of the invisible "allow legacy file types to be uploaded" flag in some of the projects but not others.
I can't even specify the description of a package, not even during registration or upload. Tell me your package name privately or publicly and I’ll figure out what went wrong.
I cannot even report the issues. [ ... ] because there's no clear location to report them. This is a problem, and we don’t really have a good solution for them. I
https://pypi.python.org/pypi/mouse/0.6.0 https://test.pypi.org/project/mouse/0.6.0 The long description was originally Markdown, and converted to RST by pandoc. I would 100% understand if this conversion triggered some bug. My gripe was that one by one my debugging tools failed in confusing ways. think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends
up getting filed in the incorrect place, we just redirect people tot he correct place.
Just my 2 cents, but I often not post a bug report at all for fear of wasting the wrong person's time. I'll keep this case in mind, but you may be missing some important reports with this configuration. The "packaging-problems" repo was almost perfect, but I avoided it due to the feeling of being abandoned or exclusive for PyPI developers. An explicit "It's ok to post X, Y and Z bugs here if you are unsure" would have changed my mind.
This is a service used by ~everyone in the Python community without even a single full time person on it.
When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. Particularly for changes like this where there belief is that for most
I'm deeply grateful for the work you and the other contributors have put here. I see the project is in good hands, if an insufficient number of them. But this is also an incredibly worrying statement. I work in security, and I know I'll have nightmares after reading this :( people, the impact should be minimal, if there is any at all. That's awful, I'm sorry. Maybe an opt-in system during upload, to appease both crowds? I know I would set "email_me_important_announcements=True" in a heartbeat in my setup.py. Also note that people who don't like impact won't like the changes regardless if you email them or not, though I understand it's not nice to receive hate-mail. To be fair this migration is indeed mostly smooth. PyPI.org looks good, pypi.python.org is still working well (minus one or two deprecated endpoints leading to 410 gone), my setuptools automatically migrated to the legacy API, and I have nothing but praise for twine. And I found the use of HTTP error responses including detailed migration information to be a clever last-resort attempt at reaching the user. I faced several problems in my rant, but fixing any one of them would have solved my situation. I'm sure you guys can get there. Thank you for your patience and hard work, and I hope PyPI continues being one of the pillars of the community.
On Aug 4, 2017, at 8:42 PM, Lucas Boppre Niehues <lucasboppre@gmail.com> wrote:
I can't even specify the description of a package, not even during registration or upload. Tell me your package name privately or publicly and I’ll figure out what went wrong.
https://pypi.python.org/pypi/mouse/0.6.0 <https://pypi.python.org/pypi/mouse/0.6.0> https://test.pypi.org/project/mouse/0.6.0 <https://test.pypi.org/project/mouse/0.6.0>
The long description was originally Markdown, and converted to RST by pandoc. I would 100% understand if this conversion triggered some bug. My gripe was that one by one my debugging tools failed in confusing ways.
So you uploaded the wheel first, which means that’s where PyPI would have gotten the long_description from. When I download that wheel and open it up and look at mouse-0.6.0.dist-info/METADATA I see a file that looks like: --- Metadata-Version: 2.0 Name: mouse Version: 0.6.0 Summary: Hook and simulate global mouse events in pure Python Home-page: https://github.com/boppreh/mouse Author: BoppreH Author-email: boppreh@gmail.com License: MIT mouse --- So it appears that the wheel isn’t being properly generated. I would expect the full contents of your long description to show up, and not just the word “mouse’. I didn’t dig into it further to see if this is setuptools, wheel, or your pandoc version at fault, but my guess is it’s one of those. Given lots of people are successfully uploading, my first guess would be something in the pandoc build pipeline you have— but it could also be an edge case triggering a latent bug in setuptools or wheel.
I cannot even report the issues. [ ... ] because there's no clear location to report them. This is a problem, and we don’t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends up getting filed in the incorrect place, we just redirect people tot he correct place.
Just my 2 cents, but I often not post a bug report at all for fear of wasting the wrong person's time. I'll keep this case in mind, but you may be missing some important reports with this configuration. The "packaging-problems" repo was almost perfect, but I avoided it due to the feeling of being abandoned or exclusive for PyPI developers. An explicit "It's ok to post X, Y and Z bugs here if you are unsure" would have changed my mind.
I’ll see what I can do here. It might just make sense for these related set of tools to just ditch the Github issue trackers and share an issue tracker that can divide things up by project somehow, but where we can readily move issues between them. Maybe we just need something (or someones) able to triage incoming issues on a meta tracker and redirect people. Not sure, I’ll give it some thought. — Donald Stufft
So it appears that the wheel isn’t being properly generated.
Ah, this helps a lot. I didn't realize it's the first upload that defines the description for the package. Testing different configurations I found that the wheel generation chokes and ignores everything after the first Windows-style newline '\r\n'. Meanwhile sdist just copies the string exactly as-is, but inside a Unix-style file, resulting in a PKG-INFO that doesn't break anything but looks weird. And you're also right that this is a bit of pypandoc's fault, because it returns a Windows-style string even if the input was Unix-style, but this is no reason to mix styles or choke. I can open a bug report later if you prefer. Thank you for the help, that solves the only unexplained issue I had.
I created an issue on setuptools tracker about Windows newlines ( https://github.com/pypa/setuptools/issues/1126), and another on Warehouse about the "Invalid file extension" message and PEP 517 ( https://github.com/pypa/warehouse/issues/2281). I'm not creating one for ignoring duplicated files in `upload` because twine already has this feature (--skip-existing), and other people may be relying on this behavior. Also, I agree with Nathaniel Smith that numbered versioning of the API (/v0/, /v1/, /v/2) would be preferable to /legacy/.
On 5 August 2017 at 14:15, Lucas Boppre Niehues <lucasboppre@gmail.com> wrote:
Also, I agree with Nathaniel Smith that numbered versioning of the API (/v0/, /v1/, /v/2) would be preferable to /legacy/.
I don't have a lot to add to Donald's other responses (Thanks Donald!), but wanted to register a +1 on the idea of replacing the current "/legacy/" upload URL with a "/v0/" URL (with "/legacy/" preserved as a compatibility alias). The "v0" version number still indicates that it isn't really a properly defined service API, without the potential for confusion between "legacy PyPI" and "legacy upload API". I'll also note that the lack of good service announcement channels is in some respects a PSF level problem, rather than just a PyPI level one, so there's a meta-tracker issue for that here: https://github.com/python/psf-infra-meta/issues/1 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 5 August 2017 at 03:12, Donald Stufft <donald@stufft.io> wrote:
I’ll see what I can do here. It might just make sense for these related set of tools to just ditch the Github issue trackers and share an issue tracker that can divide things up by project somehow, but where we can readily move issues between them. Maybe we just need something (or someones) able to triage incoming issues on a meta tracker and redirect people. Not sure, I’ll give it some thought.
I'd be willing to subscribe to a generalised tracker and triage issues there to direct people to the correct projects. I'm less sure that a single "Python packaging tools" tracker would be worthwhile (migration of existing issues seems like it would be a big exercise). Paul
On Fri, Aug 4, 2017 at 5:42 PM, Lucas Boppre Niehues <lucasboppre@gmail.com> wrote:
The long description was originally Markdown, and converted to RST by pandoc. I would 100% understand if this conversion triggered some bug.
It's a good idea to run docutils on it yourself -- but in any case, broken RST has always resulted in plain text rendering on PyPi in the past -- as it should. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Donald, thanks for the thoughtful and complete reply to the original poster. On Fri, Aug 4, 2017 at 3:52 PM Donald Stufft <donald@stufft.io> wrote:
Going through your text, there’s a mixture of things that sort of combined here, I’m kind of just going to stream of conscious reply to them here, because I can’t reply inline.
[...]
I must have missed the deprecation period that preceded the complete removal of the old APIs. A friendly email would have been nice if you expect me to migrate to a new solution.
This is a hard balance to strike. When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. Particularly for changes like this where there belief is that for most people, the impact should be minimal, if there is any at all. I do try to at least tweet and post to distutils-sig about possibly disruptive changes like this. For this specific change, that thread was at https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html.
[...]
HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/
The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/.
[...]
I try python setup.py bdist upload
As I mentioned above, we deprecated/removed everything but sdist, wheels, and eggs as per PEP 527. We could probably do better error messages here though.
Has the team considered putting deprecation messages in the tools to warn of these deprecations before they're removed? A warning for the API deprecations in pip or the underlying libraries, though not the perfect solution, might go a long way towards informing users of impending changes. Thanks, as always, for your efforts. Sincerely, e.
one point: I probably should have named it something other than /legacy/,
yes, it should have -- names matter! and having a "legacy" in teh name when there is not "modern" or "current", or, indeed, anything else is very confusing. pypi.org, might as well get it all done at once. It might be reasonable to
name it something else now, and just keep the /legacy/ around as an alias. I’m not sure if that adds or subtracts from the confusion, but if you think that would have helped you, please open a new issue on Warehouse.
I think that would be better -- something like "version1" something would be more clear. However, we have a major jumble of semi-out of date documentation scattered all over teh internet -- so ANOTHER name change, might just make things worse -- it's also confusing to have two names for the exact same thing! tough call.
HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/
The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. A nice message to that effect would be great. Also -- I know that the various tools involved are maintained by different people, but it sure would be nice if, for instance, the latest versions of setuptools and twine would not try to upload non longer supported file types! I've thought for a couple years that we need a setuptool-lite that would ONLY do what we think is really should do -- but no one ele seemed to think that would be useful.
Maybe the tutorial is outdated, and TestPyPI supports auto-registration now?
I don’t think anyone has kept the tutorial on wiki.python.org up to date. To be frank, I don’t even know how to update wiki.python.org. Someone that does really needs to add a "this isn't updated, please see packaging.python.org note on there !! As far as keeping PyPI running as well as getting the new code base
developed and deployed… that’s about it [2]. This is a service used by ~everyone in the Python community without even a single full time person on it.
This is a core service -- we really need to figure out a better way to keep it maintained. I suspect the issue is project management, rather than volunteers -- there are a heck of a lot of talented folks that depend on these services. Of course, I have no good ideas about how to move forward -- is there anyone at the PSF that could take leadership on this? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Also -- I know that the various tools involved are maintained by different people, but it sure would be nice if, for instance, the latest versions of setuptools and twine would not try to upload non longer supported file types!
The tools should have better reporting on the errors pypi gives, but hard coding into the tools what pypi will and won’t accept makes policy decisions on pypi more painful to change.
Issues created: * Ignore dup uploads instead of error: https://github.com/pypa/warehouse/issues/2284 * Rename "/legacy" API URL: https://github.com/pypa/warehouse/issues/2285 * Propose "packaging announce" mailing list: https://github.com/pypa/packaging-problems/issues/98 o Previous issue noting challenges in communicating on packaging, many of the same concerns still seem relevant: https://github.com/pypa/packaging-problems/issues/21 * Explain what "pre-production" means on Warehouse: https://github.com/pypa/warehouse/issues/2287 * Deprecation notice added to: https://wiki.python.org/moin/TestPyPI o Issue created to enhance docs for TestPyPI: https://github.com/pypa/python-packaging-user-guide/issues/362 * Better error message for PEP 527 accepted file types: https://github.com/pypa/packaging-problems/issues/99 I hope I got all the right places for those issues. Thanks Donald for the clear reply from which it was easy to create these separate issues. I didn't realize you had so little paid time to work on packaging issues. I'm surprised with all the large companies that use Python that the PSF can't secure more donations to invest in fixing these issues. Or that someone like Redhat can't get a couple engineers to devote time for a couple years like they do to other OSS projects. (No criticism intended, just musing). *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 08/04/2017 05:51 PM, Donald Stufft wrote:
On Aug 4, 2017, at 4:43 PM, Lucas Boppre Niehues <lucasboppre@gmail.com <mailto:lucasboppre@gmail.com>> wrote:
I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened.
Disclaimers: - Any and all feedback is welcome, but please be charitable. - I'm angry *at* things, I hope not to offend anybody. - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet.
For formatting reasons the text is here:
https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21
Thank you for your attention _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <mailto:Distutils-SIG@python.org> https://mail.python.org/mailman/listinfo/distutils-sig
Going through your text, there’s a mixture of things that sort of combined here, I’m kind of just going to stream of conscious reply to them here, because I can’t reply inline.
error: No dist file created in earlier command
Yes, ``setup.py upload`` *requires* you to create the files it is uploading within the same command during the upload. This is one of the reasons that twine as a tool exists and is generally recommended.
It's the exact same file. You could add a check for it and ignore, but no, it had to be an error.
This seems like a reasonable feature request, please file an issue with Warehouse asking for it so I don’t forget it.
It seems that long_description in setup.py was missed entirely.
I’m not sure what happened here. If you share the package I can probably figure it out though. I’m not currently aware of any bugs with how long_description is handled.
By the way, why was this feature included in the register command?
This was not an explicit feature of the register command, and more of an accident in the way things got implemented in PyPI’s decade+ history.
[Stuff about editing the description]
You located the issue on Warehouse. The fundamental issue here is trying to find the right balance between constraining authors so that end users can have a consistent behavior between packages and giving authors power to best manage their own projects. Discussion about editing the description is probably best handled on the warehouse issue though (Which I forgot to update with my thoughts!).
My setup.py has been communicating with https://upload.pypi.org/legacy/ by default. Note the legacy part.
This is confusing, and it is essentially due to the fact there are two things here which are considered “legacy”. One is the legacy codebase/deployment that currently powers pypi.python.org <http://pypi.python.org> which are are slowly replacing and migrating things over to the new, modern code base that powers pypi.org <http://pypi.org>. The other thing is the actual upload API itself, which has stayed the same currently, but which I plan to replace at some point in the future. This API is also considered “legacy” (it just doesn’t have a replacement yet). So the legacy in https://upload.pypi.org/legacy/ has to do with the API rather than the location/deployment.
I probably should have named it something other than /legacy/, my goal was mostly that changing URLs is hard (it requires a bunch of documentation/updates in different packages and N years for that to percolate out) so since we were forcing people to change the URLs with the move from pypi.python.org <http://pypi.python.org> to pypi.org <http://pypi.org>, might as well get it all done at once. It might be reasonable to name it something else now, and just keep the /legacy/ around as an alias. I’m not sure if that adds or subtracts from the confusion, but if you think that would have helped you, please open a new issue on Warehouse.
The "new APIs" seem to be missing at last one important feature
Some of this is purposeful as we attempt to rein in some of the more “random” features that PyPI has grown over time. Some of it is accidental as there’s not really any documentation about what PyPI provides, and people have, over the decade of it’s existence, depended on all sorts of random bits and pieces of PyPI which makes it hard to tell what people are even using.
I must have missed the deprecation period that preceded the complete removal of the old APIs. A friendly email would have been nice if you expect me to migrate to a new solution.
This is a hard balance to strike. When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. Particularly for changes like this where there belief is that for most people, the impact should be minimal, if there is any at all. I do try to at least tweet and post to distutils-sig about possibly disruptive changes like this. For this specific change, that thread was at https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html.
The new website is "pre-production”.
This is mostly just a warning at this point that the UI might return funky results, some things might not be great yet, and to not start throwing the full force of people’s production traffic at it. The current hosting arrangement can’t handle the full amount of traffic we get from ``pip install …`` (but uploads are fine!) and we’re working on building on the new infrastructure with our future hosting arrangement that can handle the full production load.
HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/
The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/.
Maybe the tutorial is outdated, and TestPyPI supports auto-registration now?
I don’t think anyone has kept the tutorial on wiki.python.org <http://wiki.python.org> up to date. To be frank, I don’t even know how to update wiki.python.org <http://wiki.python.org>. Generally we’ve been trying to consolidate packaging documentation onto packaging.python.org <http://packaging.python.org> — although I don’t think we have one for TestPyPI yet.
TestPyPI itself is also severely under maintained. I don’t use it so I rarely even think about it and my time is stretched thin to an extreme degree. I plan on getting rid of TestPyPI at some point and replacing it with something better, You can see discussion about that at https://github.com/pypa/warehouse/issues/726.
I try python setup.py bdist upload
As I mentioned above, we deprecated/removed everything but sdist, wheels, and eggs as per PEP 527. We could probably do better error messages here though.
I can't even specify the description of a package, not even during registration or upload.
Tell me your package name privately or publicly and I’ll figure out what went wrong.
I cannot even report the issues. [ ... ] because there's no clear location to report them.
This is a problem, and we don’t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends up getting filed in the incorrect place, we just redirect people tot he correct place.
I am trying to donate my precious free time to the Python community by offering free software, receiving absolutely no compensation whatsoever, and every step of the experience has been shitty.
I/we understand that, and we’re generally trying to do the best we can with what limited resources we have. The flip side of this is that we’re *also* (for the most part) donating our precious free time to the Python community to run PyPI (and the other PSF services too!). For the record, in July PyPI used something just under 1 Petabyte of bandwidth and served 38 billion HTTP requests.
People working on that include myself with 16h/week of paid time (split between *all* of the packaging related tools I work on) plus something like 20-30h/week of volunteer time (on top of the additional 24h/week of time I spent working on my employers projects). On the ops side there is another individual (who I won’t name here to prevent him from getting drug into this [1]) who donates his spare time as well (though I don’t know offhand how much that is).
As far as keeping PyPI running as well as getting the new code base developed and deployed… that’s about it [2]. This is a service used by ~everyone in the Python community without even a single full time person on it.
We’re trying to slowly migrate to a new code base which is (A) easier to maintain/scale taking up fewer hours (B) easier to actually work on to develop new features, which will hopefully help attract and retain new contributors and (C) has removed decades of cruft that has built up and either explicitly made something supported or explicit decided it is no longer supported, instead of a quasi supported “well it’s what PyPI does” scenario we have now.
This probably won’t be the last less than perfect phase of the migration to the new code base. My best recommendation if you don’t want to discover these things JIT (even though we *do* try to minimize them) is to subscribe to and follow distutils-sig. Alternatively if you want to get more involved, one thing we’re severely lacking is a better mechanism for communicating these things to users, and if you wanted to propose/get sign off/implement/etc something for that, then that is another possibility. Finally if you wanted you could also get involved in actually shaping the direction of these tools, either by discussing on distutils-sig and in their respective issue trackers, or by becoming an active contributor. That’s the best way to ensure that your relevant use cases are considered during the design or execution of things.
[1] Folks can yell at me if they want, don’t go yelling at other contributors. Generally the buck stops with me anyways. [2] There is a designer working on the new code base, and there have been some other contributors, but since this is about the APIs and maintenance I left them out.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 7 August 2017 at 12:19, Randy Syring <randy@thesyrings.us> wrote:
Thanks Donald for the clear reply from which it was easy to create these separate issues. I didn't realize you had so little paid time to work on packaging issues. I'm surprised with all the large companies that use Python that the PSF can't secure more donations to invest in fixing these issues. Or that someone like Redhat can't get a couple engineers to devote time for a couple years like they do to other OSS projects. (No criticism intended, just musing).
Please feel free to criticise Python's commercial redistributors loudly and publicly on this point, as our collective failure to deal with the situation appropriately is something our respective customers *should* be questioning and challenging us over, especially when support contracts are up for renegotiation :) As far as I've been able to work out, the main problem appears to be that the professional Python user base currently self-selects into two primary categories: 1. "We're happy to rely on self-support and community support, so we don't need a Python vendor" 2. "Our ops team pays a commercial Python runtime vendor, but our dev teams have no ability to file support requests with that vendor" In combination, these behaviours mean vendors' commercial demand signals are thoroughly broken, creating an environment where it becomes difficult to make the business case that increased upstream investment will lead to proportional increases in revenue. (To help illustrate the scale of the problem, while omitting names to protect the guilty: "Do people in large organisations actually use Python for web development?" is something I have had a redistributor ask as a genuine question in the same month where Python topped IEEE Spectrum's annual survey of language popularity for the first time. When we have that level of disconnect in the perception of ecosystem adoption, something significant is clearly amiss) At the Python Software Foundation level, one of the things we've realised we need to do is to provide more streamlined ways for organisations using Python to direct funding specifically towards infrastructure support for PyPI (with the Packaging Working Group then being responsible for managing those funds), since "We should fund the platform that hosts our Python dependencies" is a potentially easier pitch for engineering teams to make than advocating for their employers to become full PSF Sponsor members. Towards that end, we've recently applied to Mozilla for a development grant to help the PSF fund the design, development, and project management efforts needed to get those improved infrastructure sustainability measures in place, as well as getting to the point where the legacy PyPI service can be shut down entirely, eliminating one of the current major barriers to improving the shared infrastructure (credit goes to Eric Holscher for getting that request through to the grant submission stage - the Packaging WG had started drafting it some time ago, but we'd stalled before reaching the point of actually submitting it). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (11)
-
Alex Walters -
Chris Barker -
Chris Jerdonek -
Donald Stufft -
Eric Brunson -
James Bennett -
Lucas Boppre Niehues -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Randy Syring