What's the use case of testpypi?
When we locked down pypi to prevent uploading an sdist to overwrite a previous I remember that some people wanted a brief window to check for brown paper bag issues and be able to upload a new tarball in that window if needed. IIRC, those people were told to use testpypi for that sort of thing. Upload potential tarball to testpypi. If it works, go ahead and rerun to upload to the real pypi. If it doesn't, fix it and try again. This past week I decided to try out that workflow for a project that I'm managing the releases for and ran into a snag. testpypi has the same behaviour as production pypi wherein you can only upload a given sdist once and afterwards it's no longer allowed. For the use case above this is problematic. It essentially changes the idea of "test your release on testpypi before making a release" into "You can have to chances to get it right if you use testpypi" which, although better than uploading directly to pypi, still leaves a lot of room for error (let's face it: I know that if I'm stuck releasing late at night due to time constraints and make a mistake, chances are better than normal that my fix won't be the perfection that my ordinary code is and could have other showstopper bugs that I'd want my testing to catch as well ;-) Is this something that we could change for testpypi? It could be implemented in many ways: straight overwrite, being able to destroy a version so that it seems to never have existed, or being able to destroy and recreate a package so that it has no uploaded sdists recorded. On the other side of the usefulness of enabling the testing use case above, such a change would be a difference between testpypi and production pypi meaning that it would no longer be testing exactly the same functionality as will be deployed in production. I'm not sure if that's a more important consideration or not. I figured that unless I asked, I would never know the answer :-) Thanks, -Toshio
TestPyPI can be useful for those who are not package maintainers, but want to learn about Python Packaging and need a 'playground' to experiment with it. Mariatta Wijaya On Mon, Oct 30, 2017 at 10:41 AM, Toshio Kuratomi <a.badger@gmail.com> wrote:
When we locked down pypi to prevent uploading an sdist to overwrite a previous I remember that some people wanted a brief window to check for brown paper bag issues and be able to upload a new tarball in that window if needed. IIRC, those people were told to use testpypi for that sort of thing. Upload potential tarball to testpypi. If it works, go ahead and rerun to upload to the real pypi. If it doesn't, fix it and try again.
This past week I decided to try out that workflow for a project that I'm managing the releases for and ran into a snag. testpypi has the same behaviour as production pypi wherein you can only upload a given sdist once and afterwards it's no longer allowed. For the use case above this is problematic. It essentially changes the idea of "test your release on testpypi before making a release" into "You can have to chances to get it right if you use testpypi" which, although better than uploading directly to pypi, still leaves a lot of room for error (let's face it: I know that if I'm stuck releasing late at night due to time constraints and make a mistake, chances are better than normal that my fix won't be the perfection that my ordinary code is and could have other showstopper bugs that I'd want my testing to catch as well ;-)
Is this something that we could change for testpypi? It could be implemented in many ways: straight overwrite, being able to destroy a version so that it seems to never have existed, or being able to destroy and recreate a package so that it has no uploaded sdists recorded.
On the other side of the usefulness of enabling the testing use case above, such a change would be a difference between testpypi and production pypi meaning that it would no longer be testing exactly the same functionality as will be deployed in production. I'm not sure if that's a more important consideration or not. I figured that unless I asked, I would never know the answer :-)
Thanks, -Toshio _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Oct 30, 2017, at 1:41 PM, Toshio Kuratomi <a.badger@gmail.com> wrote:
I figured that unless I asked, I would never know the answer :-)
TestPyPI has a couple of use cases, and it’s not very good at any of them. One use case is to serve as a staging site for production PyPI to test changes before they go to live. If I had to pick any use case as it’s “core” use case, it would be this one, it’s currently optimized for it and we’re unlikely to make changes that deviate from it. Unfortunately it’s not really good at this because staging sites are generally not a great way to prevent regression (versus something like automated testing), so with Warehouse TestPyPI is no longer a staging site for the production code, and they are deployed at the same time. Another use case is the one you specified, someone wants to test their releases prior to uploading to the “real” PyPI. TestPyPI is not great at this use case both because, as you noted, it has the same “you can only upload a file once” semantics as PyPI does, but also because it doesn’t have a full copy of everything on PyPI (and worse, it potentially has broken or insecure or malicious versions of your dependencies) so pip might either be blatantly broken OR produce subtly broken installs. Long term the solution that I believe has merit for this, is the ability to, on regular PyPI, “stage” an upload to a temporary location that would automatically be deleted after some period of time, and that once you were satisfied with it, could be published/committed and thus be available to normal ``pip install`` invocations. This not only would allow testing a release beforehand, but it would also allow projects that produce many different artifacts across a variety of platforms to collect them all and upload them and make them all available immediately instead of one at a time. You can see more about this idea/use case at https://github.com/pypa/warehouse/issues/726 <https://github.com/pypa/warehouse/issues/726>. The third relevant use case of PyPI is people simply messing around or doing testing of a generalized workflow tool but not specific to a specific package. This could also include people who are learning how to produce a package, but using a trivial throw away project (the infamous printer of nested lists is an example of this). TestPyPI can be used for this as well, but again it’s not really good at it. The key difference here is that ideally these projects are either ephemeral or namespaced to the specific user. I don’t have a great idea for how best to handle this, the immediate thought is a secondary deployment of Warehouse (such as what TestPyPI is now), that just periodically and automatically purges any projects > N days old to give it a more of a “sandbox” vibe and use case. Given the aforementioned staging feature, we would likely otherwise keep the same ruleset as exists on the production PyPI (and ideally, we’d be re-using credentials too, instead of it literally being a complete secondary deployment). That idea isn’t set in stone and I haven’t done a lot of thinking about it yet, but discussion of that particular use case and ideas to fix it would take place in https://github.com/pypa/warehouse/issues/2286 <https://github.com/pypa/warehouse/issues/2286>. So longer term, I hope to shut down TestPyPI and replace it with features and/or deployments more suited to whatever specific task/use case we’re hoping to have served by it, rather than just sort of being poor at everything. - dstufft
On 31 October 2017 at 03:41, Toshio Kuratomi <a.badger@gmail.com> wrote:
When we locked down pypi to prevent uploading an sdist to overwrite a previous I remember that some people wanted a brief window to check for brown paper bag issues and be able to upload a new tarball in that window if needed. IIRC, those people were told to use testpypi for that sort of thing. Upload potential tarball to testpypi. If it works, go ahead and rerun to upload to the real pypi. If it doesn't, fix it and try again.
Ideally we'd be recommending https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking to develop a robust pre-release artifact testing workflow. While we mention it a couple of times on packaging.python.org [1,2], and include it in the packaging related Non-PyPA Projects list [3], we don't really emphasise that it makes a much better platform for pre-release testing and private experimentation than PyPI itself does (see https://devpi.net/ for an example of a deployed instance with multiple distinct user namespaces). Given Donald's comment about the current test PyPI not being particularly good at any of its roles, perhaps it would make sense to aim to replace it with a periodically purged DevPi instance? Cheers, Nick. [1] https://packaging.python.org/guides/index-mirrors-and-caches/?highlight=devp... [2] https://packaging.python.org/guides/hosting-your-own-index/?highlight=devpi [3] https://packaging.python.org/key_projects/?highlight=devpi#non-pypa-projects -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
Ideally we'd be recommending https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking to develop a robust pre-release artifact testing workflow.
While we mention it a couple of times on packaging.python.org [1,2], and include it in the packaging related Non-PyPA Projects list [3], we don't really emphasise that it makes a much better platform for pre-release testing and private experimentation than PyPI itself does (see https://devpi.net/ for an example of a deployed instance with multiple distinct user namespaces).
Given Donald's comment about the current test PyPI not being particularly good at any of its roles, perhaps it would make sense to aim to replace it with a periodically purged DevPi instance? [...]
Do the two share enough common code for successful uploading to a devpi instance to be indicative of whether PyPI will accept or reject on the grounds of, e.g., invalid trove classifiers (this one in particular has been the most common preventable but otherwise untestable upload failure our community encounters). -- Jeremy Stanley
On Tue, Oct 31, 2017, at 12:49 PM, Jeremy Stanley wrote:
Do the two share enough common code for successful uploading to a devpi instance to be indicative of whether PyPI will accept or reject on the grounds of, e.g., invalid trove classifiers (this one in particular has been the most common preventable but otherwise untestable upload failure our community encounters).
It's feasible for tools to test this before upload. Flit does this. If anyone wants to adapt the code to use in other tools, it's here: https://github.com/takluyver/flit/blob/ca081197df229a0df07e98c6dece13e62e2bd... It's definitely useful to have a testing site based on the same code as PyPI, but I hope that with smarter tools, building invalid packages should be less common. Thomas
On 2017-10-31 13:09:34 +0000 (+0000), Thomas Kluyver wrote: [...]
If anyone wants to adapt the code to use in other tools, it's here:
https://github.com/takluyver/flit/blob/ca08119/flit/inifile.py#L63-L108 [...]
Thanks for the pointer! That would make for a really great stand-alone linting utility or maybe a flake8 plug-in (particularly if it vendored a copy of the classifiers list and included an option/flag to skip downloading). -- Jeremy Stanley
On 31 October 2017 at 22:49, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
Ideally we'd be recommending https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking to develop a robust pre-release artifact testing workflow.
While we mention it a couple of times on packaging.python.org [1,2], and include it in the packaging related Non-PyPA Projects list [3], we don't really emphasise that it makes a much better platform for pre-release testing and private experimentation than PyPI itself does (see https://devpi.net/ for an example of a deployed instance with multiple distinct user namespaces).
Given Donald's comment about the current test PyPI not being particularly good at any of its roles, perhaps it would make sense to aim to replace it with a periodically purged DevPi instance? [...]
Do the two share enough common code for successful uploading to a devpi instance to be indicative of whether PyPI will accept or reject on the grounds of, e.g., invalid trove classifiers (this one in particular has been the most common preventable but otherwise untestable upload failure our community encounters).
Even if they don't currently, that's likely an easier problem to fix than working out how to intermingle devpi's "pre-release developer sandbox" user experience with PyPI's "authoritative release publishing platform" user experience in one service, especially since the Legacy PyPI -> Warehouse migration is already encouraging several of those validation features to be extracted out to helper libraries. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 31 October 2017 at 23:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 October 2017 at 22:49, Jeremy Stanley <fungi@yuggoth.org> wrote:
Ideally we'd be recommending https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking to develop a robust pre-release artifact testing workflow.
While we mention it a couple of times on packaging.python.org [1,2], and include it in the packaging related Non-PyPA Projects list [3], we don't really emphasise that it makes a much better platform for pre-release testing and private experimentation than PyPI itself does (see https://devpi.net/ for an example of a deployed instance with multiple distinct user namespaces).
Given Donald's comment about the current test PyPI not being
On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote: particularly
good at any of its roles, perhaps it would make sense to aim to replace it with a periodically purged DevPi instance? [...]
Do the two share enough common code for successful uploading to a devpi instance to be indicative of whether PyPI will accept or reject on the grounds of, e.g., invalid trove classifiers (this one in particular has been the most common preventable but otherwise untestable upload failure our community encounters).
Even if they don't currently, that's likely an easier problem to fix than working out how to intermingle devpi's "pre-release developer sandbox" user experience with PyPI's "authoritative release publishing platform" user experience in one service, especially since the Legacy PyPI -> Warehouse migration is already encouraging several of those validation features to be extracted out to helper libraries.
Taking the classifiers list as a specific example: while that's *currently* stored in the live PyPI database, there isn't anything in particular that says that that needs to be the authoritative source of the list - it could be redefined as a cache of a list that PyPI picks up from somewhere else. It could, for example, be moved into a static data file in the `packaging` project, and have updates be deployed to the live service by way of new releases of that project. Tools could then decide for themselves whether to depend on the static list in the packaging project, or query PyPI to get the list for whichever version of the library PyPI itself was using. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (6)
-
Donald Stufft -
Jeremy Stanley -
Mariatta Wijaya -
Nick Coghlan -
Thomas Kluyver -
Toshio Kuratomi