Re: [Distutils] "Python Packaging User Guide"
Hi Marcus, I would very much like to get involved and help with this effort - in fact I was looking to package up some of my own work recently and stumbled on The Hitchhiker’s Guide to Packaging but could tell some of the advice was out of date having followed the discussions for a while now. What I was thinking is it could be useful my documenting exactly what I'm having to do, things to remember etc and it'd help me learn things much more deeply on the way too. My only concern however would be where to go to find the right 'new world' answers. Aside from reading the PEPs etc, are there any other points of contact? Is it ok to ask such questions on this list? Thanks, Alex J Burke.
On Tue, Jul 2, 2013 at 7:18 PM, Alex Burke <alexjeffburke@gmail.com> wrote:
Hi Marcus,
I would very much like to get involved and help with this effort - in fact I was looking to package up some of my own work recently and stumbled on The Hitchhiker’s Guide to Packaging but could tell some of the advice was out of date having followed the discussions for a while now.
What I was thinking is it could be useful my documenting exactly what I'm having to do, things to remember etc and it'd help me learn things much more deeply on the way too. My only concern however would be where to go to find the right 'new world' answers. Aside from reading the PEPs etc, are there any other points of contact? Is it ok to ask such questions on this list?
Sounds like a great idea, assemble your FAQ, and we will remember what's confusing about packaging when you haven't been embroiled in it for years.
On 3 July 2013 09:18, Alex Burke <alexjeffburke@gmail.com> wrote:
Hi Marcus,
I would very much like to get involved and help with this effort - in fact I was looking to package up some of my own work recently and stumbled on The Hitchhiker’s Guide to Packaging but could tell some of the advice was out of date having followed the discussions for a while now.
What I was thinking is it could be useful my documenting exactly what I'm having to do, things to remember etc and it'd help me learn things much more deeply on the way too. My only concern however would be where to go to find the right 'new world' answers. Aside from reading the PEPs etc, are there any other points of contact? Is it ok to ask such questions on this list?
Asking here, or filing tracker issues on https://bitbucket.org/pypa/python-packaging-user-guide would be good. Unless things have changed in the past couple of weeks, the aim is for people starting from scratch now to grab setuptools 0.7 (or later) and pip 1.4 (or later & once it is released) I'm not sure on the specific plan for wheel uploads to PyPI myself - I *believe* we're going to recommend using pip to retrieve http://wheel.readthedocs.org/en/latest/ to obtain the bdist_wheel command. There's also the problem of hosting for the setuptools & pip bootstrap scripts. I believe the plan is to host it at a canonical URL on pypi.python.org, but I'm not sure of the current status of that. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hello and sorry if I am on the wrong mailing list. I'm working on a dependency resolution resolver and wanted to implement support for the new Metadata format. (As of now i'm parsing the setup.py for dependency information and am dependent on a self build metadata database of all PyPI packages) Will there be build-in Metadata in the new PEP 426 format online on PyPI for all packages? If yes, are there already specifications on how they will be retreavable? Thanks, Alexander Schneider
On 3 July 2013 19:17, Alexander Schneider < Alexander.Schneider@uni-duesseldorf.de> wrote:
Hello and sorry if I am on the wrong mailing list.
I'm working on a dependency resolution resolver and wanted to implement support for the new Metadata format. (As of now i'm parsing the setup.py for dependency information and am dependent on a self build metadata database of all PyPI packages)
Will there be build-in Metadata in the new PEP 426 format online on PyPI for all packages? If yes, are there already specifications on how they will be retreavable?
Yes there will, but actually figuring out the details of those APIs is some time away. Note that the version currently referenced from the PEP is a little out of date ( http://mail.python.org/pipermail/distutils-sig/2013-June/021357.html). I will hopefully get it updated at the PyCon AU sprints next week. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Alexander Schneider <Alexander.Schneider <at> uni-duesseldorf.de> writes:
Will there be build-in Metadata in the new PEP 426 format online on PyPI for all packages? If yes, are there already specifications on how they will be retreavable?
I have experimental support for PEP 426 metadata available, which is up-to-date with the spec apart from the changes Nick linked to. For any given package, you can access some JSON at an URL based on the project name. For example, setuptools 0.7.5 metadata is available at http://www.red-dove.com/pypi/projects/S/setuptools/package-0.7.5.json If you deserialize the JSON at an URL like the above into a dict, the PEP 426 metadata is available in the subdict at key "index-metadata" in the top-level dict. Example from setuptools 0.7.5: "index-metadata": { "description": "omitted for brevity", "license": "PSF or ZPL", "contacts": [ { "role": "author", "name": "The fellowship of the packaging", "email": "distutils-sig@python.org" } ], "summary": "Easily download, build, install, upgrade, and uninstall Python packages", "project_urls": { "Home": "https://pypi.python.org/pypi/setuptools" }, "keywords": [ "CPAN", "PyPI", "distutils", "eggs", "package", "management" ], "metadata_version": "2.0", "extras": [ "certs", "ssl" ], "version": "0.7.5", "run_may_require": [ { "environment": "sys_platform=='win32'", "dependencies": [ "wincertstore (== 0.1)" ], "extra": "ssl" }, { "environment": "sys_platform=='win32' and python_version=='2.4'", "dependencies": [ "ctypes (== 1.0.2)" ], "extra": "ssl" }, { "dependencies": [ "certifi (== 0.0.8)" ], "extra": "certs" }, { "environment": "python_version in '2.4, 2.5'", "dependencies": [ "ssl (== 1.16)" ], "extra": "ssl" } ], "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Python Software Foundation License", "License :: OSI Approved :: Zope Public License", "Operating System :: OS Independent", "Programming Language :: Python :: 2.4", "Programming Language :: Python :: 2.5", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.1", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Archiving :: Packaging", "Topic :: System :: Systems Administration", "Topic :: Utilities" ], "name": "setuptools" }, I expect this metadata to track the PEP as changes to it are published. Currently, the top-level dict contains some legacy representations of the metadata which will be removed in due course. I would hope that once the dust settles on the PEP, this metadata (the PEP 426 part) can be migrated to PyPI. Regards, Vinay Sajip
On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If you deserialize the JSON at an URL like the above into a dict, the PEP 426 metadata is available in the subdict at key "index-metadata" in the top-level dict. Example from setuptools 0.7.5:
"index-metadata": { .... "name": "setuptools" },
I expect this metadata to track the PEP as changes to it are published. Currently, the top-level dict contains some legacy representations of the metadata which will be removed in due course.
Just an FYI, not sure if this is an issue with your converter or with the new spec, but the metadata shown for setuptools is missing something important: 0.7.x pins specific distributions of its dependencies using dependency_links URLs with #md5 hashes, so that SSL support can be installed in a reasonably secure manner, as long as you're starting from a trusted copy of the distribution. The converted metadata you show lacks this pinning. Granted, the pinning is somewhat kludged, and the specific need is perhaps a rare use case outside of installer tools themselves. But I thought it worth pointing out as a limitation of either the converter or with the spec itself in relation to version support.
On Jul 3, 2013, at 2:19 PM, PJ Eby <pje@telecommunity.com> wrote:
On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If you deserialize the JSON at an URL like the above into a dict, the PEP 426 metadata is available in the subdict at key "index-metadata" in the top-level dict. Example from setuptools 0.7.5:
"index-metadata": { .... "name": "setuptools" },
I expect this metadata to track the PEP as changes to it are published. Currently, the top-level dict contains some legacy representations of the metadata which will be removed in due course.
Just an FYI, not sure if this is an issue with your converter or with the new spec, but the metadata shown for setuptools is missing something important: 0.7.x pins specific distributions of its dependencies using dependency_links URLs with #md5 hashes, so that SSL support can be installed in a reasonably secure manner, as long as you're starting from a trusted copy of the distribution. The converted metadata you show lacks this pinning.
Granted, the pinning is somewhat kludged, and the specific need is perhaps a rare use case outside of installer tools themselves. But I thought it worth pointing out as a limitation of either the converter or with the spec itself in relation to version support. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
PEP426 does not support dependency_links. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 3, 2013, at 7:38 PM, PJ Eby <pje@telecommunity.com> wrote:
On Wed, Jul 3, 2013 at 2:34 PM, Donald Stufft <donald@stufft.io> wrote:
PEP426 does not support dependency_links.
Right - I would've expected direct references in this scenario, assuming the PEP still has them.
Yea PEP440 direct references can be used as an approximate dependency_links replacement with the caveat you can't publish them on PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
PJ Eby <pje <at> telecommunity.com> writes:
Just an FYI, not sure if this is an issue with your converter or with the new spec, but the metadata shown for setuptools is missing something important: 0.7.x pins specific distributions of its dependencies using dependency_links URLs with #md5 hashes, so that SSL support can be installed in a reasonably secure manner, as long as you're starting from a trusted copy of the distribution. The converted metadata you show lacks this pinning.
True, although I do capture the dependency links under the 'dependency-urls' key of the top level dict of the JSON I linked to. While dependency_links is not directly supported by PEP 426, the intent is there via "direct references". When installing using distlib/distil, SSL host verification and hash verification are done, even when direct references are not specified, since the versions of dependencies are pinned. For example, if I install setuptools into a fresh venv: $ pyvenv-3.3 /tmp/venv $ distil -e /tmp/venv install "setuptools [ssl,certs]" Checking requirements for setuptools (0.7.7) ... done. The following new packages will be downloaded and installed: certifi (0.0.8) [for setuptools] setuptools (0.7.7) Proceed? (y/n) y Downloading certifi-0.0.8.tar.gz to /tmp/tmpccek0f [for setuptools] 115KB @ 667 KB/s 100 % Done: 00:00:00 Unpacking ... done. Downloading setuptools-0.7.7.tar.gz to /tmp/tmpchxc1x 736KB @ 393 KB/s 100 % Done: 00:00:01 Unpacking ... done. [installation feedback snipped] Below is an extract from distil.log for the above installation, showing the downloading and verification operations: Downloading certifi-0.0.8.tar.gz to /tmp/tmpccek0f [for setuptools] Digest specified: dc5f5e7f0b5fc08d27654b17daa6ecec Host verified: pypi.python.org Digest verified: dc5f5e7f0b5fc08d27654b17daa6ecec Library location: venv site-packages Downloading setuptools-0.7.7.tar.gz to /tmp/tmpchxc1x Digest specified: 0d7bc0e1a34b70a97e706ef74aa7f37f Host verified: pypi.python.org Digest verified: 0d7bc0e1a34b70a97e706ef74aa7f37f Library location: venv site-packages Distil includes the Mozilla certs and thus is able to do SSL host validation. The hash support is currently limited to MD5 because PyPI has not supported other formats, but I expect that will be rectified in due course. Regards, Vinay Sajip
participants (7)
-
Alex Burke -
Alexander Schneider -
Daniel Holth -
Donald Stufft -
Nick Coghlan -
PJ Eby -
Vinay Sajip