Mailman 3 November 2012 - Distutils-SIG

Re: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427
by a.cavallo＠cavallinux.eu 14 Nov '12

14 Nov '12

Mmmm, interesting point and worth a discussion about different roles (developer, system admin, final user etc.) having different needs. I believe distutils is used as tool primarily (setup.py bdist_rpm/msi to create installable objects, setup.py bdist_sdist to manage the source code etc.): this complicates the landscape. Some developer has expectation (wrong IMHO) to replace a whole lot of tools with it (downloaders/installers/package managers/etc.): this is an uphill struggle against already widely deployed systems (dpkg/yum/even windows has something!). I think Tarek did some work in the past and the result is visible in the "The Hitchhiker’s Guide to Packaging" but I've no idea where it went in the end ... only is left is a punch-in-the-eye page (http://www.python.org/doc) :D These days I'm stuck with the old KISS approach and I write a setup.py to create "packages" and a makefile to create doc, run tests etc. I'm fairly happy with that. Thanks On Wed 14/11/12 08:32, "Ronald Oussoren" ronaldoussoren(a)mac.com wrote: > > I'm not convinced that distutils is untestable. A major problem with > distutils is that its API is barely documented, which effectly makes > all of it public API (simular to asyncore). IIRC that's the main > reason why distutils is frozen right now: with a lot of changes to distutils > people started to complain that this could break there there distutils > scripts. > The lack of a specification also makes testing harder, as it is > unclear what should be tested. > > Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > >

1 0

Re: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427
by a.cavallo＠cavallinux.eu 14 Nov '12

14 Nov '12

+1 Distutils is good enough: it could be better but for what is required (essentially copying files and creating packages installers) is fine. The only corner case is an absolute pain in the neck is in the cross compile scenario. Currently I don't have *any* need for "auto" tools (setuptools and descendants). As an example I always advice to avoid like a plague anything that depends on setuptools... with all the due respect I think is the poor's developer attempt to play the sys admin game. Even virtual env is a poor work around on the python interpreter not being relocatable (as in a portable app fashion). thanks On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren(a)mac.com wrote: > > On 13 Nov, 2012, at 16:00, Daniel Holth <dholth@gmail > .com> wrote: > > > I want to remove distutils from the standard > library. > Why? Distutils may not be perfect, but is usable for basic packages. It > could even be enhanced to support these peps and be even more useable, > although patches for that would run into the self-imposed freeze of > distutils development. > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://mail.python.org/mailman/listinfo/distutils-sig > >

6 5

Re: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427
by Daniel Holth 13 Nov '12

13 Nov '12

I think Metadata 1.3 is done. Who would like to czar? On Oct 22, 2012 12:53 PM, "Daniel Holth" <dholth(a)gmail.com> wrote: > http://hg.python.org/peps/rev/50e8ea1a17a0 > > Based on the other "required" field's absence in the wild, only > "Metadata-Version", "Name", "Version", and "Summary" are required. > Hopefully a clearer explanation of 0, 1, or many occurrences of each > field. >

9 15

(How) do setuptools/distribute handle circular dependencies?
by Vinay Sajip 13 Nov '12

13 Nov '12

Working on distlib, I noticed that PyPI contains projects with circular dependencies. Here's an extract from setup.py in the latest Zope2 (2.13.19): install_requires=[ ... 'Products.OFSP >= 2.13.2', ... ] + additional_install_requires, whereas the the setup.py in the latest Products.OFSP (2.13.2) has: install_requires=[ ... 'Zope2 >= 2.13.0a1', ], So, according to the declarations, each package depends on the other. Can setuptools / distribute deal with this sort of situation? If so, how does that work? Regards, Vinay Sajip

7 24

Multiple packages in a single distribution
by Alex Clark 13 Nov '12

13 Nov '12

Hi I am performing a "fun" experiment with Plone in which I plan to include a variety of namespace packages in a single distribution. This works great with non-namespaced packages. Or rather I should say: if you combine a series of name spaced packages into a flat namespace, it works. E.g. the Products and plone packages[1]: - Products/foo - Products/bar - plone/foo - plone/bar However, this completely falls down (at the distutils level, I guess) when you attempt to do the same thing with namespace packages e.g.: - plone.foo/plone/foo - plone.bar/plone/bar In distutils parlance, it looks like this (this works): packages=[ 'Products', 'plone', ], package_dir={ 'Products': 'Products', 'plone': 'plone', '': 'plone.app.content', }, namespace_packages=[ 'Products', 'plone', ], What I'd love to be able to do is to continue to add items to package_dir e.g.: package_dir={ 'Products': 'Products', 'plone': 'plone', '': 'plone.app.content', '': 'plone.app.foo', '': 'plone.app.bar', }, Because this would allow me to use git submodules. But, I'm guessing, since distutils never supported namespace packages (it's a setuptools feature, right?) then this will never work as I'd like it to. Aside from actually combining all the namespace packages, is there any other approach? Thanks for any thoughts, Alex [1] https://github.com/aclark4life/plock -- Alex Clark · https://www.gittip.com/aclark4life/

1 0

Improved dependency resolution in distlib
by Vinay Sajip 13 Nov '12

13 Nov '12

Following some changes to distlib, there have been improvements in how it can resolve dependencies. I ran a test on all source archives reachable through PyPI, and initial results seem to show that out of over 25,000 projects and 112,000 source archives, we can extract dependency metadata from all but 2,540 archives relating to 916 projects. The distlib locator code has been updated to access this metadata, and this makes it possible to resolve dependency graphs reasonably quickly, without downloading any archives. The dependency finder command-line script is at https://gist.github.com/3919026 The script, finddeps.py, when given a distribution name, will attempt to locate it and its dependencies, identify requirements which couldn't be satisfied, show a topological sort (advisory, given the existence of circular dependencies on many PyPI projects) and a download ordering. Example results are given in the same Gist for: Flask (3 dists in all, resolved in ~1 second) apycotbot (22 dists in all, resolved in ~5 seconds) collective.megaphone (242 dists in all, resolved in ~55 seconds) You can also specify constraints, e.g. $ python finddeps.py "pyramid (> 1.0, < 1.3)" I also included a separate script for testing the case described in https://github.com/pypa/pip/issues/174 as mentioned by Carl. In this case, we get the expected result, with the dependency tree looking like this: A 1.0 B 1.0 [B] D 0.9 [D (<= 0.9)] C 1.0 [C] D 0.9 [D (<= 1.1)] D 0.9 C 1.0 D 0.9 [D (<= 1.1)] B 1.0 D 0.9 [D (<= 0.9)] If anyone has the time to try out distlib, I'd appreciate some feedback on the dependency finder (or anything else for that matter) and any unexpected results you get in your trials. You should be able to run the dependency finder on most projects on PyPI, but not ones which don't have a reachable download (for example, "Goose". But there, "pip install Goose" fails, too). Thanks and regards, Vinay Sajip

1 0

Re: [Distutils] [Python-Dev] [Catalog-sig] accept the wheel PEPs425, 426, 427
by a.cavallo＠cavallinux.eu 13 Nov '12

13 Nov '12

I'll give you that number(?) but ... mercurial, docutils, jinjia2 pygments, sphinx, lxml, nose, cherrypy, django, pyqt ... all they don't need/use setuptools: that 25% left is quite an interesting field to play in. If setuptools was "significant packaging innovation" do you think people wouldn't have embraced already? Allow me to call *that* a nonsense. thanks ps. my experience is on the field, so please give me the credit of many years of experience if I'm say I'm not that keen on "auto" tools: in this kiss rules. On Tue 13/11/12 17:35, "Daniel Holth" dholth(a)gmail.com wrote: > Setuptools! You would avoid 75% of pypi. It is nonsense to pretend > that setuptools is not a significant packaging innovation. Its main > flaw is that it is based on distutils, a non-extensible design. > distutils2 is a lot of setuptools and distutils code with the > plug-ability taken out. > > Perhaps I should say that I would like distutils to become as > relevant to packaging as the cgi module is to web development. It is > not a short-term goal. > >

1 0

Changing the separator from - to ~ and allow all Unicode alphanumerics in package names...
by Daniel Holth 12 Nov '12

12 Nov '12

Although I think the ~ is a very ugly -, it could be useful to change the separator to something less commonly used than the -. It would be useful to be able to use the hyphen - in the version of a package (for semver) and elsewhere. Using it as the separator could make parsing the file name a bit trickier than is healthy. This change would affect PEP 376 which reads: This distinct directory is named as follows:: name + '-' + version + '.dist-info' python_package-1.0.0_four+seven.dist-info with today's hyphen/underscore folding: re.sub('[^A-Za-z0-9.]+', '-', version), could become python-package~1.0.0-four+seven.dist-info It would also affect pip, setuptools, and the wheel peps. If we do this, I would like to allow Unicode package names at the same time. safe_name(), the pkg_resources function that escapes package names for file names, would become re.sub(u"[^\w.]+", "_", u"package-name", flags=re.U) In other words, the rule for package names would be that they can contain any Unicode alphanumeric or _ or dot. Right now package names cannot practically contain non-ASCII because the setuptools installation will fold it all to _ and installation metadata will collide on the disk. safe_version(), presently the same as safe_name() would also need to allow + for semver. Does anyone have the energy to actually implement a proof-of-concept?

3 5

accept the wheel PEPs 425, 426, 427
by Daniel Holth 10 Nov '12

10 Nov '12

I'd like to submit the Wheel PEPs 425 (filename metadata), 426 (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has been stable since May and we are preparing a patch to support it in pip, but we need to earn consensus before including it in the most widely used installer. Wheel is a binary packaging format that is mostly based on the phenomenal work done by 'packaging' as PEP 376 et al. The resulting packages solve packaging problems by being installable without setup.py or any variation of distutils, lxml can be installed in 0.7 seconds, and a simple installer is just "unzip" inside site-packages. Wheel installers know about the major version number of the spec itself, and will refuse to install version 2.0 wheels if they do not understand them, so the format can be changed down the line. Let me know what I need to do to get it accepted, if anything needs to be added or revised, or, preferably, that it is awesome and you want to use it ASAP. Thanks, Daniel Holth

8 24

MANIFEST.in properly defined but something wrong in setup.py
by Ben tley 09 Nov '12

09 Nov '12

Dear Distutils-sig members, I'm having some trouble packaging a python project. I have read several guides but I still seem to do something wrong. Could somebody take a look to see what I am doing wrong? Packaging details project:http://stackoverflow.com/questions/13307408/python-packaging-data-f… I think the content of my MANIFEST.in is correct and I am probably doing something wrong in my setup.py file. You can find a list in that link of the configurations of setup.py I have already tried but lead to no success. Best regards

1 0