Mailman 3 March 2015 - Distutils-SIG

Building deb packages for pypy
by Stuart Longland 20 Mar '15

20 Mar '15

Hi all, I'm currently attempting to evaluate pypy for use in a few performance-critical projects. We mainly use Debian or Ubuntu as the host platform, with our packages provided as debs. Traditionally if we needed to build debs for third-party libraries, we'd use `stdeb` to generate the source files. However I'm having a lot of fun and games trying to figure out how this is done with pypy. What is the procedure for building a deb package of a Python library using stdeb for pypy? -- _ ___ Stuart Longland - Systems Engineer \ /|_) | T: +61 7 3535 9619 \/ | \ | 38b Douglas Street F: +61 7 3535 9699 SYSTEMS Milton QLD 4064 http://www.vrt.com.au

2 2

Re: [Distutils] setup_requires for dev environments
by Tim Smith 19 Mar '15

19 Mar '15

On Thu, Mar 19, 2015 at 7:12 AM, Daniel Holth <dholth(a)gmail.com> wrote: > So I'd like to suggest the following series of small improvements to both > pip and setuptools: > > * setuptools: `python setup.py setup_requires` dumps its setup_requires > keyword in 'requirements.txt' format > > It's is already in this format, so should be trivial, but allows one to do > something like: > > $ python setup.py setup_requires > setup_requires.txt > $ pip install -r setup_requires.txt > > Or in one bash line: > > $ pip install -r <( python setup.py setup_requires ) > A way of learning about setup_requires dependencies would be helpful for homebrew-pypi-poet [1], which helps generate Homebrew formulae for applications implemented in Python. Homebrew prefers to specify all dependencies explicitly in the formula rather than allowing easy_install or pip to perform dependency resolution at formula install time. [2] We use installed metadata to learn about install_requires dependencies but setup_requires dependencies aren't captured AFAIK (are they?) so formula authors have to identify those by hand. This isn't critically painful for me but I thought it was an interesting example use case. Tim [1] https://github.com/tdsmith/homebrew-pypi-poet [2] As Chris Barker notes, --single-version-externally-managed is a good way to get setuptools-based setup.py's to just install the package; --single-version-externally-managed hands the install process over to distutils. Of course, distutils setup.py's do not support the --single-version-externally-managed option. To have a consistent CLI interface, Homebrew borrows a shim from pip's source to make sure we always call setuptools.setup() when we run setup.py so that we can hand the install back to distutils: https://github.com/Homebrew/homebrew/blob/master/Library/Homebrew/language/… -- thanks to Donald for pointing to the right place in the pip code.

2 2

JSONP: Deprecation and Intent to Remove
by Donald Stufft 19 Mar '15

19 Mar '15

For awhile now PyPI has supported JSONP on the /pypi/*/json API to allow people to access the JSON data in a cross origin request. JSONP is problematic psuedo standard which has niggly edge cases which make it hard to fully secure. Browsers have a much better standard through CORS to handle this use case. As of now this endpoint has CORS enabled on it and any new or existing consumers of this API should switch to using CORS instead of JSONP. Warehouse will not be implementing the JSONP endpoint so when we switch PyPI to the Warehouse code base anything still relying on JSONP will break. Thanks! --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

3 4

Finalizing PEP 440
by Donald Stufft 19 Mar '15

19 Mar '15

It's been ~3 months or so since PEP 440 support was released in pip and setuptools. I think at this point we've resolved any major issues with the PEP 440 spec that can be resolved without a whole new PEP. In addition it's now been out long enough that people have adjusted processes/tooling to account for it and changing things at this stage is likely going to be more disruptive than just leaving it as it is. Given that, I suggest that we mark the PEP as no longer provisional. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

2 1

API CHANGE - Migrating from MD5 to SHA2, Take 2
by Donald Stufft 08 Mar '15

08 Mar '15

Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256). Simple API ---------- Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash. Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist. JSON / XMLRPC API ----------------- Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification. Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future. Web UI ------ Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine. Other Clients ------------- - pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though. List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

4 10

Installing a file into sitepackages
by Stuart Axon 08 Mar '15

08 Mar '15

Hi, In my project, I install a .pth file into site-packages, I use the data_files... in Ubuntu this seems to work OK, but in a Windows VM the file doesn't seem to be being installed: setup( .... # Install the import hook data_files=[ (site_packages_path, ["vext_importer.pth"] if environ.get('VIRTUAL_ENV') else []), ], ) - Is there a better way to do this ? I realise it's a bit odd installing a .pth - my project is to allow certain packages to use the system site packages from a virtualenv - https://github.com/stuaxo/vext S++

2 2

Granting permissions to Xavier Fernandez and Marc Abramowitz?
by piotr.dobrogost＠autoera-serwer.home.pl piotr.dobrogost＠autoera-serwer.home.pl 07 Mar '15

07 Mar '15

Hi As an external observer of pip project at github I see two men, namely Xavier Fernandez (https://github.com/xavfernandez) and Marc Abramowitz (https://github.com/msabramo) with many valuable contributions. I think it would be beneficial if they had been granted some more permissions to the project/repo. Regards, Piotr Dobrogost

3 2

Getting more momentum for pip
by Marc Abramowitz 07 Mar '15

07 Mar '15

This post is meant to start a discussion on how to make sure that important PyPA projects like pip get enough eyes so that they can continually move forward. It is absolutely NOT meant to be critical of folks who volunteer their time to work on these projects, as I have the utmost respect for folks who do this often thankless work. And since the word "thankless" just popped into my head, let me say "thank you" right now to the folks who contribute to PyPA. I've noticed that when PRs are submitted to pip (by myself and by others), they often languish for a while and then sometimes become unmergeable because of conflicts. This makes me think that the folks who review the PRs are overburdened and/or the process needs a bit more structure (e.g.: each person thinks someone else is going to review the PR and so no one does it). So some ideas off the top of my head - please comment and/or add your own suggestions to the list: - Add more committers - this will ostensibly increase the number of folks reviewing so that the turnaround time is decreased. And takes some pressure off. - Obtain some kind of funding so that committers can be compensated for their work and don't feel as bad about spending time on it. These people have day jobs and families so maybe this would help; maybe not. - Introduce some bot or automation that periodically reminds about open PRs that are getting old or PRs that have become unmergeable, etc. Or maybe for each PR, it picks one or more persons who is responsible for reviewing that PR, so there is no ambiguity about who is responsible. The OpenStack folks have quite a bit of structure in their workflow (probably too much for PyPA projects which have less people working on them?), but perhaps there are some things that can be borrowed. In particular, changes need a certain amount of upvotes to be merged and reviewers usually request feedback from certain individuals. So I guess my suggestions boil down to: - Add more humans - Add more money to make humans more efficient - Add more computer automation #3 seems most appealing to me, but of course it requires humans to develop it in the first place, but at least it's an investment that could pay dividends. Thoughts? Marc

12 51

Granting permissions to Xavier Fernandez and Marc Abramowitz?
by Piotr Dobrogost 06 Mar '15

06 Mar '15

Hi As an external observer of pip project at github I see two men, namely Xavier Fernandez (https://github.com/xavfernandez) and Marc Abramowitz (https://github.com/msabramo) with many valuable contributions. I think it would be beneficial if they had been granted some more permissions to the project/repo. Regards, Piotr Dobrogost

4 3

[issue160] bootstrap of virtualenv.py fails on setuptools 12.4
by Hannes Doyle 05 Mar '15

05 Mar '15

New submission from Hannes Doyle: Hi, We are running virtualenv 1.7.2 with python 2.7.3 when executing the following mkdir pyenv virtualenv.py -p /<python path>/bin/python2.7 pyenv I get the errors shown in the err.txt But if I do the following it works fine: mkdir pyenv wget 'https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.e…' wget 'https://pypi.python.org/packages/source/p/pip/pip-6.0.8.tar.gz#md5=2332e6f9…' virtualenv.py --never-download -p /<python path>/bin/python2.7 pyenv And since this started to show at 2015-03-04 17:00 CET I suspect that it's related to the release of 12.4 that were released on the same day. I'm not sure if it might be here or in the virtualenv forum this should be posted though. Regards Hannes ---------- files: err.txt messages: 748 nosy: marulkan priority: bug status: unread title: bootstrap of virtualenv.py fails on setuptools 12.4 Added file: http://bugs.python.org/setuptools/file173/err.txt _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue160> _______________________________________________

1 0