Mailman 3 February 2014 - Distutils-SIG

Can't find "Handling the binary dependency management problem" thread using Google Groups interface
by Piotr Dobrogost 20 Feb '14

20 Feb '14

Hi! I found thread titled "Handling the binary dependency management problem" at https://mail.python.org/pipermail/distutils-sig/2013-December/023145.html and was trying to find it using Google Groups interface using this search https://groups.google.com/forum/#!searchin/distutils-sig/Handling$20the$20b… but with no luck. Does someone know why I can't find this thread on Google Groups? Best regards, Piotr Dobrogost

1 0

Removal of PyPI DNS mirroring infrastructure per PEP 449
by Ernest W. Durbin III 15 Feb '14

15 Feb '14

This morning the infrastructure staff executed on the directives in PEP 449 regarding the removal of DNS entries for the PyPI mirroring infrastructure. Reference: http://www.python.org/dev/peps/pep-0449/ The following DNS records were removed: - a.pypi.python.org - b.pypi.python.org - c.pypi.python.org - d.pypi.python.org - e.pypi.python.org - f.pypi.python.org - g.pypi.python.org - last.pypi.python.org The copy on the PyPI website was updated to reflect these changes https://pypi.python.org/mirrors Additionally the redirection infrastructure for these hostnames was removed from our HAproxy configuration at OSUOSL. All requests for cheeseshop.python.org will continute to redirect to https://pypi.python.org with their appropriate HTTP query. - Ernest W. Durbin III

1 0

mirror update bandwidth?
by Robert Collins 14 Feb '14

14 Feb '14

Just curious - whats the typical monthly bandwidth taken up pulling packages (say every 4 hours) from pypi with bandersnatch? -Rob -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

1 0

Setuptools 3.0b1 available for preview
by Jason R. Coombs 13 Feb '14

13 Feb '14

I'm pleased to announce that Setuptools 3.0b1 is now available for download from the project downloads page: https://bitbucket.org/pypa/setuptools/downloads This backward-incompatible release contains the changes detailed in the CHANGES.txt file: https://bitbucket.org/pypa/setuptools/src/3.0b1/CHANGES.txt?at=default Please give this beta release a try or review the changes for any impact this might have on your package. I anticipate most users will find these changes of no consequence, but if there are any issues that might affect your package that don't have an obvious solution, please file a ticket. Regards, Jason

4 4

PyPI Status Update - PostgreSQL Tuning and CDN Shielding
by Donald Stufft 12 Feb '14

12 Feb '14

Just a quick update on some recent PyPI changes. We’ve attempted to tune PostgreSQL. This has resulted in a particular problem query going from a 23s execution time to a 5s execution time. This should reduce or remove the 503 errors that occur when the mirrors slam PyPI at midnight UTC looking for new packages. This has reduced the bulk of the problem with the mirrors. Assuming stability remains the same as it is now we will not be implementing rate limiting at this time as we would greatly prefer not to have to do that. We’ve also activated CDN Shielding for all non POST requests. This means that whenever you contact a node in say the singapore POP if that node does not have the response cached instead of directly contacting the backends in Rackspace's IAD datacenter it will instead contact the Ashburn POP. This POP will serve a response that it has cached, or contact the backend if it does not have a cached response. This should further speed up PyPI as it's far more likely that the shield POP (Ashburn) will have the response cached even if the edge location does not. Further more the connectivity between the Fastly POPs are more likely to be lower latency then the connectivity between any random Fastly POP and the backends. So essentially PostgreSQL should be faster and our caching CDN should be more efficient. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

Metadata debate for ages
by anatoly techtonik 11 Feb '14

11 Feb '14

On Tue, Feb 11, 2014 at 4:33 PM, Paul Moore <notifications(a)github.com>wrote: > and I doubt that will happen, as the metadata debate has been ongoing for > ages, and it's *very* late to be coming up with requirements with > significant implications out of the blue like this > This is from thread with my rant about versioning, which set a dumb question in my head. What is the role of dot zero part in metadata versioning? I mean if there is so much debates, how about choosing a minimal first thing that can be released? Reach consensus, release. Increase version. Go another round of debate. Release. Test. Increment version. Release. It seems that people are trying to make it perfect loosing the real conflicting points in a ton of information. I am looking at 50+ pages PEP in http://www.python.org/dev/peps/pep-0426/ and the first thing I notice that I don't understand why there are provisions like MUST and why PEP named "Metadata" regulates what tools should or should not do and what tools should exists at all. To me, the "Metadata 2.0" PEP should specify only two things: 1. format(s) of the data, how it can be represented so that people can process it 2. structure of the data - a set of fields, their values and cases when they are applied That's it. The things like "Automated tools, especially public index servers, MAY impose additional length restrictions on metadata beyond those enumerated in this PEP. Such limits SHOULD be imposed where necessary to protect the integrity of a service, based on the available resources and the service provider's judgment of reasonable metadata capacity requirements." are out of scope of Metadata at all. This stuff belongs elsewhere. If there will be a problem that metadata grows too large - it should be recorded as an issue, and the next version of metadata should say - fixed issues #..., #..., ... or just include a separate PEP for those who don't know how to handle the load on their servers. What do you think? -- anatoly t.

1 0

.. *explicitly* mark the package as pre-release
by anatoly techtonik 10 Feb '14

10 Feb '14

hi, please cc i want to release some versions to pypi, but hide them from installing by explicitly marking as pre-release. i want to preserve versions numbering as 0.1, 0.2 according to semver.org point 4. is that possible? -- anatoly t.

4 5

PyPI Rate Limiting
by Ernest W. Durbin III 10 Feb '14

10 Feb '14

Since the launch of the new infrastructure for PyPI two weeks ago, I’ve been monitoring overall performance and reliability of PyPI for browsers, uploads, installers, and mirrors. Overall I am very happy, but have noticed an ongoing issue with latency spikes and 5xx errors. I believe these issues are not new, but we don’t have any of the logs or monitoring that came along with the new infrastructure. The cause of these issues is very apparently mirroring clients hitting PyPI with floods of requests at common cron intervals. Additionally, new mirrors coming online and performing their initial sync can easily cause extended periods of increased latency and errors for all users, especially if the number of workers configured to perform the sync is turned up. At 2014-02-07 at about 00:00 UTC PyPI was effectively DoS’d for 45 minutes while a major research lab performing a sync via bandersnatch. It appears their worker count may have been configured as high as 50. The design of PEP 381 mirroring clients requires calls to the PyPI XMLRPC to obtain changelogs and package serial numbers. As such, when clients are configured for high parallelism our backends can be quickly overwhelmed. In order to maintain quality of service for all clients, we will begin rate limiting requests to the following routes: - /pypi - /mirrors - /id - /oauth - /security The initial rates will be limited to 5 req/s per IP with bursts of 10 requests allowed. Client requests up to the burst limit will be delayed to maintain a 5 req/s maximum. Any requests past the 10 request burst will receive an HTTP 429 response code per RFC 6585. Tuning these parameters will be painless, so if issues arise with mirroring clients we will be very responsive to necessary modifications. Note that the routes used by installation clients (`/packages` and `/simple`) will remain unaffected as they are generally served from the CDN, and do not have as high of an overhead in our backend processes. This rate-limiting is to be considered an interim solution, as I plan to begin a discussion on some updates to mirroring infrastructure guidelines.

6 8

guidance for contributing to packaging docs
by Alex Burke 09 Feb '14

09 Feb '14

Hi all, I've been a regular reader of the packaging mailing list for a little while now and, though I don't yet feel up to working on code, wanted to see if I could help with documentation as both an exercise in my understanding as well as being able to help others (I struggled a little mid to late last year trying to figure out best practices for packaging Python code). My question might seem a little broad, but where would it be most useful to focus documentation contributions? For example I know the python packaging guide is being fleshed out, but it's seemed somewhat in a state of flux and keeping up with reorganisations has been a little tricky. Any pointers or specific suggestions would be really useful. Perhaps it would it be best to wait until Python 3.4 ships with ensurepip given how much advicethat might affect? Incidentally, thought I'd mention it is a very welcome surprise to those I have spoken to about Python that it will soon have an integrated and blessed package managing solution that ships with default builds. It seems to be a common point of confusion or even slight or against what is otherwise considered a fantastic language. Kudos to everyone involved. Thanks, Alex J Burke.

3 2

Re: [Distutils] Pypi Openid login problem
by Debjyoti Bhaumik 07 Feb '14

07 Feb '14

Hi! I am trying to login(register) to pypi with my google account. But I am getting back this error: OpenID response has been tampered with:NotAuthenticated('Replay attack detected', 9) I am using chrome on Linux. I tried to login from different pages, both this:https://pypi.python.org/pypi?:action=openid But the results are same.

1 0