Tarek Ziadé wrote:

= stability and high availability =

we went in two directions to improve PyPI :

1/ add the mirroring protocol 2/ make the PyPI server more reliable by pushing its storage in a redundant cloud. ... == better infra ==

I think the project is staled right now.

True, I don't have the needed cycles to make progress and neither do the team members. We do have the Amazon infrastructure setup (EC2, CloudFront, S3), and some progress has been made to sync the S3 storage with the local copies of the packages. The next steps would be to implement a trigger based sync scheme for packages and the simple/ index, i.e. new uploads should trigger an S3 copy process and update the simple/ index on S3 as well. For more info, please have a look at the wiki page: http://wiki.python.org/moin/CloudPyPI Here's the original proposal: http://wiki.python.org/moin/CloudPyPI/Proposal -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 27 2011)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2011-10-04: PyCon DE 2011, Leipzig, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, Sep 27, 2011 at 12:07 PM, Lennart Regebro wrote:

On Tue, Sep 27, 2011 at 11:40, Tarek Ziadé wrote:

...

1/ stability and high availability

How are opinions on setting up country-specific PyPI mirrors? The lag to the US is pretty severe in Poland, and I suspect my buildouts would benefit from having a server in Poland. Now, of course, it could be called x.pypi.python.org, but maybe we should have aliases such as pl.pypi.python.org as well?

I have no strong opinion on the issue, what do others think?

I think it's a good idea to have the closest mirror,. One long-term goal I had was to add a client-side geloc code that would prefer the closest mirror. IOW, with your IP and the mirrors IP, pick the closest IP

//Lennart

-- Tarek Ziadé | http://ziade.org

2011/9/27 Jim Fulton :

On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro wrote:

...

On Tue, Sep 27, 2011 at 11:40, Tarek Ziadé wrote:

...

1/ stability and high availability

How are opinions on setting up country-specific PyPI mirrors? The lag to the US is pretty severe in Poland, and I suspect my buildouts would benefit from having a server in Poland. Now, of course, it could be called x.pypi.python.org, but maybe we should have aliases such as pl.pypi.python.org as well?

I have no strong opinion on the issue, what do others think?

Wouldn't CloudFront make this moot?

If we state that PyPI completely depends on Amazon I guess yes. But imho, it's saner for the long term to have a community-driven protocol for mirroring so we don't rely on third-party vendors.

I'm not clear I follow the plans, but it appears that CloudFront would provide greater availability with little or no changes to clients.

Jim

-- Jim Fulton http://www.linkedin.com/in/jimfulton

-- Tarek Ziadé | http://ziade.org

On 09/27/2011 02:27 PM, Jim Fulton wrote:

...

2011/9/27 Jim Fulton:

...

On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro wrote:

...

...

1/ stability and high availability How are opinions on setting up country-specific PyPI mirrors? The lag to the US is pretty severe in Poland, and I suspect my buildouts would benefit from having a server in Poland. Now, of course, it could be called x.pypi.python.org, but maybe we should have aliases such as

On Tue, Sep 27, 2011 at 11:40, Tarek Ziadé wrote: pl.pypi.python.org as well?

I have no strong opinion on the issue, what do others think? Wouldn't CloudFront make this moot? If we state that PyPI completely depends on Amazon I guess yes.

But imho, it's saner for the long term to have a community-driven protocol for mirroring so we don't rely on third-party vendors. I understand where you're coming from but, .. I think it's saner to rely on proven technology

On Tue, Sep 27, 2011 at 7:58 AM, Tarek Ziadé wrote: than to invent our own protocol. NIH?

This also feels like a problem that has already been solved in various ways by Debian, RedHat, CPAN and others. Wichert.

On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman wrote: ..

...

I understand where you're coming from but, .. I think it's saner to rely on proven technology than to invent our own protocol. NIH?

This also feels like a problem that has already been solved in various ways by Debian, RedHat, CPAN and others.

Yes, and we've found a way similar to CPAN, with some Python specifics (PyPI download statistics mainly) Oh my, we're cycling again. Nothing personal to you or Jim, but I have a sudden fatigue on packaging because it seems like people are ignoring what's being done to complain afterwards about us suffering of some kind of NIH :) If you're seeing anything you don't like in PEP 381 (accepted a while ago), go ahead and propose some improvements. But please keep in mind that we've looked at other systems before we wrote that PEP. Cheers Tarek -- Tarek Ziadé | http://ziade.org

On Tue, Sep 27, 2011 at 3:00 PM, Wichert Akkerman wrote:

On 09/27/2011 02:51 PM, Tarek Ziadé wrote:

...

On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman  wrote: ..

...

...

I understand where you're coming from but, .. I think it's saner to rely on proven technology than to invent our own protocol. NIH?

This also feels like a problem that has already been solved in various ways by Debian, RedHat, CPAN and others.

Yes, and we've found a way similar to CPAN, with some Python specifics (PyPI download statistics mainly)

Oh my, we're cycling again.

Nothing personal to you or Jim, but I have a sudden fatigue on packaging because it seems like people are ignoring what's being done to complain afterwards about us suffering of some kind of NIH :)

It's just that my perspective is that of a simple user. And from my perspective nothing has changed in the last couple of years. Pypi still goes down occasionally, and when that happens many things start breaking. It may very well be that there are things planned or in progress, but until they are both usable and used by standard tools, which for me means buildout and setuptools, they are invisible.

Fair enough, Pip has now the mirroring protocol implemented. I think they want to make it a default option for the next major Pip release. IOW, you should not suffer for downtimes using pip. We'd need to add the same feature in easy_install and zc.buildout. But since Pip did it, I think it's possible. We have 5 mirrors run by the community (http://pypi.python.org/mirrors) and I suspect porting pip's feature to zc.buildout and easy_install would take less time than creating a <Name your cloud service> app. Cheers Tarek

Wichert.

-- Tarek Ziadé | http://ziade.org

On 9/27/11 8:51 AM, Tarek Ziadé wrote:

On Tue, Sep 27, 2011 at 2:39 PM, Wichert Akkerman wrote: ..

...

...

I understand where you're coming from but, .. I think it's saner to rely on proven technology than to invent our own protocol. NIH?

This also feels like a problem that has already been solved in various ways by Debian, RedHat, CPAN and others.

Yes, and we've found a way similar to CPAN, with some Python specifics (PyPI download statistics mainly)

Oh my, we're cycling again.

Nothing personal to you or Jim, but I have a sudden fatigue on packaging because it seems like people are ignoring what's being done to complain afterwards about us suffering of some kind of NIH :)

If you're seeing anything you don't like in PEP 381 (accepted a while ago), go ahead and propose some improvements.

But please keep in mind that we've looked at other systems before we wrote that PEP.

+1 If someone (Tarek in this case) wants to take the lead on this then I don't care where the fix was invented ;-) Obviously the less we make volunteers do, and the more we can outsource the better (I say this in Plone-land repeatedly) but sometimes inventing-something-here is the right thing to do. And it certainly would not preclude other (cloud) fixes AFAICT. Alex

Cheers Tarek

-- Alex Clark · http://aclark.net

On Tue, Sep 27, 2011 at 8:40 AM, Tarek Ziadé wrote:

On Tue, Sep 27, 2011 at 2:27 PM, Jim Fulton wrote: ...

...

I understand where you're coming from but, ..

Sorry, I don't understand what you imply here.

I understand why you don't want to rely on a proprietary solution.

...

I think it's saner to rely on proven technology than to invent our own protocol. NIH?

Ah sorry I misunderstood then. I thought CloudFront was a proprietary platform, with its own protocol.

It's a reverse proxy. You point it at s3 and at a web server and it caches. Of course, it has aspects that are specific to it's implementation.

If you're saying that we can move away from CloudFront at any time and have the same feature elsewhere, then it's perfect.

If we move to something else, *some* changes will be necessary, but we can certainly move. I agree, it's perfect. ;)

If you're saying that CloudFront is proven technology and that we should not worry about relying on them, then I think we can do better for the community to get locked-in for this, and continue to work on an open protocol where everyone can participate by providing a spare server.  But maybe that's just me ?

It's nice to have a hobby. :) But I don't want to have to update buildout *just* because of an itch to have a custom protocol.

Most of the mirroring protocol was inspired by Perl's CPAN btw.

<shrug>

But the use case is usually: PyPI is down, we fallback to a mirror. I don't think it's more complicated than this.

I don't agree. On multiple levels. PYPI is often up but slow. It's also in the wrong place. A CDN should provide better performance, reliability and locality. A client has to: - try pypi - fallback to "last" - If that's down, decide what other indexes to check I don't see how having timestamps help unless you know what the current timestamp is, unless you say that you'll reject a mirror with a timestamp more than some period in the past. It's not clear what this time delta should be and, in any case, the client needs to first validate a mirror by checking it's timestamp. I think this protocol is going to be hard to get right.

...

- It either requires extra dns calls or relies to heavily on the last mirror, which is probably likely  to be the least reliable.

Once you have the list, I don't think you require extra call.

see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py

It has to make extra dns calls to resolve the other mirror names to ips.

...

Life is short. We don't have to invent this ourselves.

Ah well, yeah -- Not sure what you are proposing right now.

If you imply that everything should be solved on server-side, and that we should not have mirroring

I think we should pick a good CDN and use it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On Tue, Sep 27, 2011 at 5:35 PM, Jim Fulton wrote:

...

...

I understand where you're coming from but, ..

...

Sorry, I don't understand what you imply here.

I understand why you don't want to rely on a proprietary solution.

But it's true that I don't want to rely on a proprietary solution. That's based on a good reason I think, mentioned at the end of this mail. ...

...

If you're saying that CloudFront is proven technology and that we should not worry about relying on them, then I think we can do better for the community to get locked-in for this, and continue to work on an open protocol where everyone can participate by providing a spare server.  But maybe that's just me ?

It's nice to have a hobby. :)

I think you've missed what we, bunch of hobbyists, did in the past two years + 5 community mirrors are up and running, collecting download stats that get merged + pip does work with the mirrors, and offer fallback options It's too bad you were not there to tell us we were wasting our time and how awesome CloudFront was ;) But at this point, the shortest road to a better PyPI is to add the mirroring support to other clients, pip showed the lead. And if zc.buildout uses Distribute, it should get this feature at some point. But having a CloudFront-based PyPI could also be interesting in parallel, I am not saying it's not. But the project is stalled, and has the defaults I've mentioned.

But I don't want to have to update buildout *just* because of an itch to have a custom protocol.

I kind of wonder how hard it would be to have a standalone pypi download client, ripped off from python 3.3's packaging, so you would not have to worry about this. And, well, you do not sound like you want to spend time in these matters in any case, so if someone brings a patch I hope you will not refuse it.

...

But the use case is usually: PyPI is down, we fallback to a mirror. I don't think it's more complicated than this.

I don't agree.  On multiple levels.  PYPI is often up but slow.

That's an orthogonal issue : any server can be slow. One better way to drastically speed up buildout is to download / build stuff in parallel imo.

It's also in the wrong place.  A CDN should provide better performance, reliability and locality.

Locality is indeed important, and picking up the nearest server is great. Reliability is also solved by the mirrors.

A client has to:

- try pypi - fallback to "last" - If that's down, decide what other indexes to check

I don't see how having timestamps help unless you know what the current timestamp is, unless you say that you'll reject a mirror with a timestamp more than some period in the past.

How hard it is to make those decisions ? Do you really think getting the current timestamp is that hard ? And the mirror timestamp, http://b.pypi.python.org/last-modified In all you've said I fail to see how complicated it is, or long to do. The ordering I see is: normal behavior: - if the cache is too old: get the list of mirrors (-> the list of mirrors and their timestamps get cached) - pick the closest one - use it the server times out: - try the "next closest"

It's not clear what this time delta should be and, in any case, the client needs to first validate a mirror by checking it's timestamp.

This is the job of the client yes. An option that says, discard mirrors that are > 1 day, or 5 hours etc. Keeping a local cache that gets updated eventually is sufficient.

I think this protocol is going to be hard to get right.

Maybe ? but if a v1 allows us to switch from server 1 being down to server 2, it's already a success, no ? servers that *we* the community, manage.

...

...

- It either requires extra dns calls or relies to heavily on the last mirror, which is probably likely  to be the least reliable.

Once you have the list, I don't think you require extra call.

see http://hg.python.org/cpython/file/84280fac98b9/Lib/packaging/pypi/mirrors.py

It has to make extra dns calls to resolve the other mirror names to ips.

Yeah, once per session. but in any case, this is not a decision you're making on every download. It's something you do when you start to download stuff, and/or when a server times out. You stick with a server once it's working

...

...

Life is short. We don't have to invent this ourselves.

Ah well, yeah -- Not sure what you are proposing right now.

If you imply that everything should be solved on server-side, and that we should not have mirroring

I think we should pick a good CDN and use it.

I won't object, because this is orthogonal to the mirroring stuff, but I am not going to scratch the mirroring efforts to move PyPI to a single shop. Every service on the planet, even Amazon, can be down. oh, my: - https://forums.aws.amazon.com/message.jspa?messageID=244986 - http://money.cnn.com/2011/04/22/technology/amazon_ec2_cloud_outage/index.htm. - http://www.labnol.org/internet/amazon-s3-cloudfront-down/5667/ - https://forums.aws.amazon.com/message.jspa?messageID=134012 - https://forums.aws.amazon.com/message.jspa?messageID=177654 Do we really want Amazon to handle PyPI ? I prefer a bunch of community mirrors. Heck, I have one at Mozilla, and might make it public one day :) Or maybe the optimal solution is our own CND proxy so we don't deal with this on client side. But in any case, I'd rather have a Pythoneer from our community behind every mirror server, I can trust </music> Cheers Tarek -- Tarek Ziadé | http://ziade.org

On Sep 27, 2011, at 10:33 AM, Tarek Ziadé wrote:

On Tue, Sep 27, 2011 at 7:24 PM, Noah Kantrowitz wrote:

...

I've copied our current draft plan for the structure of the PyPI infra. I stress this is just a draft as it stands today and is not final.

Thanks,

I've started this page with all the content : http://wiki.python.org/moin/BetterPyPI

If people have feedback please also make sure to let me know so I we can discuss it. --Noah

Am 27.09.2011 13:12, schrieb Jim Fulton:

On Tue, Sep 27, 2011 at 6:07 AM, Lennart Regebro wrote:

...

On Tue, Sep 27, 2011 at 11:40, Tarek Ziadé wrote:

...

1/ stability and high availability

How are opinions on setting up country-specific PyPI mirrors? The lag to the US is pretty severe in Poland, and I suspect my buildouts would benefit from having a server in Poland. Now, of course, it could be called x.pypi.python.org, but maybe we should have aliases such as pl.pypi.python.org as well?

I have no strong opinion on the issue, what do others think?

Wouldn't CloudFront make this moot?

I personally don't believe the CloudFront project is feasible - IMO, it just won't work. This is because there needs to be both dynamic content (at least for uploads) and static content; CloudFront can only mirror the static content. However, since this all forms a single web site, integrating them will be either infeasible or pointless. It would be possible to use CloudFront if the clients would be changed. But if so, it would be best if the CloudFront copy would just be a PEP 381 mirror. Regards, Martin

On Tue, Sep 27, 2011 at 1:25 PM, Tarek Ziadé wrote:

On Tue, Sep 27, 2011 at 5:35 PM, Jim Fulton wrote:

...

...

But I don't want to have to update buildout *just* because of an itch to have a custom protocol.

I kind of wonder how hard it would be to have a standalone pypi download client, ripped off from python 3.3's packaging, so you would not have to worry about this.

I doubt I'm going to be able to avoid worrying about it. Still a reference client implementation would be useful.

And, well, you do not sound like you want to spend time in these matters in any case,

I don't know what you mean. Not sure I care. :)

so if someone brings a patch I hope you will not refuse it.

No. I'll eventually implement it if no one else does.

...

...

But the use case is usually: PyPI is down, we fallback to a mirror. I don't think it's more complicated than this.

I don't agree.  On multiple levels.  PYPI is often up but slow.

That's an orthogonal issue :  any server can be slow.

A service can be fast even if an individual server is slow. Also, CDNs can make lots of horsepower available that is shared among multiple customers. I really doubt that anything we build will be faster.

One better way to drastically speed up buildout is to  download / build stuff in parallel imo.

Thats true and something I'd like to do at some point. That's one of the reasons I expect I'll have to worry about the protocol.

...

It's also in the wrong place.  A CDN should provide better performance, reliability and locality.

Locality is indeed important, and picking up the nearest server is great. Reliability is also solved by the mirrors.

At the expense of increased complexity on the client.

...

A client has to:

- try pypi - fallback to "last" - If that's down, decide what other indexes to check

I don't see how having timestamps help unless you know what the current timestamp is, unless you say that you'll reject a mirror with a timestamp more than some period in the past.

How hard it is to make those decisions ?

It's not "hard" conceptually, but it's still a lot of implementation complexity and a lot of extra network requests.

Do you really think getting the current timestamp is that hard ?

And the mirror timestamp,

 http://b.pypi.python.org/last-modified

In all you've said I fail to see how complicated it is, or long to do.

That's an extra HTTP request I need to make when I'm considering use of a mirror. If the first mirror I check seems to be out of date, I may need to check all the mirrors. It's an open question what should be considered potentially out of date, a timestamp older than an hour? a day?

The ordering I see is:

normal behavior: - if the cache is too old:

How old is too old?

get the list of mirrors  (-> the list of mirrors and their timestamps get cached)

They'll only get cached for the program invocation. This means I have to potentially check lots of mirrors every time someone runs buildout. I can reduce latency by doing this in parallel, but that's still a lot of requests.

- pick the closest one

How do I decide what's closest? Did you mean closest? or most up to date

- use it

the server times out: - try the "next closest"

...

It's not clear what this time delta should be and, in any case, the client needs to first validate a mirror by checking it's timestamp.

This is the job of the client yes. An option that says, discard mirrors that are > 1 day, or 5 hours etc.

"etc" is just waving hands. Selecting the right value is hard, possibly application dependent. Is this a configuration variable? Now the user has something to deal with.

Keeping a local cache that gets updated eventually is sufficient.

In process, or on disk? This just gets better and better. :)

...

I think this protocol is going to be hard to get right.

Maybe ? but if a v1 allows us to switch from server 1 being down to server 2, it's already a success, no ?

servers that *we* the community, manage.

I fail to see why this is inherently a good thing. I don't like "managing" things. Less work is good. ...

Do we really want Amazon to handle PyPI ?

Yes, or Rackspace, or Google, or AOL, or, whatever. Just not us. (I suspect some of these might even do it for free.)

I prefer a bunch of community mirrors. Heck, I have one at Mozilla, and might make it public one day  :)

Or maybe the optimal solution is our own CND proxy so we don't deal with this on client side.

Uh, yeah, sure. FWIW, it hadn't occurred to me to use a CDN until a conversation a few days ago. Doh. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On Sep 28, 2011, at 09:59 AM, Greg Ewing wrote:

Are there really no existing open-source mirroring systems out there?

rsync? :) -Barry

That's an extra HTTP request I need to make when I'm considering use of a mirror. If the first mirror I check seems to be out of date, I may need to check all the mirrors. It's an open question what should be considered potentially out of date, a timestamp older than an hour? a day?

15 minutes; if all mirrors are older than 15 minutes, it probably means that the master is down, and you should then use the newest mirror you can find.

How old is too old?

More than 15 minutes.

...

get the list of mirrors (-> the list of mirrors and their timestamps get cached)

They'll only get cached for the program invocation.

No, you can also cache the fastest mirror across invocations, and keep using it if it is younger than 15 minutes.

...

- pick the closest one

How do I decide what's closest? Did you mean closest?

He probably means the one that responds fastest.

"etc" is just waving hands. Selecting the right value is hard, possibly application dependent. Is this a configuration variable? Now the user has something to deal with.

No, it will be 15 minutes. People might request being able to configure it, but it won't really be necessary to do so. Regards, Martin