From jim at zope.com Mon Oct 16 19:48:44 2006
From: jim at zope.com (Jim Fulton)
Date: Mon, 16 Oct 2006 13:48:44 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
Message-ID: <4533C5FC.6050506@zope.com>
PyPI seems to e down or severly overloaded. Does anyone have any
idea why? Is it time to set up some mirrors?
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From jim at zope.com Tue Oct 17 15:13:04 2006
From: jim at zope.com (Jim Fulton)
Date: Tue, 17 Oct 2006 09:13:04 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <4533C5FC.6050506@zope.com>
References: <4533C5FC.6050506@zope.com>
Message-ID: <4534D6E0.1080109@zope.com>
Jim Fulton wrote:
> PyPI seems to e down or severly overloaded. Does anyone have any
> idea why? Is it time to set up some mirrors?
I'm a bit disappointed that there was no written response to this.
Shortly after I sent this, the behavior of pypi improved
quite a bit. I don't know if this was in response to my post
or if it just got better on it's own.
As we adopt setuptools and zc.buildout, pypi is becoming a critical
resource that needs to be available. I need to know if I can
rely on www.python.org/pypi, or if I need to look at doing something
else. My preference, by far, is to rely on pypi.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From jim at zope.com Tue Oct 17 15:14:42 2006
From: jim at zope.com (Jim Fulton)
Date: Tue, 17 Oct 2006 09:14:42 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <4534D6E0.1080109@zope.com>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
Message-ID: <4534D742.3060603@zope.com>
Jim Fulton wrote:
> Jim Fulton wrote:
>> PyPI seems to e down or severly overloaded. Does anyone have any
>> idea why? Is it time to set up some mirrors?
>
> I'm a bit disappointed that there was no written response to this.
> Shortly after I sent this, the behavior of pypi improved
> quite a bit. I don't know if this was in response to my post
> or if it just got better on it's own.
>
> As we adopt setuptools and zc.buildout, pypi is becoming a critical
> resource that needs to be available. I need to know if I can
> rely on www.python.org/pypi, or if I need to look at doing something
> else. My preference, by far, is to rely on pypi.
I should have noted in my note that my intent is not to complain.
Mainly, I want to make sure that we we're in agreement about the goals
and to make sure the PSF allocates the rsources needed to achieve them.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From martin at v.loewis.de Tue Oct 17 19:37:16 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 17 Oct 2006 19:37:16 +0200
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <4534D742.3060603@zope.com>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com>
Message-ID: <453514CC.8000206@v.loewis.de>
Jim Fulton schrieb:
> I should have noted in my note that my intent is not to complain.
> Mainly, I want to make sure that we we're in agreement about the goals
> and to make sure the PSF allocates the rsources needed to achieve them.
Can you volunteer yourself or somebody else to help in maintaining
PyPI?
Regards,
Martin
From martin at v.loewis.de Tue Oct 17 20:08:35 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 17 Oct 2006 20:08:35 +0200
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de>
<59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
Message-ID: <45351C23.4090003@v.loewis.de>
Ben Bangert schrieb:
> I'll volunteer to help out in maintenance. I'm in agreement with Jim
> that PyPI definitely needs some sort of mirror system.
I'm not so sure this is the definite answer. If the system is
overloaded, it might be because it consumes too many resources itself
(in which case mirroring wouldn't help), or because something else
on the machine is consuming too many resources (in which case the
installation should be moved elsewhere entirely).
Mirroring it is quite a challenge, as you have to consider write
operations to the mirror. I think efforts should first go into
improving the performance.
> While I currently
> advocate that people using setuptools based installations maintain their
> own egg repo in case PyPI is unavailable, it'd be great to have mirrors.
> On a side-note, I don't know who maintains PyPI now, but thank you for
> the great job, for a single un-mirrored site, the reliability thus far
> has been great.
It's the primary author, Richard Jones. To my knowledge, he does both
source code improvements, as well as maintaining the installation.
You might want to start with creating an installation on a test machine
and study its performance.
Regards,
Martin
From jim at zope.com Tue Oct 17 20:21:09 2006
From: jim at zope.com (Jim Fulton)
Date: Tue, 17 Oct 2006 14:21:09 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <453514CC.8000206@v.loewis.de>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de>
Message-ID: <45351F15.6090707@zope.com>
Martin v. L?wis wrote:
> Jim Fulton schrieb:
>> I should have noted in my note that my intent is not to complain.
>> Mainly, I want to make sure that we we're in agreement about the goals
>> and to make sure the PSF allocates the rsources needed to achieve them.
>
> Can you volunteer yourself or somebody else to help in maintaining
> PyPI?
Possibly. Is it an issue of people? I really don't know what the
cause of the problems yesterday were. Was it a software problem?
Load? I'd like to understand what the issues are first.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From ben at groovie.org Tue Oct 17 19:50:51 2006
From: ben at groovie.org (Ben Bangert)
Date: Tue, 17 Oct 2006 10:50:51 -0700
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <453514CC.8000206@v.loewis.de>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de>
Message-ID: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
On Oct 17, 2006, at 10:37 AM, Martin v. L?wis wrote:
> Jim Fulton schrieb:
>> I should have noted in my note that my intent is not to complain.
>> Mainly, I want to make sure that we we're in agreement about the
>> goals
>> and to make sure the PSF allocates the rsources needed to achieve
>> them.
>
> Can you volunteer yourself or somebody else to help in maintaining
> PyPI?
I'll volunteer to help out in maintenance. I'm in agreement with Jim
that PyPI definitely needs some sort of mirror system. While I
currently advocate that people using setuptools based installations
maintain their own egg repo in case PyPI is unavailable, it'd be
great to have mirrors. On a side-note, I don't know who maintains
PyPI now, but thank you for the great job, for a single un-mirrored
site, the reliability thus far has been great.
Cheers,
Ben
From jim at zope.com Tue Oct 17 20:26:54 2006
From: jim at zope.com (Jim Fulton)
Date: Tue, 17 Oct 2006 14:26:54 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de>
<59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
Message-ID: <4535206E.10008@zope.com>
Ben Bangert wrote:
> On Oct 17, 2006, at 10:37 AM, Martin v. L?wis wrote:
>
>> Jim Fulton schrieb:
>>> I should have noted in my note that my intent is not to complain.
>>> Mainly, I want to make sure that we we're in agreement about the goals
>>> and to make sure the PSF allocates the rsources needed to achieve them.
>>
>> Can you volunteer yourself or somebody else to help in maintaining
>> PyPI?
>
> I'll volunteer to help out in maintenance. I'm in agreement with Jim
> that PyPI definitely needs some sort of mirror system. While I currently
> advocate that people using setuptools based installations maintain their
> own egg repo in case PyPI is unavailable,
I'm not sure that is very practical. We're using lots of eggs
that we don't produce. They may depend on still more that we
don't produce...
> it'd be great to have mirrors.
The only obvious way to have a usable separate
repo is to create a mirror and I'm afraid having a lot of (or a few)
people creating mirrors could end up crushing pypi in the process.
I think this will require some coordination.
> On a side-note, I don't know who maintains PyPI now, but thank you for
> the great job,
Absolutely!
> for a single un-mirrored site, the reliability thus far
> has been great.
Well, yesterday wasn't the first day I had a problem.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From martin at v.loewis.de Tue Oct 17 20:36:21 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 17 Oct 2006 20:36:21 +0200
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <45351F15.6090707@zope.com>
References: <4533C5FC.6050506@zope.com>
<4534D6E0.1080109@zope.com> <4534D742.3060603@zope.com>
<453514CC.8000206@v.loewis.de> <45351F15.6090707@zope.com>
Message-ID: <453522A5.7040203@v.loewis.de>
Jim Fulton schrieb:
>>> I should have noted in my note that my intent is not to complain.
>>> Mainly, I want to make sure that we we're in agreement about the goals
>>> and to make sure the PSF allocates the rsources needed to achieve them.
>> Can you volunteer yourself or somebody else to help in maintaining
>> PyPI?
>
> Possibly. Is it an issue of people? I really don't know what the
> cause of the problems yesterday were. Was it a software problem?
> Load? I'd like to understand what the issues are first.
Primarily, it's people. Very few people understand the PyPI code well,
and nearly nobody has time research performance problems. My guess is
that the code could be improved to perform certain actions better.
For example, it's know that PyPI cannot really cope with web crawlers
too well. If too many of them hit him at a bad spot, it will go down.
It takes man-power to investigate these problems and propose solutions;
solutions based on speculation (like "let's mirror") are likely wrong.
I'm not sure whether there is any way to investigate performance
problems after the fact; if there is, it likely isn't possible in
this case. So one would have to define a performance analysis procedure
first, implement it, and wait for the performance to go down again
to understand what's commonly causing the problem. Of course, if
somebody would log in and analyse *while* the problem is happening,
there might be a much better chance at understanding the problem.
OTOH, it's really no fun to investigate something remotely on
an overloaded machine, so likely, whoever is in charge would just
make the problem go away instead of studying it.
Regards,
Martin
From amk at amk.ca Tue Oct 17 21:11:23 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Tue, 17 Oct 2006 15:11:23 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <45351F15.6090707@zope.com>
References: <4533C5FC.6050506@zope.com> <4534D6E0.1080109@zope.com>
<4534D742.3060603@zope.com> <453514CC.8000206@v.loewis.de>
<45351F15.6090707@zope.com>
Message-ID: <20061017191123.GA8987@rogue.amk.ca>
On Tue, Oct 17, 2006 at 02:21:09PM -0400, Jim Fulton wrote:
> Possibly. Is it an issue of people? I really don't know what the
> cause of the problems yesterday were. Was it a software problem?
> Load? I'd like to understand what the issues are first.
Unclear. When your catalog-sig e-mail first arrived yesterday, I
logged in to ximinez to see if the system was overloaded, and uptime
reported load averages something like 0.2, 0.5, 0.7, which made it
look like the system wasn't very loaded, but was coming down from a
greater load. (A rude crawler, maybe?) I didn't investigate further,
because it didn't look like there was any problem at the time that
needed fixing.
--amk
From richardjones at optushome.com.au Wed Oct 18 06:43:49 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Wed, 18 Oct 2006 14:43:49 +1000
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <45351C23.4090003@v.loewis.de>
References: <4533C5FC.6050506@zope.com>
<59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org>
<45351C23.4090003@v.loewis.de>
Message-ID: <200610181443.50005.richardjones@optushome.com.au>
Sorry I didn't respond in a more immediate manner - I'm quite busy with work
and organising papers for OSDC '06.
On Wednesday 18 October 2006 04:08, Martin v. L?wis wrote:
> I'm not so sure this is the definite answer. If the system is
> overloaded, it might be because it consumes too many resources itself
> (in which case mirroring wouldn't help), or because something else
> on the machine is consuming too many resources (in which case the
> installation should be moved elsewhere entirely).
We still have the problem that the PyPI browse interface is quite
CPU-intensive and if it's hit by a bot it'll definitely impact on overall
system performance.
We have a check in the browse code to see if the user agent matches:
botre = re.compile(r'^$|brains|yeti|myie2|findlinks|ia_archiver|psycheclone|
badass|crawler|slurp|spider|bot|scooter|infoseek|looksmart|jeeves', re.I)
and if it does then the browse returns an empty page. This RE is pretty
complete - I use it to redirect bots to a dedicated ZEO client at work.
I've added a robots.txt to http://cheeseshop.python.org (I always meant to,
but never got around to it). Unfortunately, I'm not
sure "Disallow: /pypi?:action=browse" will be handled properly. We'll see.
Richard
From richardjones at optushome.com.au Wed Oct 18 06:45:23 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Wed, 18 Oct 2006 14:45:23 +1000
Subject: [Catalog-sig] Cheese shop: search
In-Reply-To: <4517FAEA.3010801@colorstudy.com>
References: <4517FAEA.3010801@colorstudy.com>
Message-ID: <200610181445.23914.richardjones@optushome.com.au>
On Tuesday 26 September 2006 01:51, Ian Bicking wrote:
> Can we please get the search back?
If someone writes it, sure. The old search was far less useful that the Google
search is now.
Richard
From jim at zope.com Wed Oct 18 13:20:05 2006
From: jim at zope.com (Jim Fulton)
Date: Wed, 18 Oct 2006 07:20:05 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <200610181443.50005.richardjones@optushome.com.au>
References: <4533C5FC.6050506@zope.com> <59F2B498-249A-4E25-968D-8D74A034E4F5@groovie.org> <45351C23.4090003@v.loewis.de>
<200610181443.50005.richardjones@optushome.com.au>
Message-ID: <45360DE5.10007@zope.com>
Richard Jones wrote:
> Sorry I didn't respond in a more immediate manner - I'm quite busy with work
> and organising papers for OSDC '06.
>
> On Wednesday 18 October 2006 04:08, Martin v. L?wis wrote:
>> I'm not so sure this is the definite answer. If the system is
>> overloaded, it might be because it consumes too many resources itself
>> (in which case mirroring wouldn't help), or because something else
>> on the machine is consuming too many resources (in which case the
>> installation should be moved elsewhere entirely).
>
> We still have the problem that the PyPI browse interface is quite
> CPU-intensive and if it's hit by a bot it'll definitely impact on overall
> system performance.
Is that what happened the other day?
> We have a check in the browse code to see if the user agent matches:
>
> botre = re.compile(r'^$|brains|yeti|myie2|findlinks|ia_archiver|psycheclone|
> badass|crawler|slurp|spider|bot|scooter|infoseek|looksmart|jeeves', re.I)
>
> and if it does then the browse returns an empty page. This RE is pretty
> complete - I use it to redirect bots to a dedicated ZEO client at work.
>
> I've added a robots.txt to http://cheeseshop.python.org (I always meant to,
> but never got around to it). Unfortunately, I'm not
> sure "Disallow: /pypi?:action=browse" will be handled properly. We'll see.
Is there something somewhere that documents the architecture?
I'd like to try to offer helpful suggestions, but without more knowledge
of what's going on, that's hard to do.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optushome.com.au Wed Oct 18 13:46:54 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Wed, 18 Oct 2006 21:46:54 +1000
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <45360DE5.10007@zope.com>
References: <4533C5FC.6050506@zope.com>
<200610181443.50005.richardjones@optushome.com.au>
<45360DE5.10007@zope.com>
Message-ID: <200610182146.54648.richardjones@optushome.com.au>
On Wednesday 18 October 2006 21:20, Jim Fulton wrote:
> Richard Jones wrote:
> > We still have the problem that the PyPI browse interface is quite
> > CPU-intensive and if it's hit by a bot it'll definitely impact on overall
> > system performance.
>
> Is that what happened the other day?
Sorry, I'm just guessing. I don't have the time to look into it in more detail
at the moment.
> Is there something somewhere that documents the architecture?
No, at the moment there's just the code.
Richard
From jim at zope.com Wed Oct 18 15:39:58 2006
From: jim at zope.com (Jim Fulton)
Date: Wed, 18 Oct 2006 09:39:58 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <200610182146.54648.richardjones@optushome.com.au>
References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com>
<200610182146.54648.richardjones@optushome.com.au>
Message-ID: <45362EAE.6090308@zope.com>
Richard Jones wrote:
> On Wednesday 18 October 2006 21:20, Jim Fulton wrote:
>> Richard Jones wrote:
>>> We still have the problem that the PyPI browse interface is quite
>>> CPU-intensive and if it's hit by a bot it'll definitely impact on overall
>>> system performance.
>> Is that what happened the other day?
>
> Sorry, I'm just guessing. I don't have the time to look into it in more detail
> at the moment.
Perhaps someone should look at the logs. Ben Bangert
volunteered to help out. :) I can help out some too, if necessary.
>> Is there something somewhere that documents the architecture?
>
> No, at the moment there's just the code.
Is the code in a publicly available repository?
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From amk at amk.ca Wed Oct 18 15:58:19 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 18 Oct 2006 09:58:19 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <45362EAE.6090308@zope.com>
References: <4533C5FC.6050506@zope.com>
<200610181443.50005.richardjones@optushome.com.au>
<45360DE5.10007@zope.com>
<200610182146.54648.richardjones@optushome.com.au>
<45362EAE.6090308@zope.com>
Message-ID: <20061018135819.GA18165@localhost.localdomain>
On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote:
> Perhaps someone should look at the logs. Ben Bangert
> volunteered to help out. :) I can help out some too, if necessary.
Does Bangert have SSH access to python.org? If not, I suggest he
write to pydotorg at python.org and provide an SSH key. I can add him to
ximinez.
(And is he someone you know reasonably well? Is he a Zope
Corp. employee?)
> Is the code in a publicly available repository?
https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one
point it used SQLite, but I don't know if Richard left traces of
SQLite support in the code.
--amk
From jim at zope.com Wed Oct 18 16:04:50 2006
From: jim at zope.com (Jim Fulton)
Date: Wed, 18 Oct 2006 10:04:50 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <20061018135819.GA18165@localhost.localdomain>
References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> <45362EAE.6090308@zope.com>
<20061018135819.GA18165@localhost.localdomain>
Message-ID: <45363482.7050108@zope.com>
A.M. Kuchling wrote:
> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote:
>> Perhaps someone should look at the logs. Ben Bangert
>> volunteered to help out. :) I can help out some too, if necessary.
>
> Does Bangert have SSH access to python.org? If not, I suggest he
> write to pydotorg at python.org and provide an SSH key. I can add him to
> ximinez.
>
> (And is he someone you know reasonably well? Is he a Zope
> Corp. employee?)
I don't know him at all, other than through this list.
He is not a ZC employee.
>> Is the code in a publicly available repository?
>
> https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one
> point it used SQLite, but I don't know if Richard left traces of
> SQLite support in the code.
Thanks.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optushome.com.au Wed Oct 18 23:21:36 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Thu, 19 Oct 2006 07:21:36 +1000
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <20061018135819.GA18165@localhost.localdomain>
References: <4533C5FC.6050506@zope.com> <45362EAE.6090308@zope.com>
<20061018135819.GA18165@localhost.localdomain>
Message-ID: <200610190721.37079.richardjones@optushome.com.au>
On Wednesday 18 October 2006 23:58, A.M. Kuchling wrote:
> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote:
> > Is the code in a publicly available repository?
>
> https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one
> point it used SQLite, but I don't know if Richard left traces of
> SQLite support in the code.
Follow the "Developers" link from the http://cheeseshop.python.org/pypi
sidebar for more information.
Richard
From jim at zope.com Wed Oct 18 23:49:35 2006
From: jim at zope.com (Jim Fulton)
Date: Wed, 18 Oct 2006 17:49:35 -0400
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <200610190721.37079.richardjones@optushome.com.au>
References: <4533C5FC.6050506@zope.com> <45362EAE.6090308@zope.com>
<20061018135819.GA18165@localhost.localdomain>
<200610190721.37079.richardjones@optushome.com.au>
Message-ID: <4536A16F.1010204@zope.com>
Richard Jones wrote:
> On Wednesday 18 October 2006 23:58, A.M. Kuchling wrote:
>> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote:
>>> Is the code in a publicly available repository?
>> https://svn.python.org/packages/trunk/ . Requires PostgreSQL; at one
>> point it used SQLite, but I don't know if Richard left traces of
>> SQLite support in the code.
>
> Follow the "Developers" link from the http://cheeseshop.python.org/pypi
> sidebar for more information.
Thanks.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From ianb at colorstudy.com Thu Oct 19 00:14:59 2006
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 18 Oct 2006 17:14:59 -0500
Subject: [Catalog-sig] PyPI overloaded(?)
In-Reply-To: <20061018135819.GA18165@localhost.localdomain>
References: <4533C5FC.6050506@zope.com> <200610181443.50005.richardjones@optushome.com.au> <45360DE5.10007@zope.com> <200610182146.54648.richardjones@optushome.com.au> <45362EAE.6090308@zope.com>
<20061018135819.GA18165@localhost.localdomain>
Message-ID: <4536A763.5020904@colorstudy.com>
A.M. Kuchling wrote:
> On Wed, Oct 18, 2006 at 09:39:58AM -0400, Jim Fulton wrote:
>> Perhaps someone should look at the logs. Ben Bangert
>> volunteered to help out. :) I can help out some too, if necessary.
>
> Does Bangert have SSH access to python.org? If not, I suggest he
> write to pydotorg at python.org and provide an SSH key. I can add him to
> ximinez.
>
> (And is he someone you know reasonably well? Is he a Zope
> Corp. employee?)
I can vouch for Ben; he's the developer of Pylons (http://pylonshq.com/)
--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
From ianb at colorstudy.com Fri Oct 20 18:15:46 2006
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 20 Oct 2006 11:15:46 -0500
Subject: [Catalog-sig] Cheese shop: search
In-Reply-To: <200610181445.23914.richardjones@optushome.com.au>
References: <4517FAEA.3010801@colorstudy.com>
<200610181445.23914.richardjones@optushome.com.au>
Message-ID: <4538F632.4010302@colorstudy.com>
Richard Jones wrote:
> On Tuesday 26 September 2006 01:51, Ian Bicking wrote:
>> Can we please get the search back?
>
> If someone writes it, sure. The old search was far less useful that the Google
> search is now.
I thought my email outlined why the Google search is completely useless.
To summarize: it's useless. The old search was a little awkward, but
it worked. Simply putting it back as it was would be a step forward.
And actually the only improvement I'd like to see in the old search is a
way to search all fields at once, and then a search form on the front
page -- but even without these the search is functional. Well, if we
wanted to be really fancy we'd want a full text indexer in the database,
which probably isn't installed, and then create a field that contained
the full text of each entry by simply concatenating items, and probably
giving weight to some metadata by doing something scrappy like:
full_text = title + title + keywords + keywords + topics + topics +
description
Otherwise if we are sticking with Google, cheese shop at least needs to
tell google not to index the browse pages. And probably needs to give
google a sitemap so it will index all the packages more prompty, as it's
not indexing nearly often enough right now.
--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
From fdrake at gmail.com Fri Oct 20 18:41:38 2006
From: fdrake at gmail.com (Fred Drake)
Date: Fri, 20 Oct 2006 12:41:38 -0400
Subject: [Catalog-sig] Cheese shop: search
In-Reply-To: <4538F632.4010302@colorstudy.com>
References: <4517FAEA.3010801@colorstudy.com>
<200610181445.23914.richardjones@optushome.com.au>
<4538F632.4010302@colorstudy.com>
Message-ID: <9cee7ab80610200941l72a06d38tc81be933293777de@mail.gmail.com>
On 10/20/06, Ian Bicking <ianb at colorstudy.com> wrote:
> I thought my email outlined why the Google search is completely useless.
> To summarize: it's useless. The old search was a little awkward, but
> it worked. Simply putting it back as it was would be a step forward.
Agreed!
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From timcera at earthlink.net Thu Oct 26 06:51:36 2006
From: timcera at earthlink.net (Tim Cera)
Date: Thu, 26 Oct 2006 00:51:36 -0400
Subject: [Catalog-sig] Add requirements to XMLRPC?
Message-ID: <200610260051.36324.timcera@earthlink.net>
Hello,
It would really be nice for a project that I am working on to get the
package requirements from XMLRPC. What are the issues with adding the
requirements to XMLPRC?
Kindest regards,
Tim Cera
From richardjones at optushome.com.au Thu Oct 26 23:43:38 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Fri, 27 Oct 2006 07:43:38 +1000
Subject: [Catalog-sig] Add requirements to XMLRPC?
In-Reply-To: <200610260051.36324.timcera@earthlink.net>
References: <200610260051.36324.timcera@earthlink.net>
Message-ID: <200610270743.38255.richardjones@optushome.com.au>
On Thursday 26 October 2006 14:51, Tim Cera wrote:
> It would really be nice for a project that I am working on to get the
> package requirements from XMLRPC. What are the issues with adding the
> requirements to XMLPRC?
Someone has to write the code. I check it, commit it and install it.
http://wiki.python.org/moin/CheeseShopDev has more info.
Richard
From jeremy.kloth at 4suite.org Sat Oct 28 20:12:53 2006
From: jeremy.kloth at 4suite.org (Jeremy Kloth)
Date: Sat, 28 Oct 2006 12:12:53 -0600
Subject: [Catalog-sig] Add requirements to XMLRPC?
In-Reply-To: <200610270743.38255.richardjones@optushome.com.au>
References: <200610260051.36324.timcera@earthlink.net>
<200610270743.38255.richardjones@optushome.com.au>
Message-ID: <200610281212.54391.jeremy.kloth@4suite.org>
On Thursday 26 October 2006 3:43 pm, Richard Jones wrote:
> On Thursday 26 October 2006 14:51, Tim Cera wrote:
> > It would really be nice for a project that I am working on to get the
> > package requirements from XMLRPC. What are the issues with adding the
> > requirements to XMLPRC?
>
> Someone has to write the code. I check it, commit it and install it.
The attached patch adds the 'requires', 'provides' and 'obsoletes' fields
to the release_data() RPC method.
Note this is untested, but copied from webui.py where is builds the PKG-INFO.
--
Jeremy Kloth
http://4suite.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rpc.py
Type: application/x-python
Size: 3063 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061028/e41249b7/attachment.bin
From jeremy.kloth at 4suite.org Sat Oct 28 20:22:40 2006
From: jeremy.kloth at 4suite.org (Jeremy Kloth)
Date: Sat, 28 Oct 2006 12:22:40 -0600
Subject: [Catalog-sig] Add requirements to XMLRPC?
In-Reply-To: <200610281212.54391.jeremy.kloth@4suite.org>
References: <200610260051.36324.timcera@earthlink.net>
<200610270743.38255.richardjones@optushome.com.au>
<200610281212.54391.jeremy.kloth@4suite.org>
Message-ID: <200610281222.40239.jeremy.kloth@4suite.org>
On Saturday 28 October 2006 12:12 pm, Jeremy Kloth wrote:
> On Thursday 26 October 2006 3:43 pm, Richard Jones wrote:
> > On Thursday 26 October 2006 14:51, Tim Cera wrote:
> > > It would really be nice for a project that I am working on to get the
> > > package requirements from XMLRPC. What are the issues with adding the
> > > requirements to XMLPRC?
> >
> > Someone has to write the code. I check it, commit it and install it.
>
> The attached patch adds the 'requires', 'provides' and 'obsoletes' fields
> to the release_data() RPC method.
>
> Note this is untested, but copied from webui.py where is builds the
> PKG-INFO.
Maybe this time I'll attach the patch, not the whole file. :)
--
Jeremy Kloth
http://4suite.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rpc.patch
Type: text/x-diff
Size: 622 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061028/356f270a/attachment.bin
From dangoor at gmail.com Sun Oct 29 19:21:21 2006
From: dangoor at gmail.com (Kevin Dangoor)
Date: Sun, 29 Oct 2006 13:21:21 -0500
Subject: [Catalog-sig] cheeseshop down?
Message-ID: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com>
It looks like the cheeseshop is down. "Bad Gateway" error from the
python.org link and I can't get a connection at all on
cheeseshop.python.org
Kevin
From jeremy.kloth at 4suite.org Mon Oct 30 07:00:08 2006
From: jeremy.kloth at 4suite.org (Jeremy Kloth)
Date: Sun, 29 Oct 2006 23:00:08 -0700
Subject: [Catalog-sig] Add requirements to XMLRPC?
In-Reply-To: <200610281222.40239.jeremy.kloth@4suite.org>
References: <200610260051.36324.timcera@earthlink.net>
<200610281212.54391.jeremy.kloth@4suite.org>
<200610281222.40239.jeremy.kloth@4suite.org>
Message-ID: <200610292300.08918.jeremy.kloth@4suite.org>
On Saturday 28 October 2006 12:22 pm, Jeremy Kloth wrote:
> > On Thursday 26 October 2006 3:43 pm, Richard Jones wrote:
> > > Someone has to write the code. I check it, commit it and install it.
This patch adds the information available for the 'files' action to the RPC
release_urls() command.
--
Jeremy Kloth
http://4suite.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rpc.patch
Type: text/x-diff
Size: 782 bytes
Desc: not available
Url : http://mail.python.org/pipermail/catalog-sig/attachments/20061029/a0c7ae45/attachment.bin
From richardjones at optushome.com.au Mon Oct 30 08:21:31 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 30 Oct 2006 18:21:31 +1100
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com>
Message-ID: <200610301821.31281.richardjones@optushome.com.au>
On Monday 30 October 2006 05:21, Kevin Dangoor wrote:
> It looks like the cheeseshop is down. "Bad Gateway" error from the
> python.org link and I can't get a connection at all on
> cheeseshop.python.org
Appeared to be a temporary thing (it's back). I don't have time to look into
what might have happened right now though.
Richard
From jim at zope.com Mon Oct 30 12:08:23 2006
From: jim at zope.com (Jim Fulton)
Date: Mon, 30 Oct 2006 06:08:23 -0500
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <200610301821.31281.richardjones@optushome.com.au>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com>
<200610301821.31281.richardjones@optushome.com.au>
Message-ID: <4545DD27.9060502@zope.com>
Richard Jones wrote:
> On Monday 30 October 2006 05:21, Kevin Dangoor wrote:
>> It looks like the cheeseshop is down. "Bad Gateway" error from the
>> python.org link and I can't get a connection at all on
>> cheeseshop.python.org
>
> Appeared to be a temporary thing (it's back).
Did it recover on it's own? Or did someone do something
to bring it back?
> I don't have time to look into
> what might have happened right now though.
This is a fairly serious problem. In the past, you questioned
why anyone would want to implement their own package indexes.
A major reason is reliability. Personally, I'd prefer that there
be only one package index, but it has to be very available, at least
the parts that setuptools uses. PyPI is great, but it needs to
be much more available.
I'll refrain, for now, from making suggestions, although I really want to,
because I haven't had a chance to study the current implementation.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From amk at amk.ca Mon Oct 30 13:59:19 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 30 Oct 2006 07:59:19 -0500
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <4545DD27.9060502@zope.com>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com>
<200610301821.31281.richardjones@optushome.com.au>
<4545DD27.9060502@zope.com>
Message-ID: <20061030125919.GA4868@localhost.localdomain>
On Mon, Oct 30, 2006 at 06:08:23AM -0500, Jim Fulton wrote:
> Did it recover on it's own? Or did someone do something
> to bring it back?
Someone edited the web server's config file and broke it; Anthony
Baxter fixed it later.
--amk
From jim at zope.com Mon Oct 30 15:54:38 2006
From: jim at zope.com (Jim Fulton)
Date: Mon, 30 Oct 2006 09:54:38 -0500
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <20061030125919.GA4868@localhost.localdomain>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au> <4545DD27.9060502@zope.com>
<20061030125919.GA4868@localhost.localdomain>
Message-ID: <4546122E.2080408@zope.com>
A.M. Kuchling wrote:
> On Mon, Oct 30, 2006 at 06:08:23AM -0500, Jim Fulton wrote:
>> Did it recover on it's own? Or did someone do something
>> to bring it back?
>
> Someone edited the web server's config file and broke it; Anthony
> Baxter fixed it later.
Ah, cool. Thanks.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From martin at v.loewis.de Mon Oct 30 22:08:20 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 30 Oct 2006 22:08:20 +0100
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <4545DD27.9060502@zope.com>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au>
<4545DD27.9060502@zope.com>
Message-ID: <454669C4.6080206@v.loewis.de>
Jim Fulton schrieb:
>> I don't have time to look into what might have happened right now
>> though.
>
> This is a fairly serious problem. In the past, you questioned why
> anyone would want to implement their own package indexes.
And I would still question that. I can understand why people are
unhappy if the service isn't reliable; I can't understand why they
then think they can do better their own.
> I'll refrain, for now, from making suggestions, although I really
> want to, because I haven't had a chance to study the current
> implementation.
I hope the suggestion won't be "let's build another package database".
Other suggestions certainly would be welcome. As we found in this
specific case, it really was an admin error, by some unknown admin.
Would your suggestion have such error from happening?
Regards,
Martin
From jim at zope.com Mon Oct 30 22:29:42 2006
From: jim at zope.com (Jim Fulton)
Date: Mon, 30 Oct 2006 16:29:42 -0500
Subject: [Catalog-sig] cheeseshop down?
In-Reply-To: <454669C4.6080206@v.loewis.de>
References: <708D8DD6-9A5C-40A6-A3C5-16D407EE54B0@gmail.com> <200610301821.31281.richardjones@optushome.com.au>
<4545DD27.9060502@zope.com> <454669C4.6080206@v.loewis.de>
Message-ID: <45466EC6.2020606@zope.com>
Martin v. L?wis wrote:
> Jim Fulton schrieb:
>>> I don't have time to look into what might have happened right now
>>> though.
>> This is a fairly serious problem. In the past, you questioned why
>> anyone would want to implement their own package indexes.
>
> And I would still question that. I can understand why people are
> unhappy if the service isn't reliable; I can't understand why they
> then think they can do better their own.
Because the current one is down or unresponsive more often then
many of us will be able to deal with.
>> I'll refrain, for now, from making suggestions, although I really
>> want to, because I haven't had a chance to study the current
>> implementation.
>
> I hope the suggestion won't be "let's build another package database".
That wasn't what I had in mind, no.
> Other suggestions certainly would be welcome. As we found in this
> specific case, it really was an admin error, by some unknown admin.
I guess the fact that the admin who caused the problem is unknown
is a problem in itself.
> Would your suggestion have such error from happening?
I don't know. It might have, depending on what it was. :)
I wouldn't make a suggestion without finding out more about the
implementation. It has been suggested in the past that we might want
to create mirrors. That *would* have helped in this situation.
Please understand that I'm not trying to criticize anyone.
PyPI is a great system, when it works.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org