[Catalog-sig] PyPI down again...

[M.-A. Lemburg]

Mathieu Leduc-Hamel wrote:
>>
>> Agreed, that's why I think it would be useful to simply put
>> all meta data into a SQLite database file and ship that as
>> static file as well. Local clients could then download the
>> database file (probably only a few MB) and work on it locally.
>>
> I don't think it would be easy to do that right now since the database
> store more informations than only the metadata of the all packages, you
> don't wanna give all the informations about users accounts by example...

PyPI uses PostgreSQL as database backend, so the SQLite database
file would be a (partial) copy of that database. Of course, it
would have to only contain meta-data that is also visible via
the web GUI.

> And, I don't understand how it can be perform with always updated
> informations like ones on the pypi website, your database is always updated,
> than it's not possible to have a completely updated one...

True, but I think only very few users are really after real-time data
from PyPI. Those can use the true RPC interfaces.

For the others, a static copy created and updated every 10-20 minutes
or so, is likely good enough.

Anyway, it's an idea based on the 80/20 rule :-)

> Search might not by the big deal if the data and a good cache interface is
> implemented, the number of parallel connexion is something else...
>
>> This would also make searches in PyPI a lot faster... not only
>> because searches could be done locally, but also because the
>> server wouldn't have to handle the load of those searches from
>> hundreds of clients.
>>
>>> But, definitively, yes, I'll consider that.
>>
>> Thanks !

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 14 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                34 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/