To continue the discussion about a rewrite or a cleanup of the Pypi codebase, I'm from Montreal-Python usergroup and I'm say that yes at the first the current codebase of pypi seem to be very unclear and difficult to maintain.<div>
<br></div><div>But it's not an impossible mission and we are currently in the process of:</div><div><br></div><div>- Adding functional test. The test coverage is now around 40% percent.</div><div>- When we'll reach a more complete coverage, we want to replace the psycopg api by SQLAlchemy</div>
<div>- Replace many manual manipulation of the metadata by a more robust and straightforward way of dealing with (distutils2 might be the option there)</div><div><br></div><div>At first I was thinking about rewriting everything using the chishop project (an implementation of PyPi using django). But having the control of the code source and not dependent of any framework is maybe a better idea. </div>
<div><br></div><div>More than, despite the frequent outage, pypi is working today, then just a modernization of code base seem to be best idea.</div><div><br></div><div>By the wat, after a code review of tarek, a very useful thing might be to find a better way to deal and implement contributions coming from community. Right now Tarek is responsible of making the link between our effert and the work of Martin but we don't have any official public mirror of the source code and any roadmap.</div>
<div><br><div class="gmail_quote">On Mon, Jun 14, 2010 at 11:27 AM, M.-A. Lemburg <span dir="ltr"><<a href="mailto:mal@egenix.com">mal@egenix.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">Tarek Ziadé wrote:<br>
> On Sun, Jun 13, 2010 at 3:11 PM, M.-A. Lemburg <<a href="mailto:mal@egenix.com">mal@egenix.com</a>> wrote:<br>
> ...<br>
>><br>
>> We've had some private discussions about this, so I'm just<br>
>> going to summarize...<br>
>><br>
>> The idea here is not to override the mirror PEP ideas,<br>
>> but to use the existing PyPI installation and put the<br>
>> content needed for the most widely distributed package tool<br>
>> (currently setuptools and zc.buildout) on a content<br>
>> delivery network (CDN) in order to have it highly available<br>
>> on a managed edge network.<br>
><br>
> I think it overlaps a bit the PEP goal, which is to set up a network of mirrors,<br>
> and have them listed in the PyPI DNS so clients can switch from one mirror<br>
> to another.(and even do geoloc!)<br>
><br>
> Right now we already have "unofficial mirrors" and the idea of the PEP<br>
> would be to list them officially at PyPI and to have them collect the<br>
> stats so we cant count download hits.<br>
<br>
</div>Note that the CDN does not mirror the content of PyPI, it<br>
just takes care of delivering the requested data to the<br>
various edge servers and caching it there for a while.<br>
<br>
This is a different concept than that of a full mirror that<br>
doesn't work like a cache, but instead provides a fully<br>
functional standalone server.<br>
<br>
I still think that the concept of being able to mirror PyPI<br>
servers is a useful one.<br>
<div class="im"><br>
>> Amazon Cloudfront is such a CDN and has Python interfaces,<br>
>> hence the idea to use Cloudfront.<br>
>><br>
>> I asked for volunteers, because I didn't know enough about<br>
>> Amazon Cloudfront to write up a proposal and don't have<br>
>> the cycles available to implement such a setup myself.<br>
>><br>
>> In the meantime, I've done some research and now know<br>
>> enough to write a proposal for the PSF board to consider.<br>
>> If the board thinks it's a good idea, we'll need to<br>
>> pursue finding volunteers to implement it.<br>
><br>
> Well maybe this is the best path to follow right now, as it will be done faster,<br>
> without having to interact with much people to set it up, so it's a quick win.<br>
><br>
> But it will probably kill the mirroring protocol idea from the PEP in<br>
> the process,<br>
> which I think is superior in the long term since it provides a<br>
> standardized ground<br>
> for the community to set up mirrors independently from <a href="http://pypi.python.org" target="_blank">pypi.python.org</a>.<br>
<br>
</div>We'll have to see.<br>
<br>
Note that the CDN will only deal with the static data on PyPI,<br>
not the RPC or the web GUI access.<br>
<br>
Since static data is all that setuptools et al. currently use<br>
for fetching the data, we'll see an improved uptime for easy_install<br>
and esp. zc.buildout which by nature of their concepts rely on having<br>
a high availability of the PyPI static data resources.<br>
<br>
If, in the future, package tools start to rely on RPC for<br>
fetching data, the situation will shift towards needing full<br>
functional mirrors again.<br>
<br>
OTOH, we could also provide a snapshot copy of the database<br>
data in form of a SQLite database on the CDN for those tools<br>
to download and use locally... there are lot's of things<br>
package tools could do :-)<br>
<div class="im"><br>
--<br>
Marc-Andre Lemburg<br>
eGenix.com<br>
<br>
</div>Professional Python Services directly from the Source (#1, Jun 14 2010)<br>
<div class="im">>>> Python/Zope Consulting and Support ... <a href="http://www.egenix.com/" target="_blank">http://www.egenix.com/</a><br>
>>> mxODBC.Zope.Database.Adapter ... <a href="http://zope.egenix.com/" target="_blank">http://zope.egenix.com/</a><br>
>>> mxODBC, mxDateTime, mxTextTools ... <a href="http://python.egenix.com/" target="_blank">http://python.egenix.com/</a><br>
________________________________________________________________________<br>
</div>2010-07-19: EuroPython 2010, Birmingham, UK 34 days to go<br>
<div class="im"><br>
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::<br>
<br>
<br>
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48<br>
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg<br>
Registered at Amtsgericht Duesseldorf: HRB 46611<br>
<a href="http://www.egenix.com/company/contact/" target="_blank">http://www.egenix.com/company/contact/</a><br>
_______________________________________________<br>
</div><div><div></div><div class="h5">Catalog-SIG mailing list<br>
<a href="mailto:Catalog-SIG@python.org">Catalog-SIG@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/catalog-sig" target="_blank">http://mail.python.org/mailman/listinfo/catalog-sig</a><br>
</div></div></blockquote></div><br></div>