On 11.10.2008 18:24 Uhr, Tarek Ziadé wrote:

Day 1 is almost over,

We worked on two elements so far: the mirroring thing, and the terminology one

these are early drafts,

http://wiki.python.org/moin/PythonPackagingTerminology http://wiki.python.org/moin/PEP_374

I think we should also investigate how other repositories like CPAN or the Ruby world deals with mirroring. A notification mechanism appears fragile to me. I believe that the mirrors should remain "dumb" in order to keep the complete system simple and solid - less moving parts are better than more. I really have a bad feeling about the approach as described in your PEP. Andreas

On 11.10.2008 21:56 Uhr, Tarek Ziadé wrote:

On Sat, Oct 11, 2008 at 7:40 PM, Andreas Jung wrote:

...

On 11.10.2008 18:24 Uhr, Tarek Ziadé wrote:

...

Day 1 is almost over,

We worked on two elements so far: the mirroring thing, and the terminology one

these are early drafts,

http://wiki.python.org/moin/PythonPackagingTerminology http://wiki.python.org/moin/PEP_374

I think we should also investigate how other repositories like CPAN or the Ruby world deals with mirroring.

or rather linux

http://www.mail-archive.com/distutils-sig@python.org/msg05791.html

...

A notification mechanism appears fragile to me. I believe that the mirrors should remain "dumb" in order to keep the complete system simple and solid -`

Well at some point you need a protocol, otherwise your mirror is this "dumb" thing we cannot garantee to be a reliable thing.

...

less moving parts are better than more. I really have a bad feeling about the approach as described in your PEP.

well, I think we need more than a rsync script at some point.

The question is if you want a push or pull mechanism. z3c.pypimirror implements the pull implementation and performs an incremental update in the sense of rsync in a reliable way. Andreas

On Sun, Oct 12, 2008 at 3:57 AM, Andreas Jung wrote:

...

well, I think we need more than a rsync script at some point.

The question is if you want a push or pull mechanism. z3c.pypimirror implements the pull implementation and performs an incremental update in the sense of rsync in a reliable way.

You don't want a push mechanism, but you want a way to list the mirrors at PyPI, their states, and know if they are good mirrors or not. That is what a ping mechanism provides. So the point is not about being able to provide a reliable "copy of files program", I can use for that a "wget --mirror" or "rsync -r" and I don't need to write a program for this. On the other hand maybe the XML-RPC thing is a little heavy but we sure need to know how "fresh" a mirror is. Maybe if the last-modified header is maintained on the mirror this could be enough for PyPI and other third-party application to know about it. -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

We removed the RPC thing, and added a freshness date principle, to make thing simpler. see http://wiki.python.org/moin/PEP_374 On Sat, Oct 11, 2008 at 7:40 PM, Andreas Jung wrote:

On 11.10.2008 18:24 Uhr, Tarek Ziadé wrote:

...

Day 1 is almost over,

We worked on two elements so far: the mirroring thing, and the terminology one

these are early drafts,

http://wiki.python.org/moin/PythonPackagingTerminology http://wiki.python.org/moin/PEP_374

I think we should also investigate how other repositories like CPAN or the Ruby world deals with mirroring. A notification mechanism appears fragile to me. I believe that the mirrors should remain "dumb" in order to keep the complete system simple and solid - less moving parts are better than more. I really have a bad feeling about the approach as described in your PEP.

Andreas

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

Can you speak more on incremental update ?

What would you like to know? incremental update should be very easy to implement for a mirror tool, with no additional changes to PyPI. Regards, Martin

On Sun, Oct 12, 2008 at 1:50 PM, "Martin v. Löwis" wrote:

...

we use wget --mirror, and it is working quite good, an upgrade is around 10 minutes IIRC but I think rsync is the best way to go for a mirror.

I disagree. The best way to do the mirroring is with the specific protocol explicitly designed to do mirroring, namely to look at the changelog.

PLEASE DONT USE RSYNC OR WGET TO MIRROR PYPI. PLEASE!

could you explain why is that a problem ?

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

could you explain why is that a problem ?

It produces significant load on the master. If you look at the web stats, e.g for September: http://pypi.python.org/webstats/usage_200809.html you see that there had been 5671455 hits, or 41%, of accesses through wget. The problem with wget mirroring is that it needs to read *many* pages, to find out the *few* changes. FWIW, it's also the case that 4940769 hits originate from France. Could it be that you are alone responsible for 40% of the traffic on PyPI? Regards, Martin

On Sun, Oct 12, 2008 at 2:12 PM, "Martin v. Löwis" wrote:

...

could you explain why is that a problem ?

It produces significant load on the master. If you look at the web stats, e.g for September:

http://pypi.python.org/webstats/usage_200809.html

you see that there had been 5671455 hits, or 41%, of accesses through wget.

The problem with wget mirroring is that it needs to read *many* pages, to find out the *few* changes.

Sure,

FWIW, it's also the case that 4940769 hits originate from France. Could it be that you are alone responsible for 40% of the traffic on PyPI?

Yes, I am the only Python developer in France. That's me. Just kidding :) France has a lot of python/plone developers that triggers buildouts every day, so I am pretty sure the mirrors don't make the whole traffic in PyPI. we could probably do things better though. Here's my proposal: + see if we can locate the mirrors, so for instance, if i register a "Paris mirror" people will eventually go there because it is the nearest location for them. (à la CPAN) + create a new user agent for mirroring tools Regards Tarek

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

...

FWIW, it's also the case that 4940769 hits originate from France. Could it be that you are alone responsible for 40% of the traffic on PyPI?

Yes, I am the only Python developer in France. That's me.

Just kidding :)

France has a lot of python/plone developers that triggers buildouts every day, so I am pretty sure the mirrors don't make the whole traffic in PyPI.

Hmm. Yesterday, there were 199250 accesses to PyPI through wget. Of those, 169971 requests came from a single address (from Dedibox in France), 28966 requests from a second one (from Sakura in Japan). So it *is* wget mirrors that make the whole traffic in PyPI. Regards, Martin

On Sun, Oct 12, 2008 at 2:49 PM, "Martin v. Löwis" wrote:

...

...

FWIW, it's also the case that 4940769 hits originate from France. Could it be that you are alone responsible for 40% of the traffic on PyPI?

Yes, I am the only Python developer in France. That's me.

Just kidding :)

France has a lot of python/plone developers that triggers buildouts every day, so I am pretty sure the mirrors don't make the whole traffic in PyPI.

Hmm. Yesterday, there were 199250 accesses to PyPI through wget. Of those, 169971 requests came from a single address (from Dedibox in France), 28966 requests from a second one (from Sakura in Japan).

yes that is us, we have two mirror here, one smart one (proxy with the minimum amount on call over PyPI) and the wget one, the second one is the wget. but the mirror option use a file listing and a timestamping mecanism, so the file or pages are downloaded only if they have changed. Basically only headers are read. I'll shut it off anyway, we have the smart collective.proxy at work now, and will eventually switch to the wget one to the pypimirror one if we provide an "official" full mirror

So it *is* wget mirrors that make the whole traffic in PyPI.

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

On Sun, Oct 12, 2008 at 3:23 PM, "Martin v. Löwis" wrote:

...

But how do you want to display them ? Do you want to display the grand total on PyPI ?

Yes, exactly so.

...

In that cas each mirror should provide a counter page,

Why that? Shouldn't the mirrors then also display the grand total?

ok then we'll try to think about some solutions for that,

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

how do you collect them in PyPI ? via Apache logs ?

Exactly. It's in tools/apache_count.py Regards, Martin

Our z3c.pypimirror already performs an incremental update based on the information available from the index.html page of the simple index and the available md5 hashes. Works like a charm...

So how does it find out when a release gets made? Regards, Martin

...

...

Our z3c.pypimirror already performs an incremental update based on the information available from the index.html page of the simple index and the available md5 hashes. Works like a charm...

So how does it find out when a release gets made?

What do you mean by that?

If you only look at http://pypi.python.org/simple/ then you have no way of find out out what changed. So "the information available from the index.html page of the simple index" is not actually suitable for building incremental mirroring. What you describe is not possible. I just looked at the z3c.pypimirror source, and found that it isn't really incremental: Whenever it mirrors, it looks at *all* index.html pages, of each an every package (all 4900 of them, except when you restrict the mirror). It then only downloads any new files that may have been added/deleted, and it *is* incremental wrt. files. IIUC, it is *not* incremental wrt. the package index itself. Please correct me if I'm wrong (and please correct z3c.pypimirror if I'm not :-) Can you please set a specific useragent header, to find out what amount of traffic pypimirror produces? Currently, urllib accounts for 17% of the requests, excluding requests made through urllib by setuptools (which is a separate 18%). It's probably not all of them through pypimirror, but of the 64626 requests made through urllib yesterday, 41671 originated from zopyx.com. For real incremental mirroring, you should retrieve the changelog, and access only those package pages that have actually changed since the last time you ran the mirror (successfully). Regards, Martin

On Sun, Oct 12, 2008 at 4:32 PM, "Martin v. Löwis" wrote:

...

how do you collect them in PyPI ? via Apache logs ?

Exactly. It's in tools/apache_count.py

How often do you run it ? I guess a daily update is enough for the grand total ? Anyway, so the mirrors should be able to reuse this script for their own internal count as well, and PyPI would need to provide a way for the mirror to report them, and to get back the grand count. but i wouldn't want to make apache mandatory for the mirrors. What about this: 1/ each mirror maintain simple text-based stats pages, with the local count, reachable from an url (/local_stats) 2/ PyPI modifies its script so it injects its apache count + the registered mirrors local counts 3/ PyPI maintains a simple text stats page, with the grand count (/stats) one stat page represents one day, and the stats are presented in folders that represents the year and the month So the stats from october the 11th will be reachable at: .../local_stats/2008/10/11 The stat page can referer to the packages using a PACKAGE_NAME/FILE = HITS syntax: iw.recipe.fss/iw.recipe.fss-0.2.1.tar.gz = 123 foo.bar/foo.bar-0.3.tar.gz = 12 ... This is a fairly simple structure any mirroring tool can create, and we could provide a simple python script that generates it from the Apache logs, Regards Tarek

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

2008/10/13 "Martin v. Löwis"

...

How often do you run it ? I guess a daily update is enough for the grand total ?

I think it still runs daily. There was one complaint about that, but the user could accept that as a policy after understanding what happened (he thought the feature was broken as there was no immediate update).

...

1/ each mirror maintain simple text-based stats pages, with the local count, reachable from an url (/local_stats) 2/ PyPI modifies its script so it injects its apache count + the registered mirrors local counts 3/ PyPI maintains a simple text stats page, with the grand count (/stats)

Sounds fine to me. Expect that to become a long file, though, with one line per file (roughly 20000 files with at least one download).

Maybe we could use one subfolder per alphabet letter, like what is done in packages/ at PyPI that would lower it down to roughly 1000 items per pages,

...

one stat page represents one day, and the stats are presented in folders that represents the year and the month

I wonder whether it might be easier to have a single file, with the totals for that server.

You would need to specify a timestamp for each single download though, to make sure PyPI knows which hits to count, depending on the last date it checked the mirror. if we have 1000 downloads per day, that's a huge file after a while

...

iw.recipe.fss/iw.recipe.fss-0.2.1.tar.gz = 123 foo.bar/foo.bar-0.3.tar.gz = 12

I would drop the "=" in that syntax.

Ok I'll upgrade the proposal, reflecting these infos

Regards, Martin _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Tarek Ziadé - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une société du groupe Alter Way

On Mon, Oct 13, 2008 at 6:16 PM, "Martin v. Löwis" wrote:

...

Maybe we could use one subfolder per alphabet letter,

Would that simplify anything?

PyPI uses one directory per letter to reduce the number of files in a single directory, in case ext3 doesn't deal with large directories well. For the stats, the "large directories" argument wouldn't count.

OTOH, if you do have separate pages per letter, the master server would still need to download all individual files. Having them split into chunks just increases the load, rather than reducing it.

Yes I thaught you were concerned by the size of that file, rather by the number of calls PyPI would need to perform.

...

You would need to specify a timestamp for each single download though, to make sure PyPI knows which hits to count, depending on the last date it checked the mirror.

No. It would just compute the grand total from scratch each time.

ok OTHO you would lose an interesting info: how downloads evolve in time. As a packager, I can see some interesting use cases. For example when foo 2.0 gets out, I can watch foo 1.0 downloads decrease and foo 2.0 raise. (if not make sure i have promoted 2.0 correctly) People would be able to generate interesting statistics tools from there. This would be possible of course only if PyPI provides the same timestamped pages for the grand total. This leads to another point we did not discuss yet: it would be interesting to keep the user-agent info in the mirrors, and make sure all automatic-package-grabbing softwares out there have there own user agent id For instance, knowing that 90% of the downloads of a given package where done by zc.buildout is interesting. IIRC, we cannot know it right now, and I could work on zc.buildout side for that, because it uses the setuptools user agent id Regards Tarek

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

On Tue, Oct 14, 2008 at 12:59 AM, "Martin v. Löwis" wrote:

...

OTHO you would lose an interesting info: how downloads evolve in time.

Users interested in that could produce that information themselves, though: they query the download stats once a day, and compute the first derivative.

ok

...

For instance, knowing that 90% of the downloads of a given package where done by zc.buildout is interesting. IIRC, we cannot know it right now, and I could work on zc.buildout side for that, because it uses the setuptools user agent id

In principle, it would be fine with me to track that, as long as I don't need to preserve the the complete log files.

So would the stats file then be of the form

package,filename,useragent,count

? (commas in the useragent replaced with semicolons)

sounds good, i'll write that down in the proposal as well last points I can think of, that we have discussed at the sprint: - are non open source licensed packages alllowed at PyPI ? - wouldn't it make sense for open source package to force a sdist upload before any other kind of distribution (this is a feature claimed by many people in fact, as binary distribution obsfuscate things and make it hard to install if it's not the same version, and if it was not intended by the packager) Regards

Regards, Martin

-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

- are non open source licensed packages alllowed at PyPI ?

Sure! There is no censorship applied in PyPI, except for content completely unrelated to Python. The only exception is when the package owner is unresponsive, and somebody else wants to take over the package. We need some procedure to formalize this case.

- wouldn't it make sense for open source package to force a sdist upload before any other kind of distribution (this is a feature claimed by many people in fact, as binary distribution obsfuscate things and make it hard to install if it's not the same version, and if it was not intended by the packager)

I don't want to assert quality control to the packages. If they don't upload anything, fine. If they upload broken packages, fine. If they supply invalid URLs, fine. If they mistype their email addresses, names, or licensing terms, fine. If they fail to provide source code, fine. Users should contact the authors and report problems with the registration if they find any. They sometimes mistake the PyPI tracker as tracking problems with the packages, but that happens rarely. Perhaps we can provide a form to submit a message to the package owner, to be used when everything else fails (such form would require a PyPI account for the sender, too). Regards, Martin

Martin v. Löwis wrote:

Hmm. Yesterday, there were 199250 accesses to PyPI through wget. Of those, 169971 requests came from a single address (from Dedibox in France), 28966 requests from a second one (from Sakura in Japan).

So it *is* wget mirrors that make the whole traffic in PyPI.

If it were me, I'd just IP firewall the offendors. There's not need for this kind of behaviour if there's an acceptable mirror protocol available... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

...

If it were me, I'd just IP firewall the offendors. There's not need for this kind of behaviour if there's an acceptable mirror protocol available...

Well not yet... but the PEP should be finished sometimes this week,

There was an acceptable mirror protocol available for more than a year now, irrespective of any PEP you are working on. Claiming that only with the PEP, true mirroring becomes possible, disregards the work that Jim Fulton and I put into coming up with a workable solution. Regards, Martin

Josselin Mouette:

Unless you want to be able to follow existing standards.

http://standards.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html...

All you need for that is a category called "icons". I don't see anything there about putting different sizes of icon in different places on different platforms. Once you're in the icons location, the path from there to the flavour of icon you want is fixed by that spec. So the way you reference it is using something like. resource.read(module, "icon", "%s/48x48/apps/foo.png" % theme) -- Greg