From zooko at zooko.com Wed Mar 5 22:24:48 2008
From: zooko at zooko.com (zooko)
Date: Wed, 5 Mar 2008 14:24:48 -0700
Subject: [Catalog-sig] requested Trove Classifier: "Framework :: Setuptools
Plugin"
Message-ID: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
Folks:
Per this thread on distutils-sig, it would be useful to have a
classifier with which to identify setuptools plugins:
http://mail.python.org/pipermail/distutils-sig/2008-March/008847.html
What needs to be done to make this possible?
Regards,
Zooko
From richardjones at optushome.com.au Sun Mar 9 01:03:13 2008
From: richardjones at optushome.com.au (Richard Jones)
Date: Sun, 9 Mar 2008 11:03:13 +1100
Subject: [Catalog-sig] requested Trove Classifier: "Framework ::
Setuptools Plugin"
In-Reply-To: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
Message-ID: <200803091103.13519.richardjones@optushome.com.au>
On Thu, 6 Mar 2008, zooko wrote:
> Folks:
>
> Per this thread on distutils-sig, it would be useful to have a
> classifier with which to identify setuptools plugins:
>
> http://mail.python.org/pipermail/distutils-sig/2008-March/008847.html
>
> What needs to be done to make this possible?
Sorry for the slow response, I've been busy.
I would prefer to see the classifier named similarly to the other frameworks -
ie. no "Plugin". Otherwise I have no objection.
Richard
From pje at telecommunity.com Sun Mar 9 02:13:25 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 08 Mar 2008 20:13:25 -0500
Subject: [Catalog-sig] requested Trove Classifier: "Framework ::
Setuptools Plugin"
In-Reply-To: <200803091103.13519.richardjones@optushome.com.au>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
<200803091103.13519.richardjones@optushome.com.au>
Message-ID: <20080309011307.E058B3A40AC@sparrow.telecommunity.com>
At 11:03 AM 3/9/2008 +1100, Richard Jones wrote:
>On Thu, 6 Mar 2008, zooko wrote:
> > Folks:
> >
> > Per this thread on distutils-sig, it would be useful to have a
> > classifier with which to identify setuptools plugins:
> >
> > http://mail.python.org/pipermail/distutils-sig/2008-March/008847.html
> >
> > What needs to be done to make this possible?
>
>Sorry for the slow response, I've been busy.
>
>I would prefer to see the classifier named similarly to the other
>frameworks -
>ie. no "Plugin". Otherwise I have no objection.
My concern is that people will use the tag merely if their code
*uses* setuptools or is compatible with it, because there isn't any
other guidance about how it's meant to be used. Would you be alright
with say, "Setuptools Extensions"?
From zooko at zooko.com Sun Mar 9 03:08:02 2008
From: zooko at zooko.com (zooko)
Date: Sat, 8 Mar 2008 19:08:02 -0700
Subject: [Catalog-sig] requested Trove Classifier: "Framework ::
Setuptools Plugin"
In-Reply-To: <200803091103.13519.richardjones@optushome.com.au>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
<200803091103.13519.richardjones@optushome.com.au>
Message-ID: <9F191149-1E82-43B2-B935-53967A905030@zooko.com>
> I would prefer to see the classifier named similarly to the other
> frameworks -
> ie. no "Plugin". Otherwise I have no objection.
Well, we should be careful distinguish between a package which is a
setuptools plugin -- i.e. a unit of code which can used to extend the
setuptools build tool -- from a package which is built using setuptools.
The classifier I'm looking for is a way to find packages of the
former kind.
It was Philip J. Eby's suggestion, originally, to name it "setuptools
plugin" in order to make it clear that this classifier wasn't
appropriate for any package which is built using setuptools.
Regards,
Zooko
From richardjones at optushome.com.au Sun Mar 9 03:57:19 2008
From: richardjones at optushome.com.au (Richard Jones)
Date: Sun, 9 Mar 2008 13:57:19 +1100
Subject: [Catalog-sig] requested Trove Classifier: "Framework ::
Setuptools Plugin"
In-Reply-To: <9F191149-1E82-43B2-B935-53967A905030@zooko.com>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
<200803091103.13519.richardjones@optushome.com.au>
<9F191149-1E82-43B2-B935-53967A905030@zooko.com>
Message-ID: <200803091357.19180.richardjones@optushome.com.au>
On Sun, 9 Mar 2008, you wrote:
> > I would prefer to see the classifier named similarly to the other
> > frameworks -
> > ie. no "Plugin". Otherwise I have no objection.
>
> Well, we should be careful distinguish between a package which is a
> setuptools plugin -- i.e. a unit of code which can used to extend the
> setuptools build tool -- from a package which is built using setuptools.
>
> The classifier I'm looking for is a way to find packages of the
> former kind.
>
> It was Philip J. Eby's suggestion, originally, to name it "setuptools
> plugin" in order to make it clear that this classifier wasn't
> appropriate for any package which is built using setuptools.
OK, I've added "Framework :: Setuptools Plugin"
Richard
From zooko at zooko.com Sun Mar 9 05:01:04 2008
From: zooko at zooko.com (zooko)
Date: Sat, 8 Mar 2008 21:01:04 -0700
Subject: [Catalog-sig] requested Trove Classifier: "Framework ::
Setuptools Plugin"
In-Reply-To: <200803091357.19180.richardjones@optushome.com.au>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
<200803091103.13519.richardjones@optushome.com.au>
<9F191149-1E82-43B2-B935-53967A905030@zooko.com>
<200803091357.19180.richardjones@optushome.com.au>
Message-ID: <9EE5F72C-3A55-460D-B1E4-1856D69E97BD@zooko.com>
On Mar 8, 2008, at 7:57 PM, Richard Jones wrote:
> OK, I've added "Framework :: Setuptools Plugin"
Hooray!
I've added the first three setuptools plugins to be marked with that
classifier. This search shows all packages on pypi which have the
"Framework :: Setuptools Plugin" classifier:
http://pypi.python.org/pypi?:action=browse&c=524
Regards,
Zooko
From mereandor at gmail.com Fri Mar 21 14:24:16 2008
From: mereandor at gmail.com (mereandor at gmail.com)
Date: Fri, 21 Mar 2008 14:24:16 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
Message-ID: <200803211424.16665.mereandor@gmail.com>
hi!
I'm currently planning to integrate the PyPI into the paludis package management system (http://paludis.pioto.org).
I did some research on which interfaces are available to retrieve data from the index. The two most promising interfaces are the XML-RPCs and http://pypi.python.org/simple. But both lack a compact index (information that I can download with only one request) that contains at least the package names and the available versions.
A syncable directory (rsync, svn, ... you name it) that contains the PKGINFO file for each package/version combination would be even better, since this would reduce the communication with the server (and thus the server load) to a minimum.
Would it be possibe to extend the existing interfaces so that they fit this needs?
I'll gladly help working on those new interfaces if help is wanted.
Thanks in advance for any suggestions.
Greetings,
Roman
From martin at v.loewis.de Fri Mar 21 19:45:22 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 21 Mar 2008 19:45:22 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
In-Reply-To: <200803211424.16665.mereandor@gmail.com>
References: <200803211424.16665.mereandor@gmail.com>
Message-ID: <47E40242.90703@v.loewis.de>
> I did some research on which interfaces are available to retrieve
> data from the index. The two most promising interfaces are the
> XML-RPCs and http://pypi.python.org/simple. But both lack a compact
> index (information that I can download with only one request) that
> contains at least the package names and the available versions.
I can't quite understand where the need for a *single* request comes
from. The information is surely available by use of multiple requests.
> Would it be possibe to extend the existing interfaces so that they
> fit this needs?
>
> I'll gladly help working on those new interfaces if help is wanted.
> Thanks in advance for any suggestions.
Indeed, such an interface could only become reality by means of
somebody contributing it. However, before you start doing so:
have you considered alternatives, such as using multiple requests,
along with incremental updates?
If the interface were available, how would you use it? (e.g. how
often, and what for)
Regards,
Martin
From mereandor at gmail.com Fri Mar 21 20:29:37 2008
From: mereandor at gmail.com (mereandor at gmail.com)
Date: Fri, 21 Mar 2008 20:29:37 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
In-Reply-To: <47E40242.90703@v.loewis.de>
References: <200803211424.16665.mereandor@gmail.com>
<47E40242.90703@v.loewis.de>
Message-ID: <200803212029.38419.mereandor@gmail.com>
Am Freitag, 21. M?rz 2008 19:45:22 schrieb Martin v. L?wis:
> > I did some research on which interfaces are available to retrieve
> > data from the index. The two most promising interfaces are the
> > XML-RPCs and http://pypi.python.org/simple. But both lack a compact
> > index (information that I can download with only one request) that
> > contains at least the package names and the available versions.
>
> I can't quite understand where the need for a *single* request comes
> from. The information is surely available by use of multiple requests.
>
> > Would it be possibe to extend the existing interfaces so that they
> > fit this needs?
> >
> > I'll gladly help working on those new interfaces if help is wanted.
> > Thanks in advance for any suggestions.
>
> Indeed, such an interface could only become reality by means of
> somebody contributing it. However, before you start doing so:
> have you considered alternatives, such as using multiple requests,
> along with incremental updates?
>
> If the interface were available, how would you use it? (e.g. how
> often, and what for)
>
> Regards,
> Martin
>
>
The need for a single request is basically a matter of efficiency as shown below in the use case.
Usually, if all the metadata is readily available (ie. without downloading all packages) the package manager periodically (at most once a day - typically once a week or less) synchronises all "repositories" (containing all necessary package metadata). This means the metadata is stored on the user's disk for further use.
Considering the server load, synchronising seems not to be an option at the moment (IMHO) since this would mean one request for each package in the repository.
The problems with the currently available interfaces are best shown by an use case.
Let's say the user want's to update all installed packages:
Without the ability to sync (as described above) this would mean:
1. requesting one page per installed package to determine which versions are available.
2. downloading the new versions
3. resolve the dependencies and starting at step 1 for every new dependency
4. install the packages
This results in m+n+2*q requests to the server (m = number of installed packages, n = number of updateable packages, q = number of new dependencies). Typically m is by far the largest number.
With the ability to sync this would be:
1. syncing the repository
2. determining newer versions and resolving their dependencies
3. download and install the list of packages
This results in 1+n+q requests.
This shows us that it would vastly improve the situation if at least the version was available in a similar way to the simple-index http://pypi.python.org/simple or the corresponding xml-rpc.
I hope this clears things up a bit.
Regards,
Roman
From martin at v.loewis.de Fri Mar 21 21:39:01 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 21 Mar 2008 21:39:01 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
In-Reply-To: <200803212029.38419.mereandor@gmail.com>
References: <200803211424.16665.mereandor@gmail.com> <47E40242.90703@v.loewis.de>
<200803212029.38419.mereandor@gmail.com>
Message-ID: <47E41CE5.9080703@v.loewis.de>
> Usually, if all the metadata is readily available (ie. without
> downloading all packages) the package manager periodically (at most
> once a day - typically once a week or less) synchronises all
> "repositories" (containing all necessary package metadata). This
> means the metadata is stored on the user's disk for further use.
> Considering the server load, synchronising seems not to be an option
> at the moment (IMHO) since this would mean one request for each
> package in the repository.
Thanks for the explicit scenario; it is fortunately not the
case that you would have to download all package and version
information each weak.
Instead, I recommend to use the changelog method, to find out
all changes since the last week.
> I hope this clears things up a bit.
Indeed it does. For synchronization, you shouldn't at all
consider downloading all information again repeatedly. Instead,
try using incremental update methods as much as possible.
With the changelog method, daily updates become much more
reasonable; some sites query the changelog every minute
(and it is then typically empty).
Regards,
Martin
From mereandor at gmail.com Fri Mar 21 23:49:27 2008
From: mereandor at gmail.com (mereandor at gmail.com)
Date: Fri, 21 Mar 2008 23:49:27 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
In-Reply-To: <47E41CE5.9080703@v.loewis.de>
References: <200803211424.16665.mereandor@gmail.com>
<200803212029.38419.mereandor@gmail.com>
<47E41CE5.9080703@v.loewis.de>
Message-ID: <200803212349.28717.mereandor@gmail.com>
Am Freitag, 21. M?rz 2008 21:39:01 schrieb Martin v. L?wis:
> Thanks for the explicit scenario; it is fortunately not the
> case that you would have to download all package and version
> information each weak.
>
> Instead, I recommend to use the changelog method, to find out
> all changes since the last week.
>
> > I hope this clears things up a bit.
>
> Indeed it does. For synchronization, you shouldn't at all
> consider downloading all information again repeatedly. Instead,
> try using incremental update methods as much as possible.
>
> With the changelog method, daily updates become much more
> reasonable; some sites query the changelog every minute
> (and it is then typically empty).
>
> Regards,
> Martin
I would still prefer the full dump for several reasons:
* changelog would only give me the information what package has changed, I still would have to use release_data to get the new data
* the benefits of changelog decrease the longer the update interval is (and it's typically longer than a day; sometimes several months - depends on the user)
* getting a dump of all data is not that expensive as search({'name':''}) takes about 5 seconds including the time for the data-transfer and displaying all the data
* the problem of the first sync remains (at least once for each system installation/user)
Incremental updates add a certain amount of complexity:
* I need to be sure that the data is sane before the sync so that I can be sure that it is sane after it
* I need to know the exact time of the last sync
* The algorithm for an incremental sync is considerably more complex
Regards,
Roman
From martin at v.loewis.de Sat Mar 22 00:39:30 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 00:39:30 +0100
Subject: [Catalog-sig] Request: Interface to index of package-metadata
In-Reply-To: <200803212349.28717.mereandor@gmail.com>
References: <200803211424.16665.mereandor@gmail.com> <200803212029.38419.mereandor@gmail.com> <47E41CE5.9080703@v.loewis.de>
<200803212349.28717.mereandor@gmail.com>
Message-ID: <47E44732.7010704@v.loewis.de>
> * changelog would only give me the information what package has
> changed, I still would have to use release_data to get the new data
I'm beginning to lose track. I thought you only wanted package names
and versions; now you seem to also want additional data?
> * the benefits of changelog decrease the longer the update interval
> is (and it's typically longer than a day; sometimes several months -
> depends on the user)
Yes, but only slightly so. Even after several months, the changelog
will be much smaller than the full database.
In any case, I don't think I can find time to work on this within
the next year or so, even if I wanted to. So you will have to
provide patches if you really think you need it.
Regards,
Martin