From pje at telecommunity.com Wed Jul 5 17:36:03 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Jul 2006 11:36:03 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <F33D84E3-E754-481C-B7E5-3AD620F5CF97@zope.com>
References: <D5993AEE-1BB6-46A9-95D0-1031BC5EF58A@zope.com>
<5.1.1.6.0.20060623162538.01eabc20@sparrow.telecommunity.com>
<D5993AEE-1BB6-46A9-95D0-1031BC5EF58A@zope.com>
Message-ID: <5.1.1.6.0.20060705112046.021c5720@sparrow.telecommunity.com>
At 04:04 AM 7/5/2006 -0400, Jim Fulton wrote:
>On Jun 23, 2006, at 4:51 PM, Jim Fulton wrote:
>...
>>
>>That's a lot of screen scraping. :)
>>
>>It would be good to capture this as part of the documentation IMO
>>
>>>I'm considering adding XML-RPC support to easy_install in 0.7,
>>>though. PyPI now has a nice XML-RPC API that is more responsive
>>>than the web UI, and it supports case-insensitive partial match
>>>searches, making it suitable for easy_install to query when a typed-
>>>in name doesn't exactly match the spelling of a PyPI entry.
>>
>>I think that would be much better.
>
>I just wanted to emphasize that I think this would be a good
>idea.
Patches welcome. :) Note that there should still be a fallback to the
screen scraping code in case of a problem with the XML-RPC, to allow people
to continue using static mirrors of PyPI or imitation PyPIs without needing
to support XML-RPC.
> I was just talking to Richard, and he pointed out that the
>current approach is a problem for him, because it means he can't
>evolve the pypi UI without risking breaking setuptools.
What I would suggest is creating a "microformat" for marking up web pages
with sniffable information. For example, adding rel="homepage" and
rel="download" to the links that go to those URLs.
In other words, invisible hints on the page to supplement the visible
information. Then, I could change easy_install to start using the
invisible hints, and drop the visible ones, freeing PyPI to evolve the UI
again.
While the XML-RPC API would be great, I still want easy_install to be able
to use a package index that's made from static files, and that requires
some kind of screen scraping. So, let's make it invisible scraping of a
documented format, so that anybody can use it, with whatever visual formats
they like.
Currently, easy_install gets most of its information from URLs; the only
actual scraping of visible data is of the title, the download MD5's, and
the table cells that identify links as being to the home page or download
URL (since it needs to specifically identify these in order to spider them).
The MD5 information dependency could be removed if PyPI included "#md5=..."
at the end of the download URLs; easy_install can see that information and
use it. The table cell checking could be removed by adding
'rel="easy_install"' or something like that to the spiderable links.
The title checking is used to distinguish pages that list multiple packages
from pages that list single packages. I don't have any ready ideas as to
how that could or should be represented in a semantic (as opposed to
visual) way. Your thoughts?
From richardjones at optusnet.com.au Thu Jul 6 11:09:23 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Thu, 06 Jul 2006 19:09:23 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060706/b5b6bc57/attachment.asc
From fdrake at gmail.com Thu Jul 6 15:58:23 2006
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 6 Jul 2006 09:58:23 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au>
References: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au>
Message-ID: <9cee7ab80607060658i31dfd34l4c1eeed7e5330129@mail.gmail.com>
On 7/6/06, richardjones at optusnet.com.au <richardjones at optusnet.com.au> wrote:
> Phillip J. Eby <pje at telecommunity.com> wrote:
> > Patches welcome. :) Note that there should still be a fallback to the
> > screen scraping code in case of a problem with the XML-RPC, to allow
> > people
> > to continue using static mirrors of PyPI or imitation PyPIs without
> > needing
> > to support XML-RPC.
>
> Why?
So we can easily have alternate or additional package repositories
implemented simply as (simple) HTML files and the downloadable
packages. We want to be able to have an internal repository that
plays the easy_install game without running our own PyPI.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From pje at telecommunity.com Thu Jul 6 16:03:56 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 10:03:56 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607060909.k6699NLd004491@mail12.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060706094755.02bf1d48@sparrow.telecommunity.com>
At 07:09 PM 7/6/2006 +1000, richardjones at optusnet.com.au wrote:
>Phillip J. Eby <pje at telecommunity.com> wrote:
> > Patches welcome. :) Note that there should still be a fallback to the
> > screen scraping code in case of a problem with the XML-RPC, to allow
> > people
> > to continue using static mirrors of PyPI or imitation PyPIs without
> > needing
> > to support XML-RPC.
>
>Why?
Why not? ;)
From easy_install's point of view, PyPI is just a place to find links for
a given package name. Preferably links that go directly to downloads, but
also to pages that might contain downloads. If someone doesn't want to use
PyPI as the source of download links, shouldn't they be able to use their
own, without having to implement an XML-RPC interface?
Actually, the question of "how do I get easy_install to use something other
than PyPI?" has been becoming somewhat of a FAQ recently. Well, two people
have asked about it in the last couple of weeks, anyway. And it would've
sucked to have to say "well, first you need an XML-RPC server..." :)
Nonetheless, there are various aspects of easy_install's behavior and
performance that could be significantly improved by using XML-RPC, so I
definitely want it to do that in 0.7. I'm just wary of removing the
existing behavior until it's clear that it's unnecessary for it to.
From richardjones at optusnet.com.au Thu Jul 6 17:03:33 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Fri, 07 Jul 2006 01:03:33 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607061503.k66F3X7S005870@mail20.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060707/d96eaade/attachment.pot
From richardjones at optusnet.com.au Thu Jul 6 17:39:12 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Fri, 07 Jul 2006 01:39:12 +1000
Subject: [Catalog-sig] PyPI XML-RPC changes
Message-ID: <200607061539.k66FdCV0007024@mail07.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060707/54e6525a/attachment.asc
From exarkun at divmod.com Thu Jul 6 17:43:48 2006
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 6 Jul 2006 11:43:48 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607061503.k66F3X7S005870@mail20.syd.optusnet.com.au>
Message-ID: <20060706154348.29014.1424458626.divmod.quotient.24966@ohm>
On Fri, 07 Jul 2006 01:03:33 +1000, richardjones at optusnet.com.au wrote:
>> Phillip J. Eby <pje at telecommunity.com> wrote:
>> Why not? ;)
>
>That was actually what I was afraid the reasoning was ;)
>
>I guess I just go all wobbly in the knees at the thought of having to maintain a "screen scraping" interface.
>
>Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about this very issue last night. I proposed that we detect the user-agent of the setuptools client, and in response send back really minimalist HTML (no surrounding page template). Probably overkill, but this may have been after we'd had beer :)
Making this explicit actually makes it a good idea.
http://host/pypiurl?format=simple
It doesn't even have to be html at this point, either. Return a nicely
structured xml or csv or whatever document. And then you can completely
omit setuptools hints from the actual markup, since setuptools won't
ever care about that.
Jean-Paul
From fdrake at gmail.com Thu Jul 6 17:44:16 2006
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 6 Jul 2006 11:44:16 -0400
Subject: [Catalog-sig] PyPI XML-RPC changes
In-Reply-To: <200607061539.k66FdCV0007024@mail07.syd.optusnet.com.au>
References: <200607061539.k66FdCV0007024@mail07.syd.optusnet.com.au>
Message-ID: <9cee7ab80607060844n397a96c5i498ee5cd8465d19a@mail.gmail.com>
On 7/6/06, richardjones at optusnet.com.au <richardjones at optusnet.com.au> wrote:
> The XML-RPC interface is being reviewed at present, and some quite reasonable changes are being suggested. These will alter the actual method names, so I'd like to know whether anyone on this list is already actually *using* the methods as posted to this list.
Do calls to the XML -RPC methods get logged? That would be a good indicator.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From richardjones at optusnet.com.au Thu Jul 6 17:50:38 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Fri, 07 Jul 2006 01:50:38 +1000
Subject: [Catalog-sig] PyPI XML-RPC changes
Message-ID: <200607061550.k66Foc5W030225@mail26.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060707/fb9e8066/attachment.asc
From richardjones at optusnet.com.au Thu Jul 6 17:52:36 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Fri, 07 Jul 2006 01:52:36 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607061552.k66Fqan8021109@mail07.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060707/c478e8ab/attachment.pot
From fdrake at gmail.com Thu Jul 6 18:36:16 2006
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 6 Jul 2006 12:36:16 -0400
Subject: [Catalog-sig] PyPI XML-RPC changes
In-Reply-To: <200607061550.k66Foc5W030225@mail26.syd.optusnet.com.au>
References: <200607061550.k66Foc5W030225@mail26.syd.optusnet.com.au>
Message-ID: <9cee7ab80607060936x58f03254h8a54dd080adbd127@mail.gmail.com>
On 7/6/06, richardjones at optusnet.com.au <richardjones at optusnet.com.au> wrote:
> Yep, they do, and someone here also suggested this approach. There is a chance though that someone has distributed code that hasn't been exercised yet. Slim, of course.
That code's only interesting if it gets executed. If it was
distributed without even a test run, it's probably buggy anyway. ;-)
Frequency of execution is harder to make predictions about, though. I
figure if you look through a few months (like 3) of logs, that's
plenty. Any calls that aren't found can be changed with impunity.
Of course, one more round of emails on the topic means it'll be
cheaper to leave the existing names regardless of how much better the
new names are.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From pje at telecommunity.com Thu Jul 6 18:56:03 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 12:56:03 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607061503.k66F3X7S005870@mail20.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
At 01:03 AM 7/7/2006 +1000, richardjones at optusnet.com.au wrote:
> > Phillip J. Eby <pje at telecommunity.com> wrote:
> > Why not? ;)
>
>That was actually what I was afraid the reasoning was ;)
>
>I guess I just go all wobbly in the knees at the thought of having to
>maintain a "screen scraping" interface.
You don't need to -- at least not in the long term. Once setuptools 0.7
supports the XML-RPC interface, it won't need the other scraping tricks to
read PyPI. Those would be left in for people who are creating their own
package indexes, not constraining further development of PyPI itself.
Please keep in mind that easy_install makes *extremely* minimal assumptions
about PyPI's interface:
1. It assumes that baseURL/projectname will get to the current version of
projectname, or a page with a list of projectname's active versions
2. It assumes that links within PyPI of the form
baseURL/something1/something2 are links to version 'something2' of a
project named 'something1'
3. It assumes that going to baseURL directly will result in a page with
links to all available packages in the form described in #2.
4. It assumes that if baseURL/projectname returns a page containing the
text "Index of Packages</title>", it is a list of links of the form
described in #2.
5. It looks for and follows the first links following the strings "<th>Home
Page" and "<th>Download URL" in a project page.
6. It makes assumptions about how to find MD5 data on a PyPI page, but if
it fails to do so, it simply won't check the MD5 of downloads.
Also note that even with an XML-RPC interface, easy_install will *still*
need to read an HTML page to gather links, because it's valid for people to
provide links in their long_description using reStructuredText. It's just
that assumptions 1, 3, and 4 (and maybe 5) would not be necessary.
Also note that in a pinch, you can put the strings easy_install is looking
for inside HTML comments. Easy_install really isn't that bright. ;)
However, if you can provide *all* of this data via the API (including an
html-formatted long description), then the screen scraping can go away as
far as PyPI is concerned.
>Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about
>this very issue last night. I proposed that we detect the user-agent of
>the setuptools client, and in response send back really minimalist HTML
>(no surrounding page template). Probably overkill, but this may have been
>after we'd had beer :)
There's a simpler solution that could be implemented: adding a
'rel="easy-install"' attribute to links that easy_install should
follow. Currently, those links are the project's home page URL, download
URL, and the links to specific versions that show up when you go to a
project that has multiple active versions. Adding it to these, and *only*
these links would give easy_install enough information to do the right
thing. However, support would have to wait for setuptools 0.7 anyhow, so
there's little reason to do this.
Hm. I just tried to make multiple versions of PEAK active, and it seems
like you can't get the page that lists multiple versions any more. No
wonder some people have been having problems downloading older versions of
certain packages. :(
How are people supposed to get to older package versions now? That is,
what's the point of being able to have multiple active versions if you
can't find them? Is this an intended change, or a bug?
>Could you provide a clear list of all the specific changes you wish for us
>to make at the Sprint?
I've provided a list above of what changes I want you *not* to make. How's
that? ;)
> > Nonetheless, there are various aspects of easy_install's behavior and
> > performance that could be significantly improved by using XML-RPC, so I
> > definitely want it to do that in 0.7. I'm just wary of removing the
> > existing behavior until it's clear that it's unnecessary for it to.
>
>Oh - another thing that occurred to me -- does setuptools auto update itself?
What do you mean? You can run "easy_install -u setuptools" to upgrade to
the latest release at any time. But it doesn't go out looking for updates
on its own.
From richardjones at optusnet.com.au Fri Jul 7 11:41:50 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Fri, 07 Jul 2006 19:41:50 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060707/3fa8eff3/attachment.pot
From jim at zope.com Fri Jul 7 12:55:44 2006
From: jim at zope.com (Jim Fulton)
Date: Fri, 7 Jul 2006 06:55:44 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
Message-ID: <1DEFC153-D9BB-4FA9-A1FA-65E5F4056AF2@zope.com>
I'd like to suggest that we take a step back. It feels as though we
are reacting rather than designing.
I think we have the following goals:
1. setuptools should be able to read indexes robustly and efficiently.
2. It should be straightforward, and preferably *easy* for people to
implement their own indexes. This is very important to me. :)
Perhaps:
3. It should be easy to mirror an index
4. It should be possible to create a read index as a static HTTP server.
And I suggest:
5. It should be possible to provide an end-user experience for an
index without affecting the setuptools interface
4. It should be possible to write other setuptools-like applications
for accessing indexes. This means that the web-service (small w-s)
should be well defined and/or that setuptools should expose a
Python API for accessing indexes.
From a design perspective:
a. screen scraping is bad
b. the web API should be simple and well defined.
I suggest, as others have suggested, that we create an *alternate*
web API for reading an index focussed on cleanliness and on making
the API as easy as possible to implement for both index and client
developers. If we agree with all of the goals stated above, I think
this should be static HTTP interface using XHTML or some other XML
dialect. Perhaps we could even use specific HTML class attrs to
make it possible to combine the pypi and user interfaces if an index
implementor desires.
Thoughts?
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From pje at telecommunity.com Fri Jul 7 17:52:29 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 11:52:29 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060707114135.02d91280@sparrow.telecommunity.com>
At 07:41 PM 7/7/2006 +1000, richardjones at optusnet.com.au wrote:
> > 3. It assumes that going to baseURL directly will result in a page with
> > links to all available packages in the form described in #2.
>
>This has been removed as it seems completely unnecessary (a flat listing
>of all 1400+ packages, that is). The XML-RPC interface provides the
>functionality you require here.
Note that this will BREAK easy_install in the field, as it will no longer
be possible for easy_install to find packages with odd punctuation in their
names, or which the user has incorrectly specified the case of. E.g. if
someone asks for "sqlobject" instead of "SQLObject", it will no longer work.
This is a pretty serious breakage.
If you really *must* remove this, then you need to add name
canonicalization so that going to /pypi/sqlobject works the same as
/pypi/SQLObject. Similarly, going to "/pypi/foo-bar" should work the same
as "/pypi/Foo & Bar", and so on. That is, a case-insensitive,
"safe_name()" match (see pkg_resources code or docs for the definition of
safe_name()).
If you can't support this (which I've previously asked you to over the last
year so that I could remove the list dependency), PLEASE put the package
list back, because you just broke easy_install's ability to support user
friendly names.
You've known for a *year* that easy_install depended on this feature of PyPI:
http://mail.python.org/pipermail/catalog-sig/2005-June/000654.html
If you needed it to go away, giving me some notice would have been
nice. Not just to me, but to all the people who use PyPI via
easy_install. This gratuitous breakage is not nice.
From pje at telecommunity.com Fri Jul 7 18:07:19 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 12:07:19 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707114135.02d91280@sparrow.telecommunity.co
m>
References: <200607070941.k679fo2f025860@mail23.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060707115709.03fe2ea8@sparrow.telecommunity.com>
At 11:52 AM 7/7/2006 -0400, Phillip J. Eby wrote:
>If you needed it to go away, giving me some notice would have been
>nice. Not just to me, but to all the people who use PyPI via
>easy_install. This gratuitous breakage is not nice.
It looks like easy_install is still actually working at the moment; I guess
that means you kept the list working if the user-agent is
easy_install. Thank you, and sorry for jumping to conclusions. I took
"This has been removed" to mean that it had been, well, removed. :)
I will mention again, though, that it *can* be removed entirely, without
needing any in-the-field upgrades, if project name matches in URLs can be
made case-insensitive and canonical via safe_name:
def safe_name(name):
"""Convert an arbitrary string to a standard distribution name
Any runs of non-alphanumeric/. characters are replaced with a single '-'.
"""
return re.sub('[^A-Za-z0-9.]+', '-', name)
If you can fall back to matching safe_name(pkg_name).lower() against
search_name.lower() if the URL is not exact, then easy_install will not
need the full package listing *or* the XML-RPC interface. It will work
as-is in the field today, with no patching required. The only thing the
full package listing is used for is to do this search. (Likewise, it
would've been the main thing the XML-RPC API would've been used for.)
From pje at telecommunity.com Fri Jul 7 18:18:32 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 12:18:32 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <1DEFC153-D9BB-4FA9-A1FA-65E5F4056AF2@zope.com>
References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
At 06:55 AM 7/7/2006 -0400, Jim Fulton wrote:
> From a design perspective:
>
>a. screen scraping is bad
As long as you define "screen scraping" as "dependency on visible
characteristics of HTML", then I agree. easy_install shouldn't be relying
on the visible bits of HTML that it currently uses to scope out PyPI.
Relying on a particular URL layout is not screen-scraping, though, and
using the URL layout as part of the API has some good properties for ease
of implementation in static form. So does using href's to obtain link
information.
What we should be doing is adding non-visible markup (e.g. class="" or
rel="") information to the links to allow index creators to direct
easy_install without affecting visible page characteristics.
>b. the web API should be simple and well defined.
>
>I suggest, as others have suggested, that we create an *alternate*
>web API for reading an index focussed on cleanliness and on making
>the API as easy as possible to implement for both index and client
>developers. If we agree with all of the goals stated above, I think
>this should be static HTTP interface using XHTML or some other XML
>dialect. Perhaps we could even use specific HTML class attrs to
>make it possible to combine the pypi and user interfaces if an index
>implementor desires.
>
>Thoughts?
+1 on static pages. I don't, however, see a reason to require valid
XML. Or rather, I don't expect to implement XML parsing in easy_install;
if the spec is too complex to implement with regular expression matching,
it's probably too complex for people to throw together an index with what's
at hand. In particular, I'd like it to be practical to put together a
simple index just using Apache's built-in directory indexes, as long as
they use the right URL hierarchy. That means that class or rel attributes
should only be required for links that are requesting non-index pages to be
spidered.
From richardjones at optusnet.com.au Fri Jul 7 18:51:27 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sat, 08 Jul 2006 02:51:27 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607071651.k67GpRvT020914@mail20.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060708/a445612a/attachment.pot
From richardjones at optusnet.com.au Fri Jul 7 19:01:37 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sat, 08 Jul 2006 03:01:37 +1000
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
Message-ID: <200607071701.k67H1b22026443@mail16.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060708/c514dd7d/attachment.asc
From jim at zope.com Fri Jul 7 19:32:42 2006
From: jim at zope.com (Jim Fulton)
Date: Fri, 7 Jul 2006 13:32:42 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
Message-ID: <A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
On Jul 7, 2006, at 12:18 PM, Phillip J. Eby wrote:
> At 06:55 AM 7/7/2006 -0400, Jim Fulton wrote:
>> From a design perspective:
>>
>> a. screen scraping is bad
>
> As long as you define "screen scraping" as "dependency on visible
> characteristics of HTML", then I agree. easy_install shouldn't be
> relying on the visible bits of HTML that it currently uses to scope
> out PyPI.
Yup
> Relying on a particular URL layout is not screen-scraping, though,
> and using the URL layout as part of the API has some good
> properties for ease of implementation in static form. So does
> using href's to obtain link information.
Yes.
> What we should be doing is adding non-visible markup (e.g. class=""
> or rel="") information to the links to allow index creators to
> direct easy_install without affecting visible page characteristics.
Yes
>> b. the web API should be simple and well defined.
>>
>> I suggest, as others have suggested, that we create an *alternate*
>> web API for reading an index focussed on cleanliness and on making
>> the API as easy as possible to implement for both index and client
>> developers. If we agree with all of the goals stated above, I think
>> this should be static HTTP interface using XHTML or some other XML
>> dialect. Perhaps we could even use specific HTML class attrs to
>> make it possible to combine the pypi and user interfaces if an index
>> implementor desires.
>>
>> Thoughts?
>
> +1 on static pages. I don't, however, see a reason to require
> valid XML. Or rather, I don't expect to implement XML parsing in
> easy_install; if the spec is too complex to implement with regular
> expression matching, it's probably too complex for people to throw
> together an index with what's at hand. In particular, I'd like it
> to be practical to put together a simple index just using Apache's
> built-in directory indexes, as long as they use the right URL
> hierarchy. That means that class or rel attributes should only be
> required for links that are requesting non-index pages to be spidered.
I would find parsing much easier with an XML parser than with
regular expressions.
I think it would be much more robust too.
I do want to see something that is well documented and pretty easy to
implement.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From pje at telecommunity.com Fri Jul 7 20:02:43 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 14:02:43 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <200607071701.k67H1b22026443@mail16.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060707134531.02d1d778@sparrow.telecommunity.com>
At 03:01 AM 7/8/2006 +1000, richardjones at optusnet.com.au wrote:
>And I will mention again that I'm not willing to just whack this code into
>the package index without some thought to to ramifications given that
>distutils has no such name mangling or limitation.
...which is arguably a serious bug in the distutils. Actually, it's
several bugs having to do with inconsistent filename mangling throughout
the distutils' various commands, and a design limitation of not
anticipating that people would register Python packages with name's like
"Bob's Incredible Package for Python". :)
The ramifications of adding a fallback search to URL lookup (and you need
*only* add it to URL lookup for "/pypi/projectname") are these:
1. Exact matches will work exactly as they do now
2. URLs that now produce "Not Found" would produce a list of links that
easy_install is already capable of sorting through -- just a shorter one
than the whole package index
3. There is no ramification three. :)
4. Users who manually type a URL to find a package will get a nice helpful
list of links. :)
5. You can remove the full package listing.
>We will have potential (however remote) of name collision and we'll have
>to deal with that somehow.
Yes, and we should deal with it by rejecting registration of colliding
names. The difference between "you can't register an identically-named
package" and "you can't register a package that differs only in letter case
and punctuation from another package" is very small, but also very *good*
for human users, who are not going to be able to remember whether they want
"Bob's very Incredible Package" or "Bob's very incredible package", or be
able to tell the difference between "My Super Package" and "My Super
Package" at a glance. (One of those names has an extra space in it, in
case you can't tell.)
In truth, the name collision problem is already *possible*, and PyPI should
simply reject registration for colliding package names. Allowing people to
register say, "SQLObject" and "sqlobject" or "Zope" and "zope" or "foo bar"
and "Foo!Bar" as if they were actually different packages isn't doing
anybody a favor.
Meanwhile, if you go to /pypi/SQLObject now, you get back a list of links
that includes SQLObject2, so there's obviously some kind of mangling taking
place for URL searches already. If you're concerned about it being
ambiguous, just make the fallback search result in a list of matching
packages, *exactly the same way that going to "SQLObject" does now*, just
with a broader match criterion:
http://cheeseshop.python.org/pypi/SQLObject
From fdrake at gmail.com Fri Jul 7 20:04:12 2006
From: fdrake at gmail.com (Fred Drake)
Date: Fri, 7 Jul 2006 14:04:12 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
References: <5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
Message-ID: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com>
On 7/7/06, Jim Fulton <jim at zope.com> wrote:
> > +1 on static pages. I don't, however, see a reason to require
> > valid XML. Or rather, I don't expect to implement XML parsing in
> > easy_install; if the spec is too complex to implement with regular
> > expression matching, it's probably too complex for people to throw
> > together an index with what's at hand. In particular, I'd like it
> > to be practical to put together a simple index just using Apache's
> > built-in directory indexes, as long as they use the right URL
> > hierarchy. That means that class or rel attributes should only be
> > required for links that are requesting non-index pages to be spidered.
>
> I would find parsing much easier with an XML parser than with
> regular expressions.
> I think it would be much more robust too.
XHTML would be best, though I agree we shouldn't care about validity
so much as just well-formedness (which is required). I think it
should be possible to do it with valid XHTML, though, since whether
that's desired or not is a python.org policy concern. (Not that I
suspect we'll ever really care about that.)
Of course, it should be possible to parse with htmllib and HTMLParser as well.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From pje at telecommunity.com Fri Jul 7 20:31:01 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 14:31:01 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com
>
References: <A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
Message-ID: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
At 02:04 PM 7/7/2006 -0400, Fred Drake wrote:
>On 7/7/06, Jim Fulton <jim at zope.com> wrote:
> > > +1 on static pages. I don't, however, see a reason to require
> > > valid XML. Or rather, I don't expect to implement XML parsing in
> > > easy_install; if the spec is too complex to implement with regular
> > > expression matching, it's probably too complex for people to throw
> > > together an index with what's at hand. In particular, I'd like it
> > > to be practical to put together a simple index just using Apache's
> > > built-in directory indexes, as long as they use the right URL
> > > hierarchy. That means that class or rel attributes should only be
> > > required for links that are requesting non-index pages to be spidered.
> >
> > I would find parsing much easier with an XML parser than with
> > regular expressions.
> > I think it would be much more robust too.
>
>XHTML would be best, though I agree we shouldn't care about validity
>so much as just well-formedness (which is required). I think it
>should be possible to do it with valid XHTML, though, since whether
>that's desired or not is a python.org policy concern. (Not that I
>suspect we'll ever really care about that.)
>
>Of course, it should be possible to parse with htmllib and HTMLParser as well.
I still think requiring even HTML validity or well-formedness is YAGNI; one
could indeed just pull all well-formed URLs from the page. EasyInstall
uses this case-insensitive regular expression to find only href'd urls:
href\s*=\s*['"]?([^'"> ]+)
In the absence of a requirement for more information than this (perhaps
coupled with a "rel" attribute in the same element), I'm wary of starting
out by requiring even well-formedness, because it's way overkill for the
requirements as I understand them.
One of the advantage of defining the URL layout as part of the API is that
it gives you enough contextual information to decide what links should be
followed, and which ones are purely informational.
Indeed, the only reason to look at anything *but* hrefs is to indicate that
an *external* (i.e. non-index) link should be followed, to spider for other
download links. So if following external links is out of scope for the API
we want to define, then *any* information other than the URLs in an API
page are YAGNI.
Now, all of this is based on my assumption that the use case here is
somebody wants to throw together a rough-and-ready package index that tools
should be able to use to find *downloadable distributions*. If you and Jim
have much more elaborate use cases in mind, then of course some
well-formedness might be useful.
On the other hand, if such rigor is required, then it seems like we should
just be using machine-readable data in the first place, rather than using a
dual-purpose format like HTML or XHTML. Just go with a specialized XML
dialect or some kind of text format (ZConfig? ;) ) and be done with it.
From pje at telecommunity.com Fri Jul 7 20:50:36 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 14:50:36 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.co
m>
References: <9cee7ab80607071104wb987d3avd2dd9e872822346c@mail.gmail.com >
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
Message-ID: <5.1.1.6.0.20060707144633.01edd910@sparrow.telecommunity.com>
At 02:31 PM 7/7/2006 -0400, Phillip J. Eby wrote:
>On the other hand, if such rigor is required, then it seems like we should
>just be using machine-readable data in the first place, rather than using a
>dual-purpose format like HTML or XHTML. Just go with a specialized XML
>dialect or some kind of text format (ZConfig? ;) ) and be done with it.
FWIW, I just discovered DOAP, and it seems like a good basis for this sort
of thing. PyPI isn't generating 'file-release' info in its DOAP output,
but probably should, if this is going to be used for that sort of
thing. Likewise, I think that the reST-to-HTML translation should be
carried over into the long-description content.
(This is all assuming, of course, that you have use cases for which the
richness of DOAP is relevant, anyway, as opposed to just trying to find
download links for projects with known names.)
From jim at zope.com Fri Jul 7 20:52:51 2006
From: jim at zope.com (Jim Fulton)
Date: Fri, 7 Jul 2006 14:52:51 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
References: <A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
Message-ID: <46433695-C678-4CE9-AC62-253D01C3EF69@zope.com>
On Jul 7, 2006, at 2:31 PM, Phillip J. Eby wrote:
> At 02:04 PM 7/7/2006 -0400, Fred Drake wrote:
>> On 7/7/06, Jim Fulton <jim at zope.com> wrote:
>> > > +1 on static pages. I don't, however, see a reason to require
>> > > valid XML. Or rather, I don't expect to implement XML parsing in
>> > > easy_install; if the spec is too complex to implement with
>> regular
>> > > expression matching, it's probably too complex for people to
>> throw
>> > > together an index with what's at hand. In particular, I'd
>> like it
>> > > to be practical to put together a simple index just using
>> Apache's
>> > > built-in directory indexes, as long as they use the right URL
>> > > hierarchy. That means that class or rel attributes should
>> only be
>> > > required for links that are requesting non-index pages to be
>> spidered.
>> >
>> > I would find parsing much easier with an XML parser than with
>> > regular expressions.
>> > I think it would be much more robust too.
>>
>> XHTML would be best, though I agree we shouldn't care about validity
>> so much as just well-formedness (which is required). I think it
>> should be possible to do it with valid XHTML, though, since whether
>> that's desired or not is a python.org policy concern. (Not that I
>> suspect we'll ever really care about that.)
>>
>> Of course, it should be possible to parse with htmllib and
>> HTMLParser as well.
>
> I still think requiring even HTML validity or well-formedness is
> YAGNI; one could indeed just pull all well-formed URLs from the
> page. EasyInstall uses this case-insensitive regular expression to
> find only href'd urls:
>
> href\s*=\s*['"]?([^'"> ]+)
>
> In the absence of a requirement for more information than this
> (perhaps coupled with a "rel" attribute in the same element), I'm
> wary of starting out by requiring even well-formedness, because
> it's way overkill for the requirements as I understand them.
But I thought we *were* talking about adding rel or class tags so
that we
could determine information about the intended use of a URL.
> One of the advantage of defining the URL layout as part of the API
> is that it gives you enough contextual information to decide what
> links should be followed, and which ones are purely informational.
Perhaps someone should propose an API and we'll see. :)
> Indeed, the only reason to look at anything *but* hrefs is to
> indicate that an *external* (i.e. non-index) link should be
> followed, to spider for other download links. So if following
> external links is out of scope for the API we want to define, then
> *any* information other than the URLs in an API page are YAGNI.
Who said following external links is out of scope.
> Now, all of this is based on my assumption that the use case here
> is somebody wants to throw together a rough-and-ready package index
> that tools should be able to use to find *downloadable
> distributions*. If you and Jim have much more elaborate use cases
> in mind, then of course some well-formedness might be useful.
setuptools has a notion of an index. That notion is not at all well
defined.
Currently, the index has linkes that are followed to find package
links elsewhere.
This seems reasonably useful. I dunno. I'm not sure I care. What I
do care
about is that the index API should be well defined so that we can
implement
alternate indexes and alternate tools to read indexes. I'm not
looking to
satisfy use cases beyond what we have now.
All I want is an API. :) I'm not bent on XML.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From pje at telecommunity.com Fri Jul 7 22:20:11 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 16:20:11 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <46433695-C678-4CE9-AC62-253D01C3EF69@zope.com>
References: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote:
>On Jul 7, 2006, at 2:31 PM, Phillip J. Eby wrote:
>
>>At 02:04 PM 7/7/2006 -0400, Fred Drake wrote:
>>>On 7/7/06, Jim Fulton <jim at zope.com> wrote:
>>> > > +1 on static pages. I don't, however, see a reason to require
>>> > > valid XML. Or rather, I don't expect to implement XML parsing in
>>> > > easy_install; if the spec is too complex to implement with
>>>regular
>>> > > expression matching, it's probably too complex for people to
>>>throw
>>> > > together an index with what's at hand. In particular, I'd
>>>like it
>>> > > to be practical to put together a simple index just using
>>>Apache's
>>> > > built-in directory indexes, as long as they use the right URL
>>> > > hierarchy. That means that class or rel attributes should
>>>only be
>>> > > required for links that are requesting non-index pages to be
>>>spidered.
>>> >
>>> > I would find parsing much easier with an XML parser than with
>>> > regular expressions.
>>> > I think it would be much more robust too.
>>>
>>>XHTML would be best, though I agree we shouldn't care about validity
>>>so much as just well-formedness (which is required). I think it
>>>should be possible to do it with valid XHTML, though, since whether
>>>that's desired or not is a python.org policy concern. (Not that I
>>>suspect we'll ever really care about that.)
>>>
>>>Of course, it should be possible to parse with htmllib and
>>>HTMLParser as well.
>>
>>I still think requiring even HTML validity or well-formedness is
>>YAGNI; one could indeed just pull all well-formed URLs from the
>>page. EasyInstall uses this case-insensitive regular expression to
>>find only href'd urls:
>>
>> href\s*=\s*['"]?([^'"> ]+)
>>
>>In the absence of a requirement for more information than this
>>(perhaps coupled with a "rel" attribute in the same element), I'm
>>wary of starting out by requiring even well-formedness, because
>>it's way overkill for the requirements as I understand them.
>
>But I thought we *were* talking about adding rel or class tags so
>that we
>could determine information about the intended use of a URL.
Yes -- but they're only needed to support following second-order external
links: i.e., links to non-index HTML pages.
>>One of the advantage of defining the URL layout as part of the API
>>is that it gives you enough contextual information to decide what
>>links should be followed, and which ones are purely informational.
>
>Perhaps someone should propose an API and we'll see. :)
I thought I already did. :) Here it is again:
baseURL/ should return a page containing href links to projects
baseURL/projectname should return a page containing href links to version pages
baseURL/projectname/version should return a page with download links
(ideally with MD5 info)
Links are found via href="" attributes
URLs' trailing path components are used to identify distributions.
This is a sufficient API to allow querying packages for downloading
purposes, as long as all download links are found in the index's
pages. Additional information is only needed to allow following external
links to *other index pages*.
Coincidentally, easy_install is already mostly compatible with such an API;
it would mostly be a matter of *removing* things from easy_install, rather
than adding them.
>>Indeed, the only reason to look at anything *but* hrefs is to
>>indicate that an *external* (i.e. non-index) link should be
>>followed, to spider for other download links. So if following
>>external links is out of scope for the API we want to define, then
>>*any* information other than the URLs in an API page are YAGNI.
>
>Who said following external links is out of scope.
Nobody; I was just saying that *if* it were out of scope, the class/rel
stuff would become unnecessary.
>>Now, all of this is based on my assumption that the use case here
>>is somebody wants to throw together a rough-and-ready package index
>>that tools should be able to use to find *downloadable
>>distributions*. If you and Jim have much more elaborate use cases
>>in mind, then of course some well-formedness might be useful.
>
>setuptools has a notion of an index. That notion is not at all well
>defined.
It's mostly operationally defined in terms of what PyPI did when it was
written.
>Currently, the index has linkes that are followed to find package
>links elsewhere.
>This seems reasonably useful. I dunno. I'm not sure I care. What I
>do care
>about is that the index API should be well defined so that we can
>implement
>alternate indexes and alternate tools to read indexes. I'm not
>looking to
>satisfy use cases beyond what we have now.
Sure. I'm just saying we only need something beyond href="" links if they
are intended to be followed by tools looking for package links.
The reason this is necessary, is that it's not sufficient to just follow
links that point outside the package index; PyPI has links on its pages
that go to other parts of python.org, so there needs to be something that
distinguishes "links that might help find downloads". Links that *are*
downloads are detected via URL content.
From jim at zope.com Fri Jul 7 22:45:59 2006
From: jim at zope.com (Jim Fulton)
Date: Fri, 7 Jul 2006 16:45:59 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
Message-ID: <9A0CD067-84D0-4BF0-A5D9-9306AFCDD42E@zope.com>
On Jul 7, 2006, at 4:20 PM, Phillip J. Eby wrote:
> At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote:
...
>> Perhaps someone should propose an API and we'll see. :)
>
> I thought I already did. :) Here it is again:
>
> baseURL/ should return a page containing href links to projects
> baseURL/projectname should return a page containing href links to
> version pages
> baseURL/projectname/version should return a page with download
> links (ideally with MD5 info)
> Links are found via href="" attributes
> URLs' trailing path components are used to identify distributions.
Hm. I hadn't seen this before. Perhaps I'm missing some messages from
this thread.
By "download links", do you mean links to distributions? Or to links
to pages containing links to distributions.
Can the links to projects, links to version pages, or download links
point off site?
Can any of these pages contain other links?
> This is a sufficient API to allow querying packages for downloading
> purposes, as long as all download links are found in the index's
> pages. Additional information is only needed to allow following
> external links to *other index pages*.
so, for example:
http://www.python.org/pypi/ZODB3/3.6.0
Has a link to http://www.zope.org/Products/ZODB3.6.
Is this a download link? Or an off-site index link. I'm having a
little trouble
following the jargon.
>> setuptools has a notion of an index. That notion is not at all well
>> defined.
>
> It's mostly operationally defined in terms of what PyPI did when it
> was written.
Right, not well defined. :)
I'm not criticizing. What it does was great as a prototype,
but it would be good move beyond this.
>> Currently, the index has linkes that are followed to find package
>> links elsewhere.
>> This seems reasonably useful. I dunno. I'm not sure I care. What I
>> do care
>> about is that the index API should be well defined so that we can
>> implement
>> alternate indexes and alternate tools to read indexes. I'm not
>> looking to
>> satisfy use cases beyond what we have now.
>
> Sure. I'm just saying we only need something beyond href="" links
> if they are intended to be followed by tools looking for package
> links.
>
> The reason this is necessary, is that it's not sufficient to just
> follow links that point outside the package index; PyPI has links
> on its pages that go to other parts of python.org, so there needs
> to be something that distinguishes "links that might help find
> downloads". Links that *are* downloads are detected via URL content.
Right. That's why I think the hrefs we care about should be marked
with class
attributes or some such.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From pje at telecommunity.com Sat Jul 8 03:12:03 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 21:12:03 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <9A0CD067-84D0-4BF0-A5D9-9306AFCDD42E@zope.com>
References: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com>
At 04:45 PM 7/7/2006 -0400, Jim Fulton wrote:
>On Jul 7, 2006, at 4:20 PM, Phillip J. Eby wrote:
>
>>At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote:
>...
>>>Perhaps someone should propose an API and we'll see. :)
>>
>>I thought I already did. :) Here it is again:
>>
>>baseURL/ should return a page containing href links to projects
>>baseURL/projectname should return a page containing href links to
>>version pages
>>baseURL/projectname/version should return a page with download
>>links (ideally with MD5 info)
>>Links are found via href="" attributes
>>URLs' trailing path components are used to identify distributions.
>
>Hm. I hadn't seen this before. Perhaps I'm missing some messages from
>this thread.
>
>By "download links", do you mean links to distributions?
Yes.
>Or to links
>to pages containing links to distributions.
No, these would be either "index pages", or "external links"
>Can the links to projects, links to version pages, or download links
>point off site?
Download links can be anywhere, since they are identified from the tail of
the URL. The links to project or version pages are defined by the URL
hierarchy of the API.
>Can any of these pages contain other links?
All of them can contain download links. Index pages can link to other
index pages. Index pages linked to anything else are ignored, unless we
allow "external links", in which case a method of identifying them is required.
Currently, easy_install identifies only uses two kinds of external links:
home page and "download URL". These are identified via HTML snippets that
PyPI uses. This is one of only two pieces of "screen scraping" (as opposed
to URL inspection and link detection) that easy_install has. (The other is
used to distinguish between a page that lists links to projects, from an
actual project page, as sometimes PyPI can display the former at a URL that
is nominally for the latter.)
>>This is a sufficient API to allow querying packages for downloading
>>purposes, as long as all download links are found in the index's
>>pages. Additional information is only needed to allow following
>>external links to *other index pages*.
>
>so, for example:
>
> http://www.python.org/pypi/ZODB3/3.6.0
>
>Has a link to http://www.zope.org/Products/ZODB3.6.
>Is this a download link? Or an off-site index link. I'm having a
>little trouble
>following the jargon.
It's an "external link", and thus only followed if it's seen to be the
"home page" or "download URL" on a package version page.
>>Sure. I'm just saying we only need something beyond href="" links
>>if they are intended to be followed by tools looking for package
>>links.
>>
>>The reason this is necessary, is that it's not sufficient to just
>>follow links that point outside the package index; PyPI has links
>>on its pages that go to other parts of python.org, so there needs
>>to be something that distinguishes "links that might help find
>>downloads". Links that *are* downloads are detected via URL content.
>
>Right. That's why I think the hrefs we care about should be marked
>with class
>attributes or some such.
Yes, as long as we care about supporting the external links. I'm not
certain we do, at least for the "third-party index" case.
From richardjones at optusnet.com.au Sat Jul 8 10:24:50 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sat, 08 Jul 2006 18:24:50 +1000
Subject: [Catalog-sig] Ickiness
Message-ID: <200607080824.k688OpFE026560@mail27.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060708/31c2e915/attachment.pot
From jim at zope.com Sat Jul 8 13:38:06 2006
From: jim at zope.com (Jim Fulton)
Date: Sat, 8 Jul 2006 07:38:06 -0400
Subject: [Catalog-sig] [Distutils] Specification for package indexes?
In-Reply-To: <5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060706122834.02056f00@sparrow.telecommunity.com>
<5.1.1.6.0.20060707120827.02d8f2b8@sparrow.telecommunity.com>
<A1A2411D-7DB6-4259-A07A-5F93A64C4703@zope.com>
<5.1.1.6.0.20060707142217.02d15bd0@sparrow.telecommunity.com>
<5.1.1.6.0.20060707160846.032ffc88@sparrow.telecommunity.com>
<5.1.1.6.0.20060707175405.01edc3a8@sparrow.telecommunity.com>
Message-ID: <E82D2DED-6DB1-4AF3-999D-AF1BB06BCEC0@zope.com>
On Jul 7, 2006, at 9:12 PM, Phillip J. Eby wrote:
> At 04:45 PM 7/7/2006 -0400, Jim Fulton wrote:
...
>> By "download links", do you mean links to distributions?
>
> Yes.
>
>
>> Or to links
>> to pages containing links to distributions.
>
> No, these would be either "index pages", or "external links"
Which seems to be an important use case now.
>
>> Can the links to projects, links to version pages, or download links
>> point off site?
>
> Download links can be anywhere, since they are identified from the
> tail of the URL. The links to project or version pages are defined
> by the URL hierarchy of the API.
Hm. Why does it matter? I understand that you want to be able to go to
index_url/project first, but I don't see that it matters where
versions actually are.
For that matter, I could see value in a minimal index that just
pointed to
external project pages. In which case, going to index_url/project
might even
be allowed to redirect to an offsite project page. Of course, this
couldn't be
implemented with a static server, but could still be a valuable option.
>
>> Can any of these pages contain other links?
>
> All of them can contain download links. Index pages can link to
> other index pages. Index pages linked to anything else are
> ignored, unless we allow "external links", in which case a method
> of identifying them is required.
I think we want external links. We have them now. In fact, I think
there is value in a
project index that has no distributions or even version information
but provides
a central place to find project pages.
Note that, in a separate discussion, you pointed out that some
considered it
bad form to put interim project releases on pypi. If pypi could have
links to
remote project pages, then those sites could have different policies
as needed by
a project.
> Currently, easy_install identifies only uses two kinds of external
> links: home page and "download URL". These are identified via HTML
> snippets that PyPI uses. This is one of only two pieces of "screen
> scraping" (as opposed to URL inspection and link detection) that
> easy_install has. (The other is used to distinguish between a page
> that lists links to projects, from an actual project page, as
> sometimes PyPI can display the former at a URL that is nominally
> for the latter.)
>
>>> This is a sufficient API to allow querying packages for downloading
>>> purposes, as long as all download links are found in the index's
>>> pages. Additional information is only needed to allow following
>>> external links to *other index pages*.
>>
>> so, for example:
>>
>> http://www.python.org/pypi/ZODB3/3.6.0
>>
>> Has a link to http://www.zope.org/Products/ZODB3.6.
>> Is this a download link? Or an off-site index link. I'm having a
>> little trouble
>> following the jargon.
>
> It's an "external link", and thus only followed if it's seen to be
> the "home page" or "download URL" on a package version page.
Right, which is currently identified by sniffing the surrounding HTML.
>
>>> Sure. I'm just saying we only need something beyond href="" links
>>> if they are intended to be followed by tools looking for package
>>> links.
>>>
>>> The reason this is necessary, is that it's not sufficient to just
>>> follow links that point outside the package index; PyPI has links
>>> on its pages that go to other parts of python.org, so there needs
>>> to be something that distinguishes "links that might help find
>>> downloads". Links that *are* downloads are detected via URL
>>> content.
>>
>> Right. That's why I think the hrefs we care about should be marked
>> with class
>> attributes or some such.
>
> Yes, as long as we care about supporting the external links. I'm
> not certain we do, at least for the "third-party index" case.
I think we do. I'm pretty sure we do for pypi and I sure has heck
don't want a different
api for pypi and for other indexes. I'd really like to see a single
index api.
I would *like* to see the possibility of allowing off-site (off-
index) projects,
although I could live without this.
I have to say again that all of these details can get quite confusing.
Maybe I'm alone in being confused by this, but I don't think so.
I've spent
a lot of time on and off over the last few months trying to leverage
setuptools
and now pypi and while I've had a lot of success, it has been harder
than I
think it should be. I think that this is an impediment to greater
adoption of and
benefit from setuptools. I think we need to do a good job of
documenting and
explaining this API. I also think we need to write up some best
practices
or rational to guide people toward better use of setuptools and pypi
together.
I'm happy to help with this once we have agreement and once I
understand what we agree to. :)
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optusnet.com.au Sat Jul 8 15:02:10 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sat, 08 Jul 2006 23:02:10 +1000
Subject: [Catalog-sig] safe_names again
Message-ID: <200607081302.k68D2AaW011573@mail05.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060708/ca90f3e2/attachment.pot
From jim at zope.com Sat Jul 8 15:11:02 2006
From: jim at zope.com (Jim Fulton)
Date: Sat, 8 Jul 2006 09:11:02 -0400
Subject: [Catalog-sig] safe_names again
In-Reply-To: <200607081302.k68D2AaW011573@mail05.syd.optusnet.com.au>
References: <200607081302.k68D2AaW011573@mail05.syd.optusnet.com.au>
Message-ID: <9558562B-825A-45A4-89D3-6FDA90815A50@zope.com>
On Jul 8, 2006, at 9:02 AM, richardjones at optusnet.com.au wrote:
> [sorry for the terrible email quoting / formatting - I'm stuck in
> webmail ATM. I'm also having trouble following the discussion PJE
> and Jim are having - and that's a concern to me because some of the
> stuff I'm reading worries me.]
Don't feel bad, I have trouble following this too. :/
> I have created a branch for this and have begun the slow process of
> working in the setuptools name mangling. It's going to take some
> time since package names are used all over the place and form an
> integral part of the database referential structure.
IMO, no one should be writing code at this point. I think we should
slow down, decide on and carefully document an API before we do any
more coding. (Where "we" mostly refers to you and Phillip. :)
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optusnet.com.au Sat Jul 8 15:26:18 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sat, 08 Jul 2006 23:26:18 +1000
Subject: [Catalog-sig] safe_names again
Message-ID: <200607081326.k68DQIV1024749@mail31.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060708/bb64c71e/attachment.asc
From jim at zope.com Sat Jul 8 16:08:17 2006
From: jim at zope.com (Jim Fulton)
Date: Sat, 8 Jul 2006 10:08:17 -0400
Subject: [Catalog-sig] safe_names again
In-Reply-To: <200607081326.k68DQIV1024749@mail31.syd.optusnet.com.au>
References: <200607081326.k68DQIV1024749@mail31.syd.optusnet.com.au>
Message-ID: <1D30D765-834F-4F15-AC3D-09B280B63C07@zope.com>
On Jul 8, 2006, at 9:26 AM, richardjones at optusnet.com.au wrote:
>> Jim Fulton <jim at zope.com> wrote:
>> Don't feel bad, I have trouble following this too. :/
>
> Great, Johannes (who is next to me and in a similar state of mind)
> and I feel much better now :)
>
>
>>> I have created a branch for this and have begun the slow process of
>>> working in the setuptools name mangling. It's going to take some
>>> time since package names are used all over the place and form an
>>> integral part of the database referential structure.
>>
>> IMO, no one should be writing code at this point. I think we should
>> slow down, decide on and carefully document an API before we do any
>> more coding. (Where "we" mostly refers to you and Phillip. :)
>
> My position on all this is that:
>
> 1. PyPI is the central index for metadata about python packages. I
> am not aware of any other such indexes, nor am I aware of an
> intention to create one. I would not be in favour of doing so,
> since it would fragment the community.
I am mostly concerned that we have a well-defined API, however, I
think there is a potential place for other indexes. There are all
sorts of reasons one might not want or be able to put software in
PyPI. For example, an organization may have proprietary information
that they can't put in PyPI. They should be able to run their own
index.
This would not be quire so important if setuptools provides an option
to not look in an index or, if having found packages via find-links,
it was willing to not look at an index.
> 2. I'm not particularly interested in developing APIs for PyPI
I'm gomma interpret this as meaning that you don't want to have to
develop (define) an API yourself, but you would be happy if there was
an API. I hope I'm right.
> though I am happy to add in XML-RPC functions as needed since
> they're trivial to write.
So you are unconcerned by the use case for static indexes? Or static
mirrors of indexes? Do you really want to prevent such a use case
from being met?
> 3. I would really like to see an emphasis on moving away from
> scraping HTML since the HTML is changed for *cosmetic* reasons. If
> need be we can produce "static" XML (like the DOAP) to go alongside
> it which will not change the next time the pydotorg layout is changed.
I think everyone agrees with you. I think screen scraping was a
valuable expedient when setuptools was being prototyped, but it's
time to move beyond that. Obviously, we have to agree what to do,
but that means an API.
> 4. Patches, and additional svn committers, are always welcome.
I'd be willing to help out once we agree *clearly* what we're doing.
That means, among other things, an API.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optusnet.com.au Sat Jul 8 16:37:21 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sun, 09 Jul 2006 00:37:21 +1000
Subject: [Catalog-sig] safe_names again
Message-ID: <200607081437.k68EbMEC016454@mail18.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060709/224b061c/attachment.asc
From pje at telecommunity.com Sat Jul 8 20:07:31 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 08 Jul 2006 14:07:31 -0400
Subject: [Catalog-sig] safe_names again
In-Reply-To: <200607081302.k68D2AaW011573@mail05.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060708135905.01ec4910@sparrow.telecommunity.com>
At 11:02 PM 7/8/2006 +1000, richardjones at optusnet.com.au wrote:
>[sorry for the terrible email quoting / formatting - I'm stuck in webmail
>ATM. I'm also having trouble following the discussion PJE and Jim are
>having - and that's a concern to me because some of the stuff I'm reading
>worries me.]
>
>I have created a branch for this and have begun the slow process of
>working in the setuptools name mangling. It's going to take some time
>since package names are used all over the place and form an integral part
>of the database referential structure.
>
>My "plan" for implementation of this is as follows:
>
>1. Convert meta-data supplied by end users using safe_name. We call this
>"name" internally (replacing the current use of that column). The original
>name is retained for display purposes, stored as "display_name" on the
>packages table.
>
>2. All user input of names for filtering must be mangled before searching
>is performed.
>
>3. Find all places where we display names and convert them to use the
>"display_name" column.
It's just a suggestion, but you might find it easier to phase in by simply:
1. Reject creating a package if its safe-name conflicts with another package
2. Provide search facilities that search on mangled name (which is used by
#1 to verify a package that's about to be added.
This seems like a more conservative route with a lot less effort involved
if you want to ease into it. In the simplest case it requires no database
schema changes, although if performance is an issue, adding a mangled name
column for the searches would be useful, but not strictly necessary. More
to the point, these two changes do not require wholesale refactoring of PyPI.
Of course, if you feel that all searches in PyPI should work this way, then
your plan makes more sense. I wasn't sure if you felt that way or not. On
the other hand, by leaving "name" alone, and having a "safe_name" column
for searching, you can make the change more gradual, if that's a
concern. You could make safe_name be uniquely indexed, while leaving name
as the primary key, and just trap the error when inserting a conflicting
safe_name.
Anyway, just some random thoughts to see if there's an easier way for you
to do this.
From pje at telecommunity.com Sat Jul 8 20:33:48 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 08 Jul 2006 14:33:48 -0400
Subject: [Catalog-sig] Ickiness
In-Reply-To: <200607080824.k688OpFE026560@mail27.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060708142629.01f321b0@sparrow.telecommunity.com>
At 06:24 PM 7/8/2006 +1000, richardjones at optusnet.com.au wrote:
>With the new display formatting implemented yesterday, there is now a
><table style="display: none"> in the package view page so that setuptools
>may scrape "<th>Download URL" etc.
You can also do this: <!-- <th>Download URL --> right before the
link. EasyInstall is very stupid and doesn't actually parse any HTML.
The same thing is true for the "'Index of Packages</title>'" string, which
just has to appear somewhere in the page if it's a multi-package list.
While these strings obviously aren't as "clean" as using rel="" info in the
links, they will work for now and allow you to do whatever you want with
the visual appearance.
The only other visual dependency is on MD5 information, which is extracted
using this pattern:
PYPI_MD5 = re.compile(
'<a href="([^"#]+)">([^<]+)</a>\n\s+\\(<a href="[^?]+\?:action=show_md5'
'&digest=([0-9a-f]{32})">md5</a>\\)'
)
However, if this pattern doesn't match, then EasyInstall will simply
proceed without MD5 verification.
From dangoor at gmail.com Sun Jul 9 02:19:12 2006
From: dangoor at gmail.com (Kevin Dangoor)
Date: Sat, 8 Jul 2006 20:19:12 -0400
Subject: [Catalog-sig] PyPI XML-RPC changes
In-Reply-To: <200607061539.k66FdCV0007024@mail07.syd.optusnet.com.au>
References: <200607061539.k66FdCV0007024@mail07.syd.optusnet.com.au>
Message-ID: <C3FAC62B-F308-46CF-83D8-669F0F08A181@gmail.com>
On Jul 6, 2006, at 11:39 AM, richardjones at optusnet.com.au wrote:
> The XML-RPC interface is being reviewed at present, and some quite
> reasonable changes are being suggested. These will alter the actual
> method names, so I'd like to know whether anyone on this list is
> already actually *using* the methods as posted to this list. If so,
> then I'll leave aliases in the code. If not, I'll just do the rename.
>
> The changes, BTW, are:
>
> package_urls -> release_urls
> package_data -> release_data
> package_stable_version -> package_stable_release
I have code that uses package_data.
Kevin
From richardjones at optusnet.com.au Sun Jul 9 09:51:07 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sun, 09 Jul 2006 17:51:07 +1000
Subject: [Catalog-sig] Ickiness
Message-ID: <200607090751.k697pCvN001127@mail15.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060709/5bc0e7f3/attachment.pot
From richardjones at optusnet.com.au Sun Jul 9 10:01:34 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sun, 09 Jul 2006 18:01:34 +1000
Subject: [Catalog-sig] PyPI XML-RPC changes
Message-ID: <200607090801.k6981YwX012215@mail31.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060709/8f2a7a3b/attachment.asc
From richardjones at optusnet.com.au Sun Jul 9 10:03:57 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Sun, 09 Jul 2006 18:03:57 +1000
Subject: [Catalog-sig] Ickiness
Message-ID: <200607090804.k69842gT027039@mail14.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060709/ccb86b9e/attachment.pot
From pje at telecommunity.com Tue Jul 11 03:09:57 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 10 Jul 2006 21:09:57 -0400
Subject: [Catalog-sig] "Package Index API" draft
Message-ID: <5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com>
This draft reflects the in-development versions of setuptools 0.7a1 and
0.6b4; it does not describe older setuptools versions except as noted under
the "Backward Compatibility" section. The items described under "Backward
Compatibility" need to be kept in PyPI until everyone in the field has
upgraded to setuptools 0.6b4 or better. (Note that 0.6b4 is not released yet!)
Package Index "API"
-------------------
Custom package indexes (and PyPI) must follow the following rules for
EasyInstall to be able to look up and download packages:
1. Except where stated otherwise, "pages" are HTML or XHTML, and "links"
refer to ``href`` attributes.
2. Individual project version pages' URLs must be of the form
``base/projectname/version``, where ``base`` is the package index's base
URL.
3. Omitting the ``/version`` part of a project page's URL (but keeping the
trailing ``/``) should result in a page that is either:
a) The single active version of that project, as though the version had
been explicitly included, OR
b) A page with links to all of the active version pages for that project.
4. Individual version pages should contain direct links to downloadable
distributions where possible. It is explicitly permitted for a project's
"long_description" to include URLs, and these should be formatted as HTML
links by the package index, as EasyInstall does no special processing to
identify what parts of a page are index-specific and which are part of the
project's supplied description.
5. Where available, MD5 information should be added to download URLs by
appending a fragment identifier of the form ``#md5=...``, where ``...`` is
the 32-character hex MD5 digest. EasyInstall will verify that the
downloaded file's MD5 digest matches the given value.
6. Individual project version pages should identify any "homepage" or
"download" URLs using ``rel="homepage"`` and ``rel="download"`` attributes
on the HTML elements linking to those URLs. Use of these attributes will
cause EasyInstall to always follow the provided links, unless it can be
determined by inspection that they are downloadable distributions. If the
links are not to downloadable distributions, they are retrieved, and if
they are HTML, they are scanned for download links. They are *not* scanned
for additional "homepage" or "download" links, as these are only processed
for pages that are part of a package index site.
7. The root URL of the index, if retrieved with a trailing ``/``, must result
in a page containing links to *all* projects' active version pages.
(Note: This requirement is a workaround for the absence of case-
insensitive ``safe_name()`` matching of project names in URL paths. If
project names are matched in this fashion (e.g. via the PyPI server,
mod_rewrite, or a similar mechanism), then it is not necessary to
include this all-packages listing page.)
8. If a package index is accessed via a ``file://`` URL, then EasyInstall
will automatically use ``index.html`` files, if present, when trying to
read a directory with a trailing ``/`` on the URL.
Backward Compatibility
~~~~~~~~~~~~~~~~~~~~~~
Package indexes that wish to support setuptools versions prior to 0.6b4
should also follow these rules:
* Homepage and download links must be preceded with ``"<th>Home Page"`` or
``"<th>Download URL"``, in addition to (or instead of) the ``rel=""``
attributes on the actual links. These marker strings do not need to be
visible, or uncommented, however! For example, the following is a valid
homepage link that will work with any version of setuptools::
<li>
<strong>Home Page:</strong>
<!-- <th>Home Page -->
<a rel="homepage" href="http://sqlobject.org">http://sqlobject.org</a>
</li>
Even though the marker string is in an HTML comment, older versions of
EasyInstall will still "see" it and know that the link that follows is the
project's home page URL.
* The pages described by paragraph 3(b) of the preceding section *must*
contain the string ``"Index of Packages</title>"`` somewhere in their text.
This can be inside of an HTML comment, if desired, and it can be anywhere
in the page. (Note: this string MUST NOT appear on normal project pages,
as described in paragraphs 2 and 3(a)!)
In addition, for compatibility with PyPI versions that do not use ``#md5=``
fragment IDs, EasyInstall uses the following regular expression to match
PyPI's displayed MD5 info (broken onto two lines for readability)::
<a href="([^"#]+)">([^<]+)</a>\n\s+\(<a href="[^?]+\?:action=show_md5
&digest=([0-9a-f]{32})">md5</a>\)
From pje at telecommunity.com Tue Jul 11 21:12:16 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 15:12:16 -0400
Subject: [Catalog-sig] setuptools 0.6b4 released
Message-ID: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
I have just released version 0.6b4 of setuptools, the last beta release of
setuptools 0.6. Please upgrade and test at your earliest convenience, as I
would like to issue a release candidate version next week.
Changes include numerous bug fixes and tweaks for corner cases in
easy_install processing, mostly discovered by Jim Fulton in the process of
developing his "zc.buildout" tool (a Make-like system that supports simple
installation of complex environments).
In addition, there is now a formally-documented "package index API" that
easy_install supports:
http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
So, people who wish to create their own "package indexes" for easy_install
can do so, even using static HTML -- even without a web server.
Last, but not least, the ability was added to turn off SVN revision numbers
or dates from the command line, so that you don't have to edit setup.cfg in
order to issue a release.
There are a few outstanding requests that I was *not* able to consider for
0.6 because the required changes would've been too disruptive to
stability. They have been bumped to the 0.7 list to receive further
consideration after 0.6 final is released:
* A request to change the PYTHONPATH resolution order
* A request to allow installed scripts to use -O
* Adding XML-RPC support for PyPI
From bob at redivi.com Tue Jul 11 21:33:16 2006
From: bob at redivi.com (Bob Ippolito)
Date: Tue, 11 Jul 2006 12:33:16 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
Message-ID: <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
> Last, but not least, the ability was added to turn off SVN revision
> numbers
> or dates from the command line, so that you don't have to edit
> setup.cfg in
> order to issue a release.
Would've been convenient if you said what the option was... took me a
few minutes to figure out where to look. Presumably you're referring
to the --no-svn-revision option?
Options for 'egg_info' command:
--egg-base (-e) directory containing .egg-info directories
(default: top of the source tree)
--tag-svn-revision (-r) Add subversion revision ID to version number
--tag-date (-d) Add date stamp (e.g. 20050528) to version
number
--tag-build (-b) Specify explicit tag to add to version
number
--no-svn-revision (-R) Don't add subversion revision ID [default]
--no-date (-D) Don't include date stamp [default]
--tag-build (-b) Specify explicit tag to add to version
number
Note that tag-build shows up twice, here's a patch:
Index: setuptools/command/egg_info.py
===================================================================
--- setuptools/command/egg_info.py (revision 50587)
+++ setuptools/command/egg_info.py (working copy)
@@ -28,7 +28,6 @@
('no-svn-revision', 'R',
"Don't add subversion revision ID [default]"),
('no-date', 'D', "Don't include date stamp [default]"),
- ('tag-build=', 'b', "Specify explicit tag to add to version
number"),
]
boolean_options = ['tag-date', 'tag-svn-revision']
-bob
From pje at telecommunity.com Tue Jul 11 21:53:01 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 15:53:01 -0400
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060711154900.03a9ff88@sparrow.telecommunity.com>
At 12:33 PM 7/11/2006 -0700, Bob Ippolito wrote:
>On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
>>Last, but not least, the ability was added to turn off SVN revision
>>numbers
>>or dates from the command line, so that you don't have to edit
>>setup.cfg in
>>order to issue a release.
>
>Would've been convenient if you said what the option was... took me a
>few minutes to figure out where to look. Presumably you're referring
>to the --no-svn-revision option?
Yep. See also:
http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion
In particular, the new subsection:
http://peak.telecommunity.com/DevCenter/setuptools#making-official-non-snapshot-releases
Quick tip: you can use:
http://peak.telecommunity.com/DevCenter/setuptools?action=diff
to check on the latest changes to setuptools' doc after a release. The
wiki also allows you to subscribe to receive change notices via
email. This also goes for the EasyInstall and PkgResources doc pages.
>Note that tag-build shows up twice, here's a patch:
Argh. :( Thanks for catching that. I've fixed it in the trunk and 0.6
branch.
From bob at redivi.com Thu Jul 13 06:00:06 2006
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 12 Jul 2006 21:00:06 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
Message-ID: <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com>
On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
> I have just released version 0.6b4 of setuptools, the last beta
> release of
> setuptools 0.6. Please upgrade and test at your earliest
> convenience, as I
> would like to issue a release candidate version next week.
It seems there's another bug in 0.6b4. I don't know what conditions
cause it, however:
File "/Volumes/Data/developer-external/py2app/setuptools-0.6b4-
py2.4.egg/setuptools/dist.py", line 274, in fetch_build_egg
AttributeError: Distribution instance has no attribute
'dependency_links'
It looks like there's no default for dependency_links in some cases..
so that needs to change to a getattr, or it needs to set some default.
-bob
From bob at redivi.com Thu Jul 13 06:09:20 2006
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 12 Jul 2006 21:09:20 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<8D2232FF-D626-47F6-97AF-0CD1515FB59F@redivi.com>
Message-ID: <1C91F591-0072-4E7C-9E5B-F1DC0F79FA3D@redivi.com>
On Jul 12, 2006, at 9:00 PM, Bob Ippolito wrote:
>
> On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
>
>> I have just released version 0.6b4 of setuptools, the last beta
>> release of
>> setuptools 0.6. Please upgrade and test at your earliest
>> convenience, as I
>> would like to issue a release candidate version next week.
>
> It seems there's another bug in 0.6b4. I don't know what conditions
> cause it, however:
>
> File "/Volumes/Data/developer-external/py2app/setuptools-0.6b4-
> py2.4.egg/setuptools/dist.py", line 274, in fetch_build_egg
> AttributeError: Distribution instance has no attribute
> 'dependency_links'
>
> It looks like there's no default for dependency_links in some cases..
> so that needs to change to a getattr, or it needs to set some default.
Ok, it seems this happens whenever you do develop, install, etc. when
all dependencies aren't available. easy_install seems to work though.
This patch seems to make everything work as expected:
===================================================================
--- setuptools/dist.py (revision 50587)
+++ setuptools/dist.py (working copy)
@@ -271,7 +271,7 @@
for key in opts.keys():
if key not in keep:
del opts[key] # don't use any other settings
- if self.dependency_links:
+ if getattr(self, 'dependency_links', None):
links = self.dependency_links[:]
if 'find_links' in opts:
links = opts['find_links'][1].split() + links
-bob
From bob at redivi.com Thu Jul 13 09:43:32 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 13 Jul 2006 00:43:32 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
Message-ID: <2E9807AF-1C61-4E49-88A9-DC32CD9D0336@redivi.com>
On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
> I have just released version 0.6b4 of setuptools, the last beta
> release of
> setuptools 0.6. Please upgrade and test at your earliest
> convenience, as I
> would like to issue a release candidate version next week.
Here's another patch I'd like to see ASAP. Some vendors have Q&A
procedures that disallow empty files without special exception, so we
should ensure that setuptools doesn't create empty safety flag files.
Index: setuptools/command/bdist_egg.py
===================================================================
--- setuptools/command/bdist_egg.py (revision 50611)
+++ setuptools/command/bdist_egg.py (working copy)
@@ -360,7 +360,9 @@
if safe is None or bool(safe)<>flag:
os.unlink(fn)
elif safe is not None and bool(safe)==flag:
- open(fn,'w').close()
+ f = open(fn,'w')
+ f.write(fn)
+ f.close()
safety_flags = {
True: 'zip-safe',
-bob
From jim at zope.com Thu Jul 13 21:57:45 2006
From: jim at zope.com (Jim Fulton)
Date: Thu, 13 Jul 2006 15:57:45 -0400
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
Message-ID: <DBD3CA80-99D4-4F42-AB9A-35DC3960A2C9@zope.com>
On Jul 11, 2006, at 3:12 PM, Phillip J. Eby wrote:
> I have just released version 0.6b4 of setuptools, the last beta
> release of
> setuptools 0.6. Please upgrade and test at your earliest
> convenience, as I
> would like to issue a release candidate version next week.
With the new version, I'm still having the problem that the --no-deps
option is ignored by setup.py develop.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optushome.com.au Sat Jul 15 06:42:46 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Sat, 15 Jul 2006 14:42:46 +1000
Subject: [Catalog-sig] Cheese Shop Tutorial
Message-ID: <200607151442.46661.richardjones@optushome.com.au>
I've added a wiki page with a basic tutorial:
http://wiki.python.org/moin/CheeseShopTutorial
This is linked from the Cheese Shop sidebar. Improvements welcome :)
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/catalog-sig/attachments/20060715/7eb3c60d/attachment.htm
From bob at redivi.com Mon Jul 17 16:17:18 2006
From: bob at redivi.com (Bob Ippolito)
Date: Mon, 17 Jul 2006 07:17:18 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
References: <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
Message-ID: <32A53630-2832-4FD0-8752-5E15B4F4A4FE@redivi.com>
On Jul 11, 2006, at 12:33 PM, Bob Ippolito wrote:
>
> On Jul 11, 2006, at 12:12 PM, Phillip J. Eby wrote:
>
>> Last, but not least, the ability was added to turn off SVN revision
>> numbers
>> or dates from the command line, so that you don't have to edit
>> setup.cfg in
>> order to issue a release.
>
> Would've been convenient if you said what the option was... took me a
> few minutes to figure out where to look. Presumably you're referring
> to the --no-svn-revision option?
It seems that --no-svn-revision is only part of the solution to
making releases less of a hassle. The problem is that setup.cfg is
still included in sdist, so any user that builds your egg from source
is going to have it tagged with ".dev_r0". This is especially
problematic for dependencies because it may download the correct
version of source, but ".dev_r0" ranks lower so it will say the
dependency is not satisfied.
What's the best solution to this, barring deleting setup.cfg from the
release branch?
-bob
From pje at telecommunity.com Mon Jul 17 23:35:32 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Jul 2006 17:35:32 -0400
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <32A53630-2832-4FD0-8752-5E15B4F4A4FE@redivi.com>
References: <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
Message-ID: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
At 07:17 AM 7/17/2006 -0700, Bob Ippolito wrote:
>It seems that --no-svn-revision is only part of the solution to
>making releases less of a hassle. The problem is that setup.cfg is
>still included in sdist, so any user that builds your egg from source
>is going to have it tagged with ".dev_r0". This is especially
>problematic for dependencies because it may download the correct
>version of source, but ".dev_r0" ranks lower so it will say the
>dependency is not satisfied.
>
>What's the best solution to this, barring deleting setup.cfg from the
>release branch?
Argh. Well, you could put an exclude in MANIFEST.in to drop the setup.cfg,
but that sucks. Maybe the right thing to do is to include the setup.cfg,
but to update it with the tag and version settings that were in effect when
the sdist was built.
From pje at telecommunity.com Tue Jul 18 01:10:09 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Jul 2006 19:10:09 -0400
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com>
References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060717190924.02447288@sparrow.telecommunity.com>
At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote:
>On Jul 17, 2006, at 2:35 PM, Phillip J. Eby wrote:
>>Argh. Well, you could put an exclude in MANIFEST.in to drop the
>>setup.cfg, but that sucks. Maybe the right thing to do is to
>>include the setup.cfg, but to update it with the tag and version
>>settings that were in effect when the sdist was built.
>
>That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision).
>Any chance of getting this in setuptools 0.6 or should I
>start adding MANIFEST.in files to the relevant projects?
That depends on how big the change ends up being. If it looks simple and
unlikely to fail, I could live with adding it to 0.6.
From bob at redivi.com Mon Jul 17 23:49:31 2006
From: bob at redivi.com (Bob Ippolito)
Date: Mon, 17 Jul 2006 14:49:31 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
References: <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
Message-ID: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com>
On Jul 17, 2006, at 2:35 PM, Phillip J. Eby wrote:
> At 07:17 AM 7/17/2006 -0700, Bob Ippolito wrote:
>> It seems that --no-svn-revision is only part of the solution to
>> making releases less of a hassle. The problem is that setup.cfg is
>> still included in sdist, so any user that builds your egg from source
>> is going to have it tagged with ".dev_r0". This is especially
>> problematic for dependencies because it may download the correct
>> version of source, but ".dev_r0" ranks lower so it will say the
>> dependency is not satisfied.
>>
>> What's the best solution to this, barring deleting setup.cfg from the
>> release branch?
>
> Argh. Well, you could put an exclude in MANIFEST.in to drop the
> setup.cfg, but that sucks. Maybe the right thing to do is to
> include the setup.cfg, but to update it with the tag and version
> settings that were in effect when the sdist was built.
That's not a bad idea (update setup.cfg on sdist w/ --no-svn-
revision). Any chance of getting this in setuptools 0.6 or should I
start adding MANIFEST.in files to the relevant projects?
-bob
From richardjones at optushome.com.au Tue Jul 18 10:55:56 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Tue, 18 Jul 2006 18:55:56 +1000
Subject: [Catalog-sig] "Package Index API" draft
In-Reply-To: <5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com>
Message-ID: <200607181855.56357.richardjones@optushome.com.au>
On Tuesday 11 July 2006 11:09, Phillip J. Eby wrote:
> This draft reflects the in-development versions of setuptools 0.7a1 and
> 0.6b4; it does not describe older setuptools versions except as noted under
> the "Backward Compatibility" section. The items described under "Backward
> Compatibility" need to be kept in PyPI until everyone in the field has
> upgraded to setuptools 0.6b4 or better. (Note that 0.6b4 is not released
> yet!)
This looks good - wanna add it to the wiki with a link on the CheeseShopDev
page please?
Richard
From pje at telecommunity.com Tue Jul 18 17:53:46 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Jul 2006 11:53:46 -0400
Subject: [Catalog-sig] "Package Index API" draft
In-Reply-To: <200607181855.56357.richardjones@optushome.com.au>
References: <5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com>
<5.1.1.6.0.20060710210032.03ac2e20@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060718115157.0204be78@sparrow.telecommunity.com>
At 06:55 PM 7/18/2006 +1000, Richard Jones wrote:
>On Tuesday 11 July 2006 11:09, Phillip J. Eby wrote:
> > This draft reflects the in-development versions of setuptools 0.7a1 and
> > 0.6b4; it does not describe older setuptools versions except as noted under
> > the "Backward Compatibility" section. The items described under "Backward
> > Compatibility" need to be kept in PyPI until everyone in the field has
> > upgraded to setuptools 0.6b4 or better. (Note that 0.6b4 is not released
> > yet!)
>
>This looks good - wanna add it to the wiki with a link on the CheeseShopDev
>page please?
I added a link to:
http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
in the "Developing the Cheese Shop" section of the CheeseShopDev page.
The above link is the home of the HTML version of the doc for the
currently-released stable version of setuptools (0.6b4 right now).
From pje at telecommunity.com Tue Jul 18 18:59:11 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Jul 2006 12:59:11 -0400
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <05D15BCF-8880-4DA1-99E0-52FE69EA1DB2@redivi.com>
References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com>
<AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com>
<5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com>
At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote:
>That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision).
>Any chance of getting this in setuptools 0.6 or should I
>start adding MANIFEST.in files to the relevant projects?
Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It
even handles date and SVN revision tags correctly, by converting them to a
single --tag-build string and disabling the other tagging options. So if
you just build from an sdist without doing anything special, you get the
exact same version the sdist was built with, regardless of how the version
was originally specified.
From pje at telecommunity.com Thu Jul 20 04:00:03 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Jul 2006 22:00:03 -0400
Subject: [Catalog-sig] Recent UI changes on PyPI
Message-ID: <5.1.1.6.0.20060719214938.02622318@sparrow.telecommunity.com>
No, don't worry, this isn't about problems with EasyInstall. :)
I just noticed that most of the package metadata has been moved to the
bottom, below the long description. However, this makes it harder to tell
at a glance who's responsible for the package, what license it has, etc.,
if the long description is more than a paragraph or two. Most of my
packages include a slice of their release notes in the long description,
usually bumping the total page size such that the other metadata no longer
appears on the first screenful.
I don't think this is a good thing, as it seems to create an impression
that the package's description is being provided by the Cheese Shop. That
is, that the Cheese Shop is somehow *responsible* for the package. Putting
the author, home page, and so on at the top previously provided a strong
hint that the description was just part of a bunch of data supplied by the
package author, and not an article or review being written by the
maintainers of the CheeseShop itself.
As you might guess from this, I think the change is bad and at least the
metadata should be put back to the top, although I do not know what the
original reason for making this change was. I don't think it makes a
difference whether the files go at the top or the bottom, but the metadata
*really* belongs up-top, and the description fields should probably be
prefixed with something like "Package Description" to help hint that this
is author-supplied info.
From pje at telecommunity.com Thu Jul 20 04:40:23 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Jul 2006 22:40:23 -0400
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks :(
Message-ID: <5.1.1.6.0.20060719223347.02027348@sparrow.telecommunity.com>
This post *is* about EasyInstall breakage in the new UI change, I'm afraid,
albeit a relatively minor one. Per the API description here:
http://peak.telecommunity.com/DevCenter/EasyInstall#backward-compatibility
EasyInstall expects a PyPI-listed package's MD5 to be found in parentheses,
not square brackets.
Whoever changed this, also didn't implement the non-visual way to indicate
MD5s (using "#md5=" link fragments, as described in paragraph 5 of the API
doc), so deployed versions of EasyInstall can no longer check md5's for
packages downloaded from PyPI.
EasyInstall only screenscrapes MD5's because PyPI doesn't include them in
its download URLs; if they are included in the URLs, then the
backward-compatibility regex can be removed, and it will work properly with
deployed versions of EasyInstall.
From richardjones at optusnet.com.au Thu Jul 20 05:45:09 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Thu, 20 Jul 2006 13:45:09 +1000
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
Message-ID: <200607200345.k6K3j9Ws000488@mail25.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060720/8bcab24e/attachment.pot
From richardjones at optusnet.com.au Thu Jul 20 05:47:09 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Thu, 20 Jul 2006 13:47:09 +1000
Subject: [Catalog-sig] Recent UI changes on PyPI
Message-ID: <200607200347.k6K3l96P027850@mail26.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060720/804517e9/attachment.asc
From pje at telecommunity.com Thu Jul 20 05:54:52 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Jul 2006 23:54:52 -0400
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <200607200345.k6K3j9Ws000488@mail25.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
At 01:45 PM 7/20/2006 +1000, richardjones at optusnet.com.au wrote:
> > Phillip J. Eby <pje at telecommunity.com> wrote:
> > EasyInstall expects a PyPI-listed package's MD5 to be found in parentheses,
> > not square brackets.
>
>Crap, sorry, I thought I'd caught all the changes, but I missed this one.
>I'll fix it ASAP.
Note that if you add the #md5= anchor described in paragraph 5 of the API
spec, neither of us will need to maintain this bit of
backward-compatibility hackery any more. I'd love to have one more bit of
screen-scraping I can remove from 0.6c1.
I know you've mentioned before that you're not sure what browsers will do
with it, but the only way to find that out for sure is to give it a try. ;-)
Seriously, if browsers had a problem with downloads that have trailing junk
on their URLs, then a whole lot of SourceForge projects, porn sites, and
other dynamic downloading applications would be quite seriously out of
luck. :)
From pje at telecommunity.com Thu Jul 20 06:35:05 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Jul 2006 00:35:05 -0400
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <200607200345.k6K3j9Ws000488@mail25.syd.optusnet.com.au>
Message-ID: <5.1.1.6.0.20060720003006.02622318@sparrow.telecommunity.com>
At 01:45 PM 7/20/2006 +1000, richardjones at optusnet.com.au wrote:
> > Phillip J. Eby <pje at telecommunity.com> wrote:
> > EasyInstall expects a PyPI-listed package's MD5 to be found in parentheses,
> > not square brackets.
>
>Crap, sorry, I thought I'd caught all the changes, but I missed this one.
>I'll fix it ASAP.
FYI, the fix doesn't work either, as there's still a title="" attribute in
there. Please see the regex in the API doc.
(Yes, it's a terribly crufty regex; it was always intended as merely a
backward-compatibility stopgap until PyPI started supporting #md5 anchors.)
From constant.beta at gmail.com Thu Jul 20 12:19:58 2006
From: constant.beta at gmail.com (=?ISO-8859-2?Q?Micha=B3_Kwiatkowski?=)
Date: Thu, 20 Jul 2006 12:19:58 +0200
Subject: [Catalog-sig] Recent UI changes on PyPI
In-Reply-To: <200607200347.k6K3l96P027850@mail26.syd.optusnet.com.au>
References: <200607200347.k6K3l96P027850@mail26.syd.optusnet.com.au>
Message-ID: <5e8b0f6b0607200319s48690cafpaa324d19d56f8a6c@mail.gmail.com>
On 7/20/06, richardjones at optusnet.com.au <richardjones at optusnet.com.au> wrote:
> Yes, with longer descriptions the "content" of the page can
> be lost down the bottom. What do other people think? I'm
> leaning towards changing it back.
I agree with Phillip, metadata should be on the top.
Cheers,
mk
--
. o . >> http://joker.linuxstuff.pl <<
. . o It's easier to get forgiveness for being wrong
o o o than forgiveness for being right.
From richardjones at optusnet.com.au Fri Jul 21 02:16:19 2006
From: richardjones at optusnet.com.au (Richard Jones)
Date: Fri, 21 Jul 2006 10:16:19 +1000
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
Message-ID: <200607211016.19235.richardjones@optusnet.com.au>
On Thursday 20 July 2006 13:54, Phillip J. Eby wrote:
> Seriously, if browsers had a problem with downloads that have trailing junk
> on their URLs, then a whole lot of SourceForge projects, porn sites, and
> other dynamic downloading applications would be quite seriously out of
> luck. :)
Is there really prior art for this? Sourceforge doesn't do this, AFAIK.
Richard
From richardjones at optusnet.com.au Fri Jul 21 02:19:13 2006
From: richardjones at optusnet.com.au (Richard Jones)
Date: Fri, 21 Jul 2006 10:19:13 +1000
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <5.1.1.6.0.20060720003006.02622318@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060720003006.02622318@sparrow.telecommunity.com>
Message-ID: <200607211019.13422.richardjones@optusnet.com.au>
On Thursday 20 July 2006 14:35, Phillip J. Eby wrote:
> At 01:45 PM 7/20/2006 +1000, richardjones at optusnet.com.au wrote:
> > > Phillip J. Eby <pje at telecommunity.com> wrote:
> > > EasyInstall expects a PyPI-listed package's MD5 to be found in
> > > parentheses, not square brackets.
> >
> >Crap, sorry, I thought I'd caught all the changes, but I missed this one.
> >I'll fix it ASAP.
>
> FYI, the fix doesn't work either, as there's still a title="" attribute in
> there. Please see the regex in the API doc.
OK, I've removed the title attrs.
Are we convinced that screen-scraping is bad yet?
Richard
From pje at telecommunity.com Fri Jul 21 05:15:53 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Jul 2006 23:15:53 -0400
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <200607211016.19235.richardjones@optusnet.com.au>
References: <5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
<5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060720231014.026223f0@sparrow.telecommunity.com>
At 10:16 AM 7/21/2006 +1000, Richard Jones wrote:
>On Thursday 20 July 2006 13:54, Phillip J. Eby wrote:
> > Seriously, if browsers had a problem with downloads that have trailing junk
> > on their URLs, then a whole lot of SourceForge projects, porn sites, and
> > other dynamic downloading applications would be quite seriously out of
> > luck. :)
>
>Is there really prior art for this? Sourceforge doesn't do this, AFAIK.
I'm referring to query strings, actually, with respect to prior art.
A few minutes experimentation shows that the following browsers work just
fine with #md5 links:
* Mozilla 1.7
* Firefox 1.0
* Internet Explorer 6
* Lynx 2.8
* Opera 8.5.4
I think that's a pretty good indication that most web browsers know how to
handle URI fragment identifiers in compliance with the RFCs. :) I would
be surprised if there are any other browsers in substantial use on PyPI,
with the exception of Safari.
From pje at telecommunity.com Fri Jul 21 05:27:40 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Jul 2006 23:27:40 -0400
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <200607211019.13422.richardjones@optusnet.com.au>
References: <5.1.1.6.0.20060720003006.02622318@sparrow.telecommunity.com>
<5.1.1.6.0.20060720003006.02622318@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060720231559.03e77ea0@sparrow.telecommunity.com>
At 10:19 AM 7/21/2006 +1000, Richard Jones wrote:
>On Thursday 20 July 2006 14:35, Phillip J. Eby wrote:
> > At 01:45 PM 7/20/2006 +1000, richardjones at optusnet.com.au wrote:
> > > > Phillip J. Eby <pje at telecommunity.com> wrote:
> > > > EasyInstall expects a PyPI-listed package's MD5 to be found in
> > > > parentheses, not square brackets.
> > >
> > >Crap, sorry, I thought I'd caught all the changes, but I missed this one.
> > >I'll fix it ASAP.
> >
> > FYI, the fix doesn't work either, as there's still a title="" attribute in
> > there. Please see the regex in the API doc.
>
>OK, I've removed the title attrs.
>
>Are we convinced that screen-scraping is bad yet?
I never argued that this particular bit was *good*. As I said, it was
always a stopgap measure until PyPI supported a better way to retrieve the
MD5's, preferably as part of the URLs. I don't recall that you've never
proposed any alternative, and I seem to recall that not even the XML-RPC
API offers this information.
Please note that I proposed the use of #md5 links well before I implemented
the screen scraping, and asked for feedback. Your only feedback was to
inquire what happened in browsers, to which I replied -- and that was the
end of the thread. What else was I supposed to do, beat you about the head
and shoulders? At the time, you said you were busy.
However, since you now seem to have some time, maybe it would be a good
time to implement some of the things I suggested you implement a year ago
so that screen scraping wouldn't be necessary. ;)
From strawman at astraw.com Fri Jul 21 20:20:23 2006
From: strawman at astraw.com (Andrew Straw)
Date: Fri, 21 Jul 2006 11:20:23 -0700
Subject: [Catalog-sig] [Distutils] setuptools 0.6b4 released
In-Reply-To: <5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com> <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com> <5.1.1.6.0.20060711145922.03ca5aa0@sparrow.telecommunity.com> <AAE05165-EB94-4612-89A9-9FACC5819F67@redivi.com> <5.1.1.6.0.20060717173328.03393048@sparrow.telecommunity.com>
<5.1.1.6.0.20060718125558.0203e0c8@sparrow.telecommunity.com>
Message-ID: <44C11AE7.3030708@astraw.com>
Phillip J. Eby wrote:
> At 02:49 PM 7/17/2006 -0700, Bob Ippolito wrote:
>
>> That's not a bad idea (update setup.cfg on sdist w/ --no-svn- revision).
>> Any chance of getting this in setuptools 0.6 or should I
>> start adding MANIFEST.in files to the relevant projects?
>>
>
> Okay, it's in the trunk now as of 0.7a1dev-r50702 and 0.6c1dev-r50703. It
> even handles date and SVN revision tags correctly, by converting them to a
> single --tag-build string and disabling the other tagging options. So if
> you just build from an sdist without doing anything special, you get the
> exact same version the sdist was built with, regardless of how the version
> was originally specified.
>
I'm glad this is being worked on. But a related issue is still biting me
with setuptools 0.6c1 in my stdeb package (which builds debian source
packages from unmodified setup.py scripts) :
Any distutils commands using "self.distribution.get_version()" still get
tagged (at least with the svn revision), even if they're being built
from the sdist-generated .tar.gz package.
Not knowing the innards of setuptools very well, one idea would be to
add something to the .egg-info built by sdist that tells future runs of
setuptools not to add tags. This keeps setup.cfg from getting modified
but still has the right effect. There's probably a flaw I haven't
thought of, though...
Cheers!
Andrew
From richardjones at optusnet.com.au Sat Jul 22 06:36:05 2006
From: richardjones at optusnet.com.au (Richard Jones)
Date: Sat, 22 Jul 2006 14:36:05 +1000
Subject: [Catalog-sig] FYI: UI change also broke EasyInstall MD5 checks
:(
In-Reply-To: <5.1.1.6.0.20060720231014.026223f0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060719235028.02622318@sparrow.telecommunity.com>
<5.1.1.6.0.20060720231014.026223f0@sparrow.telecommunity.com>
Message-ID: <200607221436.05554.richardjones@optusnet.com.au>
On Friday 21 July 2006 13:15, Phillip J. Eby wrote:
> A few minutes experimentation shows that the following browsers work just
> fine with #md5 links:
>
> * Mozilla 1.7
> * Firefox 1.0
> * Internet Explorer 6
> * Lynx 2.8
> * Opera 8.5.4
>
> I think that's a pretty good indication that most web browsers know how to
> handle URI fragment identifiers in compliance with the RFCs. :) I would
> be surprised if there are any other browsers in substantial use on PyPI,
> with the exception of Safari.
OK, I'll enable it. Let's see how it goes...
Richard
From ianb at colorstudy.com Tue Jul 25 01:45:15 2006
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 24 Jul 2006 18:45:15 -0500
Subject: [Catalog-sig] Search gone...
Message-ID: <44C55B8B.3010706@colorstudy.com>
In the UI refactoring the link to search the Cheese Shop seems to have
disappeared (except to search through Google, which isn't quite the same).
Can we get that back? And also a search box on the front page
(preferably one that searches for text in any of title, keywords,
description).
Also, the Trove classifiers are served as text/html now instead of
text/plain: http://www.python.org/pypi?%3Aaction=list_classifiers
--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
From ianb at colorstudy.com Tue Jul 25 01:47:46 2006
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 24 Jul 2006 18:47:46 -0500
Subject: [Catalog-sig] Zope framework category
Message-ID: <44C55C22.2060802@colorstudy.com>
I'd like a Zope framework category, especially since Zope is starting to
consume typical distutils packages so they are showing up in the
Cheese Shop. But I'm not sure if there should separate Zope 2 and Zope
3 sections or not, or... I'm not sure. Does anyone here have opinions?
With Zope Five the 2/3 distinction becomes fuzzy.
--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
From fdrake at gmail.com Tue Jul 25 03:54:54 2006
From: fdrake at gmail.com (Fred Drake)
Date: Mon, 24 Jul 2006 21:54:54 -0400
Subject: [Catalog-sig] Zope framework category
In-Reply-To: <44C55C22.2060802@colorstudy.com>
References: <44C55C22.2060802@colorstudy.com>
Message-ID: <9cee7ab80607241854l4fa093ddnee9db5699efed951@mail.gmail.com>
On 7/24/06, Ian Bicking <ianb at colorstudy.com> wrote:
> I'd like a Zope framework category, especially since Zope is starting to
> consume typical distutils packages so they are showing up in the
> Cheese Shop. But I'm not sure if there should separate Zope 2 and Zope
> 3 sections or not, or... I'm not sure. Does anyone here have opinions?
I think Zope 2 and Zope 3 should be considered completely separate
frameworks, though Jim Fulton's long-term vision suggests they'll be
more closely related in the future. They will likely remain distinct
even then, since the audiences are a bit different.
> With Zope Five the 2/3 distinction becomes fuzzy.
I don't think it does. It just means that *sometimes* Zope 2 users
will be interested in Zope 3 components. Components may appear that
fit in both categories -- that's ok.
-Fred
--
Fred L. Drake, Jr. <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
From richardjones at optusnet.com.au Tue Jul 25 06:40:42 2006
From: richardjones at optusnet.com.au (richardjones at optusnet.com.au)
Date: Tue, 25 Jul 2006 14:40:42 +1000
Subject: [Catalog-sig] Search gone...
Message-ID: <200607250440.k6P4egXX002121@mail17.syd.optusnet.com.au>
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20060725/d79c4d41/attachment.pot
From jim at zope.com Fri Jul 28 20:36:57 2006
From: jim at zope.com (Jim Fulton)
Date: Fri, 28 Jul 2006 14:36:57 -0400
Subject: [Catalog-sig] Zope framework category
In-Reply-To: <44C55C22.2060802@colorstudy.com>
References: <44C55C22.2060802@colorstudy.com>
Message-ID: <E27AFBBD-5A1F-410D-8F9A-1D16EBFC6928@zope.com>
On Jul 24, 2006, at 7:47 PM, Ian Bicking wrote:
> I'd like a Zope framework category, especially since Zope is
> starting to
> consume typical distutils packages so they are showing up in the
> Cheese Shop. But I'm not sure if there should separate Zope 2 and
> Zope
> 3 sections or not, or... I'm not sure. Does anyone here have
> opinions?
> With Zope Five the 2/3 distinction becomes fuzzy.
I think a Zope 2 framework category makes sense.
Zope 3 is a collection of much smaller and independent frameworks.
Each should be handled separately as the need arises.
I suppose that as people upload distributions that are meant to plug
into some framework, they should request that the framework be
registered.
Jim
--
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
From richardjones at optushome.com.au Mon Jul 31 09:45:07 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 31 Jul 2006 17:45:07 +1000
Subject: [Catalog-sig] Zope framework category
In-Reply-To: <E27AFBBD-5A1F-410D-8F9A-1D16EBFC6928@zope.com>
References: <44C55C22.2060802@colorstudy.com>
<E27AFBBD-5A1F-410D-8F9A-1D16EBFC6928@zope.com>
Message-ID: <200607311745.07897.richardjones@optushome.com.au>
On Saturday 29 July 2006 04:36, Jim Fulton wrote:
> I suppose that as people upload distributions that are meant to plug
> into some framework, they should request that the framework be
> registered.
That's how I'm handling it at the moment :)
I've added the info about requesting new packages to the Shop's Tutorial page
(it was previously only on the Developers page).
Richard
From richardjones at optushome.com.au Mon Jul 31 11:14:03 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 31 Jul 2006 19:14:03 +1000
Subject: [Catalog-sig] Search gone...
In-Reply-To: <44C55B8B.3010706@colorstudy.com>
References: <44C55B8B.3010706@colorstudy.com>
Message-ID: <200607311914.03566.richardjones@optushome.com.au>
On Tuesday 25 July 2006 09:45, Ian Bicking wrote:
> Also, the Trove classifiers are served as text/html now instead of
> text/plain: http://www.python.org/pypi?%3Aaction=list_classifiers
This should be fixed the next time the server restarts its processes.
Richard
From jjl at pobox.com Mon Jul 31 13:46:00 2006
From: jjl at pobox.com (John J Lee)
Date: Mon, 31 Jul 2006 12:46:00 +0100 (GMT Standard Time)
Subject: [Catalog-sig] Zope framework category
In-Reply-To: <200607311745.07897.richardjones@optushome.com.au>
References: <44C55C22.2060802@colorstudy.com>
<E27AFBBD-5A1F-410D-8F9A-1D16EBFC6928@zope.com>
<200607311745.07897.richardjones@optushome.com.au>
Message-ID: <Pine.WNT.4.64.0607311244280.3020@shaolin>
On Mon, 31 Jul 2006, Richard Jones wrote:
> On Saturday 29 July 2006 04:36, Jim Fulton wrote:
>> I suppose that as people upload distributions that are meant to plug
>> into some framework, they should request that the framework be
>> registered.
>
> That's how I'm handling it at the moment :)
>
> I've added the info about requesting new packages to the Shop's Tutorial page
> (it was previously only on the Developers page).
Do we really have to keep "Framework :: TruboGears :: Applications"?
(note the spelling of "TruboGears")
John
From richardjones at optushome.com.au Mon Jul 31 13:56:24 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Mon, 31 Jul 2006 21:56:24 +1000
Subject: [Catalog-sig] Zope framework category
In-Reply-To: <Pine.WNT.4.64.0607311244280.3020@shaolin>
References: <44C55C22.2060802@colorstudy.com>
<200607311745.07897.richardjones@optushome.com.au>
<Pine.WNT.4.64.0607311244280.3020@shaolin>
Message-ID: <200607312156.24434.richardjones@optushome.com.au>
On Monday 31 July 2006 21:46, you wrote:
> On Mon, 31 Jul 2006, Richard Jones wrote:
> > On Saturday 29 July 2006 04:36, Jim Fulton wrote:
> >> I suppose that as people upload distributions that are meant to plug
> >> into some framework, they should request that the framework be
> >> registered.
> >
> > That's how I'm handling it at the moment :)
> >
> > I've added the info about requesting new packages to the Shop's Tutorial
> > page (it was previously only on the Developers page).
>
> Do we really have to keep "Framework :: TruboGears :: Applications"?
> (note the spelling of "TruboGears")
No, and it's gone now - no-one was using it :)
Richard