"Sridhar Ratnakumar" <sridharr@activestate.com> writes:

What are the possible use-cases (by the tools) of these new fields would be?

One of the important consumers of package metadata is PyPI. I imagine these fields will be presented to the user along with other data about the package. -- \ “I got an answering machine for my phone. Now when someone | `\ calls me up and I'm not home, they get a recording of a busy | _o__) signal.” —Steven Wright | Ben Finney

On Mon, Nov 23, 2009 at 10:20 AM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:

On Sun, Nov 22, 2009 at 04:23:33PM -0800, Sridhar Ratnakumar wrote:

...

On Sun, 22 Nov 2009 14:52:11 -0800, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

Hello As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2: "Repository-URL"    A string containing the URL for the package's repository.   Example:        http://svn.python.org/projects/python/trunk/ "Repository-Browse-URL"     A string containing the URL for the package's browsable repository.   Example:      http://svn.python.org/view/python/trunk "Bug-Tracker-URL"   A string containing the URL for the package's bug tracker   Example:      http://bugs.python.org/

What are the possible use-cases (by the tools) of these new fields would be? I can imagine Repository-URL to be of use (eg: pip install IPython==dev), but have no idea why the other two should be added.

Same as for Author and Author-email etc: more information for the (human) users.  I think this is very valuable when browsing packages on e.g. PyPI.

Yes. What is happening right now is that people are adding sometimes these info in the long_description, but there's no way to display these informations at PyPI in a specific location on the page, like a portlet for example. PyPI is extracting urls out of long_description to put them in the simple index for tools like easy_install to process them, but that's it. If you look at plone.org (which is PyPI -compatible) : http://plone.org/products/jyu.pathkey Each project can display in the "project resources" some of these links, but that's currently done manually. Having them in the metadata will make it easier for PyPI and other similar servers to get them and display them, and will encourage the developers to use them from within their setup.py This proposal was triggered in the discussion about the rating/commenting system : someone said that we would have much more value in displaying these links, for people to find the code and the tracker. Tarek

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers <chris@simplistix.co.uk> wrote:

...

Change-Log-URL

I don't see a lot of projects with a changelog file available online, rather a changelog file in the release in CHANGELOG and sometimes a changelog displayed in the long_description field, so it's shown at PyPI.

I think that's a poor practice; the long description shouldn't need to change every release and shouldn't include a changelog.

Can you detail on this one ?

I would like to see more projects making their changelog a separate document, so it could be referenced simply and separately from other information about the project. This field would encourage that by making it more visible.

...

Documentation-URL

How would this one be different from Home-page ?

The home page is a “first impressions” page. It usually doesn't jump straight into a user's guide and technical documentation and interface specification. Many projects have a separate index page specifically for documentation; the URL of that page would be the value for this field. -- \ “Value your freedom or you will lose it, teaches history. | `\ “Don't bother us with politics,” respond those who don't want | _o__) to learn.” —Richard Stallman, 2002 | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

How about making it "Change-Log" in that case, and have the content of the changelog directly added in the metadata ? Rather than an url.

That might be good, if the changelog was in a separate file. It's quite commonly a very long text file, as it retains the change history of the code base indefinitely. -- \ “Shepherds … look after their sheep so they can, first, fleece | `\ them and second, turn them into meat. That's much more like the | _o__) priesthood as I know it.” —Christopher Hitchens, 2008-10-29 | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sun, Nov 29, 2009 at 2:19 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

[having the changelog in the package metadata] might be good, if the changelog was in a separate file. It's quite commonly a very long text file, as it retains the change history of the code base indefinitely.

Yes, and if it's reStructuredText based, we will be able to do what people do actually, by integrating a reST-based CHANGELOG file at the end of long_description:

Let's not get ahead of ourselves :-) Asking developers to declare where their changelog file is will be much easier to get than asking them to change the format of that file. Stick to a simple standard of “this file contains the changelog”, which will let people simply move their existing changelog of whatever format into that file. Having it in a predictable location would be a big step forward, and an easy one to enforce. -- \ “Our products just aren't engineered for security.” —Brian | `\ Valentine, senior vice-president of Microsoft Windows | _o__) development, 2002 | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sun, Nov 29, 2009 at 3:05 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

Stick to a simple standard of “this file contains the changelog”, which will let people simply move their existing changelog of whatever format into that file. Having it in a predictable location would be a big step forward, and an easy one to enforce.

That doesn't differ from what we have today with long_description:

That ignores what I thought we'd just agreed in this thread: that the changelog should be *in a separate file*. Putting it into one field among many ignores the fact that it's often long, and growing every release. -- \ “It ain't so much the things we don't know that get us in | `\ trouble. It's the things we know that ain't so.” —Artemus Ward | _o__) (1834–1867), U.S. journalist | Ben Finney

Tarek Ziadé <ziade.tarek@gmail.com> writes:

On Sun, Nov 29, 2009 at 10:32 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

[…] I thought we'd just agreed in this thread: that the changelog should be *in a separate file*. Putting it into one field among many ignores the fact that it's often long, and growing every release.

It can be a file of course in your distribution, but at the end, it has to be a field in the metadata,

I don't understand the distinction you're making here. What is the difference between “in your distribution” versus “in the end”? Are you making a distinction between “source distribution (sdist)” versus “installed package on the end user's computer”? Or something else? -- \ “Dvorak users of the world flgkd!” —Kirsten Chevalier, | `\ rec.humor.oracle.d | _o__) | Ben Finney

On Mon, Nov 30, 2009 at 1:20 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

Tarek Ziadé <ziade.tarek@gmail.com> writes:

...

On Sun, Nov 29, 2009 at 10:32 PM, Ben Finney <ben+python@benfinney.id.au> wrote:

...

[…] I thought we'd just agreed in this thread: that the changelog should be *in a separate file*. Putting it into one field among many ignores the fact that it's often long, and growing every release.

It can be a file of course in your distribution, but at the end, it has to be a field in the metadata,

I don't understand the distinction you're making here. What is the difference between “in your distribution” versus “in the end”?

You have to make a distinction between the "long_description" option in setup.py, and the "Description" metadata field, that lands in the PKG-INFO file (==PEP 345 fields) The PKG-INFO file, that contains all the metadata, is built by a call to "python setup.py sdist" for example , is shipped within your distribution, and read for instance by PyPI. IOW, if we are going to add a distinct field for "Changelog", this will not be a file path, but an URL *or* a raw text exactly like the "Description" field. Having a file for your changelog, that you are reading and using to fill a metadata field is just a implementation detail. Regards Tarek

Tarek Ziadé wrote:

On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers <chris@simplistix.co.uk> wrote: [..]

...

Yes to all of the above, could we also please have:

Mailing-List-URL

+1

...

Change-Log-URL

I don't see a lot of projects with a changelog file available online,

...except every single Zope-related egg ;-) These currently get put in the long-description, often up at the top, usually making the actual description totally unreadable. Having the option to put a url in a standard field would encourage people to just put a link to the changelog.

Can you detail on this one ?

...

Documentation-URL

How would this one be different from Home-page ?

Take my tiny errorlogger package, the homepage url is here: http://www.simplistix.co.uk/software/python/errorhandler Documentation is here: http://packages.python.org/errorhandler/ These are optional fields, which I'd like to see PyPI display if they're present, but we need the ability to add them first! :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Eric Smith wrote:

...

Having the option to put a url in a standard field would encourage people to just put a link to the changelog.

Why does it need to be in a standard field?

The same reason we have standard metadata fields like homepage, etc: so users of PyPI get a consistent location in which to find the information they're looking for, no matter who the package author is or how they choose to layout their long_description. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On Mon, Nov 30, 2009 at 8:17 PM, Ian Bicking <ianb@colorstudy.com> wrote:

On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2:

"Repository-URL"    A string containing the URL for the package's repository.   Example:        http://svn.python.org/projects/python/trunk/

I like this in concept, but it's a bit vague as just a URL.  What kind of repository is it?  Sniffing is possible, but doesn't work that well.  For pip I require something like hg+http://... (with exceptions for schemes that are self-describing, like svn: and git:).

I didn't think that Pip would use this field to do some automatic process, I think the use case was to simply display that url at PyPI. OTHO, Repository-Browse-URL already plays that role, so it would be interesting to provide in "Repository-URL", urls that could be processed by installers. Do you have a particular use case in mind ?

It's also unclear whether it should point to the trunk, a branch, or a tag (related to the release).  As a URL that's only even possible for svn, as git/hg/bzr all point to the repository and URLs can't point to revisions, tags, etc.  (pip has some extensions to the URL format to accommodate these, but they aren't formally defined at this time).

Good point. I think this would end up being the trunk/default path *or* the root of their repository, because I doubt developers will update this field for every release they have. And, as you said, in mercurial for instance, we would need to "hg up some_tag" to have a similar feature than svn branches. So at the end, maybe we could just keep "Repository-Browse-URL" and drop the other one unless we can come up with some use cases for automatic processing of the url. Tarek.

On Mon, Nov 30, 2009 at 4:30 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2:

"Repository-URL" A string containing the URL for the package's repository. Example: http://svn.python.org/projects/python/trunk/

I like this in concept, but it's a bit vague as just a URL. What kind of repository is it? Sniffing is possible, but doesn't work that well. For pip I require something like hg+http://... (with exceptions for schemes

On Mon, Nov 30, 2009 at 8:17 PM, Ian Bicking <ianb@colorstudy.com> wrote: that

...

are self-describing, like svn: and git:).

I didn't think that Pip would use this field to do some automatic process, I think the use case was to simply display that url at PyPI.

OTHO, Repository-Browse-URL already plays that role, so it would be interesting to provide in "Repository-URL", urls that could be processed by installers.

Do you have a particular use case in mind ?

I don't really. There is an active use case for links like: http://repo-location/MyPackage/trunk#egg=MyPackage-dev which are used to get the dev version. These links should go to a svn index (which both easy_install and pip understand specifically) or to some automatically-generated tarball (which I believe most of the other VCSs generate through their popular frontends, e.g., http://bitbucket.org/ianb/webob/get/tip.zip). If something more like that was added (more like Trunk-URL), that would be genuinely useful. Barring that, I'd recommend removing the Repository-URL field from the recommendation until we have a clearer use case. Right now, the better use case would be a field giving the checkout command, e.g.,: Repository-Checkout: hg clone http://bitbucket.org/ianb/webob But I don't think that is particularly useful, only that it is more useful than Repository-URL. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker

On Tue, Dec 1, 2009 at 10:11 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

The value I see in that field is that:

- it helps developers structurize a bit their project description. e.g. letting them separate  the Description of the package from the changelog if any.

Package creators can already do this by including a link to the change log if they so choose.

- it will let PyPI separate the front page of the project from its changelog, e.g. in this page:  http://pypi.python.org/pypi/zc.buildout, instead of having the changelog here, it could be a link  to another page for the project.

Again, package creators can already do this.

How does that sounds ?

I don't understand the value of this proposal. We don't need more overhead for package authors to wade through, even if they end up ignoring it. No... *especially* if they end up ignoring it. -- Benji York

On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote: [snip a reiteration of the perceived benefits]

So I guess you are also against the addition of "Repository-URL" and "Bug-Tracker-URL" too ? Since those can be links in the long_description as well.

I am. Don't get me wrong, I don't think these proposals will kill kittens or anything, it's just that every addition has a cost. I (apparently) see that cost as being higher than others do, therefore I feel the small benefit is swamped by the cost. -- Benji York

On Tue, Dec 1, 2009 at 12:04 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

What is the cost you are seeing in having new additional fields ?

The cognitive load of package authors having to read, understand, and then provide or ignore the metadata. There is also the cost to implementors of PyPI replacements/mirrors/other tools.

Existing projects can leave everything in their long_description if they want, nothing is lost.

Nothing we currently have is lost, however we put a small damper on future gains. -- Benji York

2009/12/1 Eric Smith <eric@trueblade.com>:

Benji York wrote:

...

On Tue, Dec 1, 2009 at 10:50 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote: [snip a reiteration of the perceived benefits]

...

So I guess you are also against the addition of "Repository-URL" and "Bug-Tracker-URL" too ? Since those can be links in the long_description as well.

I am.

Don't get me wrong, I don't think these proposals will kill kittens or anything, it's just that every addition has a cost.  I (apparently) see that cost as being higher than others do, therefore I feel the small benefit is swamped by the cost.

I agree with Benji that adding new fields has a cost and should be avoided. Just because we can add structure doesn't mean we should.

In the specific case of Repository-URL and Bug-Tracker-URL, the idea was that PyPI would change its behavior if these fields were present (by not using the rating system). So I'm okay with those. But for other fields, like Repository-Checkout and Change-Log-URL, I haven't seen a use case yet.

Ok so, if everyone agrees on those two, I am proposing to add for now : - Repository-Browse-URL : url to a web-browseable repository (no distinction whether its the trunk, a branch etc) - Bug-Tracker-URL : url to the web tracker and skip the others, as they seem controversial. I'll post the proposal at Catalog-SIG as well, Tarek

Floris Bruynooghe <floris.bruynooghe@gmail.com> writes:

I'm still uneasy about forcing everyone to have a browsable URL

I think the intention is that all of these new fields are optional. No-one would be forced to put any of them into their package. -- \ “Every sentence I utter must be understood not as an | `\ affirmation, but as a question.” —Niels Bohr | _o__) | Ben Finney

On Tue, Dec 1, 2009 at 9:11 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

On Tue, Dec 1, 2009 at 4:19 AM, Ian Bicking <ianb@colorstudy.com> wrote: [..]

...

If something more like that was added (more like Trunk-URL), that would be genuinely useful. Barring that, I'd recommend removing the Repository-URL field from the recommendation until we have a clearer use case. Right now, the better use case would be a field giving the checkout command, e.g.,: Repository-Checkout: hg clone http://bitbucket.org/ianb/webob But I don't think that is particularly useful, only that it is more useful than Repository-URL.

Here's a proposal then, that seems to synthetize what people have been saying:

Let's drop "Repository-Browse-URL" and keep a single "Repository-URL" field, which is a free URL that can take any URL form. e.g. a browsable url, or a git/hg url etc.

I prefer Repository-Browse-URL, as it is more explicit in its use: it's a link that someone using a browser can usefully click through to. I expect it will be displayed as such on PyPI. So this link is good: http://github.com/cloudkick/libcloud And this link is bad: git://github.com/cloudkick/libcloud.git A similar distinction exists for code.google.com projects. Now for "Change-Log" vs "Change-Log-URL", I think the first one is

better, because that's what people are already doing in their packages (when they add their changelog at the end of their long_description option), and it's not hard for PyPI to store it and display it, besides Description.

This seems fine to me. Is ReST allowed? Could one potentially just do: `Changes <http://myproject.com/changes.html>`_ ? And then essentially the changelog would be a single link? I'm not sure if that's a good idea. Would it be too vague to say that if the change log is a single URL that PyPI should link directly through to the change log instead of displaying the link? (The exact UI for PyPI hasn't been proposed, but if it's something like a tab with changes, that tab could actually link offsite) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker

On Mon, Nov 30, 2009 at 01:17:18PM -0600, Ian Bicking wrote:

On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:

...

As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2:

"Repository-URL" A string containing the URL for the package's repository. Example: http://svn.python.org/projects/python/trunk/

I like this in concept, but it's a bit vague as just a URL. What kind of repository is it? Sniffing is possible, but doesn't work that well. For pip I require something like hg+http://... (with exceptions for schemes that are self-describing, like svn: and git:).

Very good point, the reason I proposed these was because I know Debian uses them but I failed to properly look at their experience. Actually turns out that currently they use: Vcs-Svn, Vcs-Git, etc. as well as Vcs-Browser. But searching around they are also having this discussion and looking that changing that because it doens't scale (needing a new field for each new VCS...). How about two fields? Repository-VCS and Repository-URL where the first is something like "git", "hg", "svn" etc.

It's also unclear whether it should point to the trunk, a branch, or a tag (related to the release). As a URL that's only even possible for svn, as git/hg/bzr all point to the repository and URLs can't point to revisions, tags, etc. (pip has some extensions to the URL format to accommodate these, but they aren't formally defined at this time).

It's probably fine to leave that up to the developer and not specify it, PyPI's just needs to display it to the user. Whatever the developer sees fit. Coming up with a schema that is and future proof and works with all VCSes out there will be pretty hard. Regards Floris