[Distutils] add "sourceURL" to the metadata 3.0 PEP.

[Wes Turner]

On Fri, Mar 24, 2017 at 4:26 AM, Wes Turner <wes.turner at gmail.com> wrote:

>
>
> On Thu, Mar 23, 2017 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 24 March 2017 at 13:24, Wes Turner <wes.turner at gmail.com> wrote:
>>
>>>
>>> On Thu, Mar 23, 2017 at 2:23 AM, Nick Coghlan <ncoghlan at gmail.com>
>>> wrote:
>>>
>>>> This means we're not going to be automating the process of getting an
>>>> editable checkout in the core tools any time soon - there are already 100k+
>>>> published packages on PyPI, so anyone that seriously wants to do this is
>>>> going to have to write their own client utility that attempts to infer it
>>>> from the metadata that already exists (probably by building atop distlib,
>>>> since that has all the necessary pieces to read the various metadata
>>>> formats, both remote and local).
>>>>
>>>> Future metadata extensions might help to make such a tool more
>>>> reliable, but *requiring* metadata changes to be made first will just make
>>>> it useless (since it wouldn't work at all until after publishers start
>>>> publishing the new metadata, which would mean waiting years before it
>>>> covered a reasonable percentage of PyPI).
>>>>
>>>
>>> Here's a way to define Requirements and a RequirementsMap with
>>> additional data:
>>> https://github.com/westurner/pyleset/blob/57140bcef5/setup.py#L118
>>>
>>> It creates a directory full of requirements[.dev].txt files:
>>> https://github.com/westurner/pyleset/tree/57140bce/requirements
>>>
>>> Additional metadata in Pipfile would be nice;
>>> but it would be fairly easy to send a PR to:
>>>
>>> BLD: setup.py: add the canonical sourceURL
>>>
>>
>> PEP 426 already has a source URL field: https://www.python.org/dev/pep
>> s/pep-0426/#source-url
>>
>> It's just not required to be a *version* control reference - it's free to
>> be a reference to a tarball or zip archive instead (just not a reference to
>> the sdist itself, since that will contain a copy of the metadata file).
>>
>> However, independently of that concern, "send a PR" is only the first
>> step in updating published metadata to accommodate tasks that package
>> *consumers* want to perform:
>>
>> 1. Someone has to write and submit the upstream project patch
>> 2. The publisher has to review and accept the change
>> 3. The publisher has to publish the new release
>> 4. Rinse-and-repeat for dozens/hundreds/thousands of projects, depending
>> on the scope of what you care about
>>
>> So the lesson we've learned is that for consumer tasks it's *always*
>> better to start by asking "How can I best achieve my objective without
>> asking publishers to change *anything*?".
>>
>> In the case of finding version control references, that's a matter of:
>>
>> - looking at Download-URL and Project-URL entries for links that "look
>> like" version control references
>> - if that doesn't turn up anything useful, scan the long description
>> - once you have a repository reference, look for promising tag names (if
>> the link didn't nominate a specific commit)
>>
>> On the *publisher* side, the equivalent question is "Can publishers
>> already choose to publish this metadata without having to wait for a
>> metadata update?".
>>
>

>
>> In this case, the answer is yes, due to the "Project-URL" field: anyone
>> is free to push for the adoption of a particular convention for tagging the
>> exact version control reference needed for "pip -e" to retrieve the
>> corresponding source code.
>>
>
https://www.google.com/search?q=python+pep+"project-url" (!)

https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use

Project-URL (multiple-use)
> A string containing a browsable URL for the project and a label for it,
> separated by a comma.
> Example:
>     Bug Tracker, http://bitbucket.org/tarek/distribute/issues/
> The label is a free text limited to 32 signs.


- Predicate URIs are often longer than 32 signs.

  (pypi:pkgname, label, URL)

  # RDF triples
  (subject, predicate, object)
  (URI, URI, *)

  # RDF quads
  (graph, s, p, o)
  (URI, URI, URI, *)


>From http://legacy.python.org/dev/peps/pep-0426/#source-url:


> For version control references, the VCS+protocol scheme SHOULD be used to
> identify both the version control system and the secure transport, and a
> version control system with hash based commit identifiers SHOULD be used.
> Automated tools MAY omit warnings about missing hashes for version control
> systems that do not provide hash based commit identifiers.
> To handle version control systems that do not support including commit or
> tag references directly in the URL, that information may be appended to the
> end of the URL using the @<commit-hash> or the @<tag>#<commit-hash>
> notation.



>> Putting those two together means that anyone that chooses to do so is
>> already free to write a tool that:
>>
>> - downloads a PyPI package
>> - looks for a "Editable Install" Project-URL, and uses that if defined
>> - otherwise looks for a promising VCS reference in Download-URL, the
>> Project-URL definitions, and the long description
>>
>
> Explicit is better than implicit.
> Simple is better than complex.


> - runs `pip -e` based on whatever it finds
>>
>

>
>> And as long as that tool is itself pip installable, there's no particular
>> reason the feature needs to be built into pip itself.
>>
>
> STORY: Users can pull the source code for each installed package (git,
> [{RPM,} (archive-within-RPM.tar.gz)])
>
> ...
>
> the npm package.json docs are a pretty good read here:
>
> - (with {name, description, url} things are already schema.org/Thing s)
> - https://docs.npmjs.com/files/package.json#bugs
> - https://docs.npmjs.com/files/package.json#repository
> - https://docs.npmjs.com/files/package.json#man
>
> ```json
>
> "bugs":
>  { "url" : "https://github.com/owner/project/issues",
>   "email" : "project at hostname.com"
>  }
>
> "repository" :
>   { "type" : "git"
>   , "url" : "https://github.com/npm/npm.git"
>   }
>
> "repository" :
>   { "type" : "svn"
>   , "url" : "https://v8.googlecode.com/svn/trunk/"
>   }
> ```
>
>
>
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170324/eac9b68f/attachment.html>