On Jun 1, 2013, at 11:57 AM, Jim Fulton wrote:

In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more.

We should not have to come up with a process for recognizing squatters on simple package names. We should have something more systematic, IMO.

Unfortunately, I think the sanest way of avoiding most package name issues is to base them on domains, as is done in the Java world. This goes against the Python philosophy of preferring flat to nested, but I still think it's better than trying to police squatters, or to encouraging races to claim top-level names.

For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves.

Jim

-- Jim Fulton http://www.linkedin.com/in/jimfulton _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

I am opposed to this. Requiring someone to have purchased a domain adds a significant to publishing a project. If there are no requirements that they have purchased the domain then it's nothing more than a convention and something that anyone who wants to do this can do. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sat, Jun 1, 2013 at 2:02 PM, Donald Stufft wrote:

On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote:

I am opposed to this. Requiring someone to have purchased a domain adds a significant to publishing a project. If there are no requirements that they have purchased the domain then it's nothing more than a convention and something that anyone who wants to do this can do.

Fair enough. A common variation on this scenario, which avoids purchasing a domain, is to use a code hosting domain and project name, so, for example: org.bitbucket.j1m.foo. Of course, using a domain name without owning it is a form of squatting. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote:

In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more.

I've heart this sentiment before, but would like to read more clearly stated problems.

We should not have to come up with a process for recognizing squatters on simple package names. We should have something more systematic, IMO.

Unfortunately, I think the sanest way of avoiding most package name issues is to base them on domains, as is done in the Java world. This goes against the Python philosophy of preferring flat to nested, but I still think it's better than trying to police squatters, or to encouraging races to claim top-level names.

I am not sure that tying to DNS namespacing is the only solution here (whatever the problem is exactly :).

For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves.

I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one. Everyone could continue to push non-namespaced (flat) packages to pypi like now but the names couldn't take the form of namespaced ones. So for example if the django community wants to introduce the concept of "vetted" plugins/addons, they could move to manage "dj.*" or so. I don't think we would suddenly drown in namespace regs if we make it a pre-condition that there need to be a couple of existing real packages that would go into it. cheers, holger

Jim

-- Jim Fulton http://www.linkedin.com/in/jimfulton _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Donald Stufft Sent: Saturday, 01 June, 2013 15:30 To: holger krekel Cc: distutils sig Subject: Re: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages. On Jun 1, 2013, at 3:21 PM, holger krekel mailto:holger@merlinux.eu > wrote: On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote: For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves. I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one. So for example if the django community wants to introduce the concept of "vetted" plugins/addons, they could move to manage "dj.*" or so. I think this example highlights some of the challenges with registering/controlling namespaces - who owns what and what is the meaning of a (distribution) package name? For example, what is the namespace used for an endorsed django plugin written by zope corporation? This problem is not present now, as the author can choose the domain which is most relevant to that plugin and its users. If there's some expectation that it should appear in a namespace managed by another organization, that necessitates a coordination between the namespace owner/manager and the project author. I think more people would claim namespaces when namespaces are better supported in Python. My expectation is Python 3.3 namespace package support will ease that challenge (when it becomes a dominant version). My inclination is to say it's not a huge problem, and later is preferable than sooner.

On Sat, Jun 1, 2013 at 6:00 PM, Paul Moore wrote:

On 1 June 2013 16:57, Jim Fulton wrote:

...

In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more.

I'd like to see some evidence that this is the case. It doesn't seem so to me - most package names are relatively discoverable and/or intuitive, and we currently have basically no namespacing. There's a long way to go before something like your suggestion is needed, in my view.

...

Unfortunately, I think the sanest way of avoiding most package name issues is to base them on domains, as is done in the Java world. This goes against the Python philosophy of preferring flat to nested, but I still think it's better than trying to police squatters, or to encouraging races to claim top-level names.

No, no, no...

There's the point Donald made that you require people to own a domain (or you create some sort of hack like org.bitbucket.username/com.github.username/...) but it also makes package names unreasonably deep, and requires an explosion of namespace packages at the top level. And it's ugly :-)

Perl manages with a relatively flat namespace and relatively informal rules for managing package names (AIUI). I'm sure Python can, too.

Paul

There's also the fact that our module namespace is separate from our distribution names namespace...

On Sun, Jun 02, 2013 at 09:32 +1000, Nick Coghlan wrote:

I'm with Jason in the "maybe eventually, but not right now" camp.

Namespace collisions are indeed a possibility and a potential concern, both in the distribution namespace and the top level import namespace.

Indeed, in this thread, i assumed we were only talking about distribution/pypi namespacing. Somewhat proper import namespacing is only available with very recent python versions which still have a long way to become mainstream.

The fact there is no 1to1 mapping between distribution names and the import namespace means that informal conflict avoidance is already possible - prepending "<qualifier>-" to the desired package name makes it possible to publish it alongside another distribution using the same name without having to change the top level import location. If the distributed packages use explicit relative imports appropriately, an integrator may even be able to use them side by side by dropping them into higher level namespace packages.

Java's "use the domain name" approach simply outsources the conflict resolution to a third party, by *requiring* that publishers acquire a domain name prior to publication. I prefer our model of initially *assuming* a lack of conflict to lower barriers to publication.

I do think we need to better handle cases where the assumption breaks down, but we shouldn't forget namespacing is already possible.

If pypi has no idea about namespaces (like i considered them in my other post) then using namespaces do not really provide much. Someone can still come along and publish within that pseudo-namespace. I would think the goal of pypi-namespaces would be to give a group control over anything that's released using it, allowing to communicate install-users certain guarantees. However, before further discussion i think there first needs to be more reasoning and stating of practical problems with the current anyone-can-register-anything-that's-not-taken model. best, holger

Cheers, Nick.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Sun, Jun 02, 2013 at 17:26 +1000, Nick Coghlan wrote:

On Sun, Jun 2, 2013 at 5:10 PM, holger krekel wrote:

...

If pypi has no idea about namespaces (like i considered them in my other post) then using namespaces do not really provide much. Someone can still come along and publish within that pseudo-namespace. I would think the goal of pypi-namespaces would be to give a group control over anything that's released using it, allowing to communicate install-users certain guarantees.

However, before further discussion i think there first needs to be more reasoning and stating of practical problems with the current anyone-can-register-anything-that's-not-taken model.

TUF actually has native support for prefix delegation, but actually *using* that is a long way down the todo list at the moment. Static dependency metadata publication and end-to-end signature support are well ahead of it and will likely keep us collectively busy for a while yet.

No worries, I understood already that it's not high on your list. I'd appreciate, however, if Jim or someone else could state the problems with missing namespacing so we can start a discussion from there later. Speaking of TUF: is there some kind of PEP like doc floating already? cheers, holger

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jun 2, 2013, at 4:21 AM, Nick Coghlan wrote:

On Sun, Jun 2, 2013 at 5:37 PM, holger krekel wrote:

...

Speaking of TUF: is there some kind of PEP like doc floating already?

Just the proof-of-concept the TUF folks created about using it to secure /simple. I'm personally sold on the technology itself as something we should deploy in the long run, but I think it makes sense to wait until we have the static dependency metadata publication and various other PyPI related infrastructure issues sorted out before we try to offer additional protection above and beyond trusting the SSL CA system and PyPI itself.

That said, one of the reasons PEP 426 calls out the "essential dependency resolution" fields is that those are the ones I think it may make sense to embed in the TUF custom metadata fields.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

If we deploy some sort of end to end signing I think TUF is a good implementation of it. I'm not sold on the possibility of reasonably doing end to end signing here though. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 6/2/13 4:21 AM, Nick Coghlan wrote:

On Sun, Jun 2, 2013 at 5:37 PM, holger krekel wrote:

...

Speaking of TUF: is there some kind of PEP like doc floating already?

Just the proof-of-concept the TUF folks created about using it to secure /simple. I'm personally sold on the technology itself as something we should deploy in the long run, but I think it makes sense to wait until we have the static dependency metadata publication and various other PyPI related infrastructure issues sorted out before we try to offer additional protection above and beyond trusting the SSL CA system and PyPI itself.

That said, one of the reasons PEP 426 calls out the "essential dependency resolution" fields is that those are the ones I think it may make sense to embed in the TUF custom metadata fields.

Nick got our proof-of-concept pretty much right, and I just want to make this correction: we offered security for both /simple and /packages, but only for a subset of packages. We were working on securing all the packages under PyPI, but were derailed by some projects with immediate deadlines. The good news is that we will be continuing our work full-time this summer, and expect to make much progress. We don't have a PEP for it, besides our design proposal[1]. I think a PEP is a good idea, and we should draft one along the way. [1] https://docs.google.com/document/d/1sHMhgrGXNCvBZdmjVJzuoN5uMaUAUDWBmn3jo7vx...

On Sat, Jun 1, 2013 at 2:54 PM, Noah Kantrowitz wrote:

On Jun 1, 2013, at 11:09 AM, Jim Fulton wrote:

...

On Sat, Jun 1, 2013 at 2:02 PM, Donald Stufft wrote:

...

On Jun 1, 2013, at 2:01 PM, Donald Stufft wrote:

I am opposed to this. Requiring someone to have purchased a domain adds a significant to publishing a project. If there are no requirements that they have purchased the domain then it's nothing more than a convention and something that anyone who wants to do this can do.

Fair enough. A common variation on this scenario, which avoids purchasing a domain, is to use a code hosting domain and project name, so, for example: org.bitbucket.j1m.foo.

Of course, using a domain name without owning it is a form of squatting.

All that means is either we move the problem (instead of one shared namespace we two or three common ones)

I don't understand why you say two or three. There would be as many namespaces as there are domains or VCS accounts. There would be many distinct namespaces, each controlled controlled by a single user or organization.

or we do it github-style and just prepend usernames at which point you can skip the whole URI thing because usernames must be unique for reasons of general sanity and I don't think it is a huge deal that a single person can't have two packages of the same name.

That's an option. (I assume you mean PyPI user names.) It would be more attractive if PyPI supported organizational accounts. (I sure wish it did.) I can't say I find the idea of tying a package name to an account name attractive, but it's a good alternative for projects without a domain.

Github-style namespacing just means that either names all suck (django/django, kennethreitz/requests) or you need to come up with some way to map un-namespaced names to their canonical form and we are more or less back at square one. If people don't mind the sucky names, they can already put that in their package name if the bare version is taken, so QED this is already doable in the current system, it just looks so ugly that no one wants to do it and enforcing the ugly seems like a poor option.

My observation of the java world is that most packages that get published to central repositories end up having domain based names. Even that sucks to some degree because flat is better than nested. I just don't think the current ad-hoc mechanism we're using now is scalable. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On Sat, Jun 1, 2013 at 4:13 PM, Jason R. Coombs wrote:

From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com@python.org] On Behalf Of Donald Stufft Sent: Saturday, 01 June, 2013 15:30 To: holger krekel Cc: distutils sig Subject: Re: [Distutils] Sooner or later, we're going to have to be more formal about how we name packages.

On Jun 1, 2013, at 3:21 PM, holger krekel wrote:

On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote:

For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves.

I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one.

So for example if the django community wants to introduce the concept

of "vetted" plugins/addons, they could move to manage "dj.*" or so.

I think this example highlights some of the challenges with registering/controlling namespaces – who owns what and what is the meaning of a (distribution) package name? For example, what is the namespace used for an endorsed django plugin written by zope corporation?

IMO the purpose of the namespace is to organize names. Whoever owns the namespace decides what names can go into it. It's purely a name management issue, not, for example, a intellectual property issue. If Zope Corporation independently creates a Django plugin, I'd expect it to go into the "zc" namespace. OTOH, Zope Corporation was an active member of the Django community, it might publish to the "dj" (or whatever) namespace, or request permission to do so. I started using the zc namespace a few years ago because I didn't want to impose our names on the Zope community.

I think more people would claim namespaces when namespaces are better supported in Python. My expectation is Python 3.3 namespace package support will ease that challenge (when it becomes a dominant version).

This is somewhat baffling to me. We've used namespaces for over a decade virtually without issue. (We;ve used namespaces far longer than that, going all the way back to "ni" with relatively minor issues.) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

On Jun 2, 2013, at 11:25 AM, Jim Fulton wrote:

On Sat, Jun 1, 2013 at 3:21 PM, holger krekel wrote:

...

On Sat, Jun 01, 2013 at 11:57 -0400, Jim Fulton wrote:

...

In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more.

I've heart this sentiment before, but would like to read more clearly stated problems.

I thought the problem was pretty clear: name collisions.

There's a parallel thread on how to detect and reclaim names that are taken and unused. I think if we had a more systematic way of naming packages, this wouldn't be an issue.

I think the parallel thread mostly involves people wanting specific names that are no longer available. I know I've claimed a few good names at the start of an idea then abandoned the idea and forgotten to deregister the name. There are still lots of good names but people get attached to ones they like and it feels kinda shitty when you have to rename your project because of a placeholder that's been there for 3 years that you didn't notice.

...

...

We should not have to come up with a process for recognizing squatters on simple package names. We should have something more systematic, IMO.

Unfortunately, I think the sanest way of avoiding most package name issues is to base them on domains, as is done in the Java world. This goes against the Python philosophy of preferring flat to nested, but I still think it's better than trying to police squatters, or to encouraging races to claim top-level names.

I am not sure that tying to DNS namespacing is the only solution here (whatever the problem is exactly :).

It's not the only solution, but it's a pretty easy one. It avoids (more more precisely reuses) a central naming authority. It's the technique used by the java ecosystem and by XML namespaces, for example.

We already have a central naming authority it's called PyPI ;)

...

...

For a while, many of us have been pretty careful to use namespaces for new packages to mitigate this issue. For example, the zc namespace is a shorter version of com.zope, but at some point, it won't be fair for us to claim zc for ourselves.

I wonder if we could allow people/groups to apply (to humans) for a namespace which they can subsequently control, like the "zc.*" one.

We could. Maybe that's the most palatable alternative. It has the huge benefit of promoting relatively flat, Pythonic, namespaces.

It has disadvantages:

- We have to set up some sort of naming authority.

- It's probably not scalable.

As mentioned above we already have an authority, and it's more or less scaled quite fine so far. I still continue to be able to get very good names, but sometimes it takes a few tries to find an unused one.

...

Everyone could continue to push non-namespaced (flat) packages to pypi like now but the names couldn't take the form of namespaced ones.

I'm not sure what you're suggesting here.

Are you saying someone could publish a package named: "zc", bit not "zc.foo"?

Or are you saying that publication of a package named "bar" would prevent someone from creating a "bar" namespace, and the other way around?

This is likely implementation details, but if I were to do it then folks could apply for a particular namespace, (For example zc.*) and it would give them ownership over all levels of that namespace, (in our example, zc, zc.foo, zc.bar, zc.foo.bar, etc but not zc4 or zc4.cool). However they would only own that namespace if they've applied and been granted it. Otherwise anyone is free to register packages in that namespace. And for what it's worth Organization accounts is something on my roadmap for PyPI.

Jim

-- Jim Fulton http://www.linkedin.com/in/jimfulton _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sat, Jun 1, 2013 at 5:57 PM, Jim Fulton wrote:

In the Python community, we've been pretty laid back about how we name packages. When we were small, this made sense. It doesn't make sense any more.

I don't think this is a problem, and I don't think domains or usernames in the package names is a solution even if it is a problem.

We should not have to come up with a process for recognizing squatters on simple package names. We should have something more systematic, IMO.

I also don't think squatting in itself is that much of a problem. Only once has someone been faster than me in stealing a package name and that was "skynet". :-) Pretty much all other package names I've ever come up with has been free. And when somebody is squatting, I think it can be dealt with manually, for the most time. In fact, I'm trying to contact the skynet author now, to see if I can get my "skynet" in there instead. ;-) [1] Something I don't like though is the plethora of non-packages, most of which are test packages of some sort. Just search for "foo". :-) I'd like that to be cleaned up. //Lennart [1] https://github.com/regebro/skynet

On 06/03/2013 09:35 AM, holger krekel wrote:

On Sun, Jun 02, 2013 at 13:51 -0400, Tres Seaver wrote:

...

On 06/02/2013 03:10 AM, holger krekel wrote:

...

Somewhat proper import namespacing is only available with very recent python versions which still have a long way to become mainstream.

I don't understand this claim at all. W'eve had packages in python for fifteen years, and extensible namespace-package support in one form or another for eight. The fact that the Python 3.3 adds support for a new spelling doesn't mean they are a new feature.

I stand corrected. To be honest, i didn't consider the setuptools extensible namespace support a proper solution but indeed it exists and is used.

FWIW, the small scientific libraries associated with the SciPy ecosystem was gathered under the "scikits" umbrella, so that you had "scikits.learn", "scikits.image", and so on. Eventually the setuptools pain made them decide to drop that and name themselves "sklearn", "skimage". Dag Sverre

On Mon, Jun 3, 2013 at 3:35 AM, holger krekel wrote:

On Sun, Jun 02, 2013 at 13:51 -0400, Tres Seaver wrote:

...

On 06/02/2013 03:10 AM, holger krekel wrote:

...

Somewhat proper import namespacing is only available with very recent python versions which still have a long way to become mainstream.

I don't understand this claim at all. W'eve had packages in python for fifteen years, and extensible namespace-package support in one form or another for eight. The fact that the Python 3.3 adds support for a new spelling doesn't mean they are a new feature.

I stand corrected. To be honest, i didn't consider the setuptools extensible namespace support a proper solution but indeed it exists and is used.

Before the setuptools namespace support, there was the pkgutil module in the standard library. Setuptools' system only added support for zipped packages. Python's current package system allows you to manipulate a package's path, used to find modules and sub-packages in the package. The Zope project used this mechanism for years support the Zope "Products" namespace package. The pkgutils module, and later setuptools' mechanism just simplified this. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton