namespace_packages: include itself ot not include
Hi, Let suppose that we have package aaa and subpackage bbb and we are going to write setup.py for bbb using setuptools: extra = dict( namespace_packages=["aaa", "aaa.bbb"], zip_safe = False, install_requires = ['setuptools'], ) if __name__=="__main__": setup(name="aaa.bbb", version="0.0.1", description="Dummy example subpackage", author="Mr. Nemo", url="http://www.the.way.org/to/hell", author_email="MrNemo@the.way.org", packages=['aaa.bbb'], license='BSD', **extra) This is the way of namespace_packages usage I against of. So, the questions is am I right standing that package bbb, subpackage of aaa must list in namespace_packages only namespaces it participate in eq 'aaa', and must not list any namespaces it is provides, in this case 'aaa.bbb'? So correct value namespace_packages=['aaa'] in the case above. Best regards, Roman Kurakin
At 02:12 PM 2/4/2011 +0300, Roman Kurakin wrote:
Hi,
Let suppose that we have package aaa and subpackage bbb and we are going to write setup.py for bbb using setuptools:
extra = dict( namespace_packages=["aaa", "aaa.bbb"], zip_safe = False, install_requires = ['setuptools'], )
if __name__=="__main__": setup(name="aaa.bbb", version="0.0.1", description="Dummy example subpackage", author="Mr. Nemo", url="http://www.the.way.org/to/hell", author_email="MrNemo@the.way.org", packages=['aaa.bbb'], license='BSD', **extra)
This is the way of namespace_packages usage I against of. So, the questions is am I right standing that package bbb, subpackage of aaa must list in namespace_packages only namespaces it participate in eq 'aaa', and must not list any namespaces it is provides, in this case 'aaa.bbb'? So correct value namespace_packages=['aaa'] in the case above.
If 'aaa.bbb' is not itself a namespace package, then namespace_packages should only contain 'aaa'. And unless you will have yet another project which is installing something under aaa.bbb, then aaa.bbb does not need to be a namespace package. namespace_packages should only list namespaces in which the current project *participates*. That is, it should list packages that are *containers* for the thing(s) this project is providing, that are *shared* by other projects.
P.J. Eby wrote:
Hi,
Let suppose that we have package aaa and subpackage bbb and we are going to write setup.py for bbb using setuptools:
extra = dict( namespace_packages=["aaa", "aaa.bbb"], zip_safe = False, install_requires = ['setuptools'], )
if __name__=="__main__": setup(name="aaa.bbb", version="0.0.1", description="Dummy example subpackage", author="Mr. Nemo", url="http://www.the.way.org/to/hell", author_email="MrNemo@the.way.org", packages=['aaa.bbb'], license='BSD', **extra)
This is the way of namespace_packages usage I against of. So, the questions is am I right standing that package bbb, subpackage of aaa must list in namespace_packages only namespaces it participate in eq 'aaa', and must not list any namespaces it is provides, in this case 'aaa.bbb'? So correct value namespace_packages=['aaa'] in the case above. If 'aaa.bbb' is not itself a namespace package, then namespace_packages should only contain 'aaa'. And unless you will have yet another project which is installing something under aaa.bbb,
At 02:12 PM 2/4/2011 +0300, Roman Kurakin wrote: then aaa.bbb does not need to be a namespace package. Ok. If it is not namespace package by itself all is clear. But in case it is? What is the difference? Why 'aaa.bbb' should even know that there is, or will be any other, for example 'aaa.bbb.ccc'? For example, in case 'aaa.bbb.ccc' is developed by some one else. namespace_packages should only list namespaces in which the current project *participates*. 'Participates' or also 'provides'? In simple case the package 'aaa.bbb' only participates in 'aaa' and provides only 'aaa.bbb'. That is, it should list packages that are *containers* for the thing(s) this project is providing, that are *shared* by other projects. In a simple case 'shared' is a little bit confusing in case we are 'extending' namespace via strict hierarchy aaa.bbb, aaa.ccc, aaa.bbb.ddd etc. Where any of packages are initial providers of their own namespaces. 'aaa' and 'aaa.bbb' share 'aaa' namespace, but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
Best regards, rik
At 12:06 AM 2/8/2011 +0300, Roman Kurakin wrote:
but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
This statement is why you're confused. A namespace is not "provided" by anyone, only participated in. Think of it this way. You and I decide to have lunch together. I am not providing the lunch. You are not providing the lunch. But we are both participating in the lunch. That is how namespace packages work. aaa.bbb participates in the aaa namespace, therefore it lists aaa as a namespace package. Better, a concrete example. zope.interface participates in the zope namespace package, so it lists 'zope' in namespace_packages. There is no "zope" package that stands by itself, there are only zope.* packages that together participate in the zope namespace.
Why 'aaa.bbb' should even know that there is, or will be any other, for example 'aaa.bbb.ccc'?
It would only do this if you indeed planned to have that happen. For example, I have a 'peak.util' namespace package. There are many projects, such as DecoratorTools, Importing, and so forth on PyPI that provide modules in that namespace. DecoratorTools is peak.util.decorators, Importing is peak.util.imports, and so on. There is NO peak.util package, however. You cannot download peak.util someplace, because it is a *namespace* package. The entire point of a namespace package is that it's just a name, and NOT an actual package. You would never import anything directly from peak.util, because it's just a place where other packages live. "peak" is also a namespace package, and there are other projects that contribute modules and subpackages to that namespace.
In a simple case 'shared' is a little bit confusing in case we are 'extending' namespace via strict hierarchy aaa.bbb, aaa.ccc, aaa.bbb.ddd etc. Where any of packages are initial providers of their own namespaces. 'aaa' and 'aaa.bbb' share 'aaa' namespace, but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
Namespace packages don't work like that. All participants in a namespace are *equal peers*, not "base plus extensions". Sure, you can DO it, but the tools don't have any notion that one package is the base and another is the extender. That's why you're getting confused: you are thinking "base and extension", but namespace packages are peer-to-peer. You participate in a namespace by having some code that goes in it, and EVERY project that participates in that namespace must declare it -- no exceptions. (If a project doesn't declare the namespace, and it ends up first on sys.path, it will block access to all the other participants in the same namespace.)
At 12:06 AM 2/8/2011 +0300, Roman Kurakin wrote:
but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
This statement is why you're confused. A namespace is not "provided" by anyone, only participated in. No I am confused by practice :-) . The problem I am thinking about is __init__.py. So __init__.py is my namespace provider I thinking of beside the scene. The
P.J. Eby wrote: problem I hit was with rpm packages I was playing with. The package 'aaa.bbb' list itself in namespace_packages and developer states that this is needed cause there is 'aaa.bbb.ccc' package. In this case, installer drops __init__.py not only for aaa/__init__.py but also for aaa/bbb/__init__.py. For 'aaa' this is correct since some one should provide __init__.py for 'aaa' before us. But who will provide __init__.py for 'aaa.bbb' if we pack it as setuptool provided it for us?
Think of it this way. You and I decide to have lunch together. I am not providing the lunch. You are not providing the lunch. But we are both participating in the lunch.
That is how namespace packages work. aaa.bbb participates in the aaa namespace, therefore it lists aaa as a namespace package. But what about 'aaa.bbb' that lists itself as a namespace package? Should it list itself or not? In case it should, the question is how to deal with __init__.py?
That is why I tried to draw such model with providers and extenders. One should make reservation if lunch is planned in some popular place. rik
Better, a concrete example. zope.interface participates in the zope namespace package, so it lists 'zope' in namespace_packages. There is no "zope" package that stands by itself, there are only zope.* packages that together participate in the zope namespace.
Why 'aaa.bbb' should even know that there is, or will be any other, for example 'aaa.bbb.ccc'?
It would only do this if you indeed planned to have that happen.
For example, I have a 'peak.util' namespace package. There are many projects, such as DecoratorTools, Importing, and so forth on PyPI that provide modules in that namespace. DecoratorTools is peak.util.decorators, Importing is peak.util.imports, and so on.
There is NO peak.util package, however. You cannot download peak.util someplace, because it is a *namespace* package.
The entire point of a namespace package is that it's just a name, and NOT an actual package. You would never import anything directly from peak.util, because it's just a place where other packages live.
"peak" is also a namespace package, and there are other projects that contribute modules and subpackages to that namespace.
In a simple case 'shared' is a little bit confusing in case we are 'extending' namespace via strict hierarchy aaa.bbb, aaa.ccc, aaa.bbb.ddd etc. Where any of packages are initial providers of their own namespaces. 'aaa' and 'aaa.bbb' share 'aaa' namespace, but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
Namespace packages don't work like that. All participants in a namespace are *equal peers*, not "base plus extensions".
Sure, you can DO it, but the tools don't have any notion that one package is the base and another is the extender. That's why you're getting confused: you are thinking "base and extension", but namespace packages are peer-to-peer. You participate in a namespace by having some code that goes in it, and EVERY project that participates in that namespace must declare it -- no exceptions.
(If a project doesn't declare the namespace, and it ends up first on sys.path, it will block access to all the other participants in the same namespace.)
At 01:07 PM 2/8/2011 +0300, Roman Kurakin wrote:
P.J. Eby wrote:
At 12:06 AM 2/8/2011 +0300, Roman Kurakin wrote:
but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
This statement is why you're confused. A namespace is not "provided" by anyone, only participated in. No I am confused by practice :-) . The problem I am thinking about is __init__.py. So __init__.py is my namespace provider I thinking of beside the scene.
There is no such thing as a namespace provider. That's why you're confused. A namespace package is just a namespace for other modules and packages. It does not have any existence of its own.
The problem I hit was with rpm packages I was playing with. The package 'aaa.bbb' list itself in namespace_packages and developer states that this is needed cause there is 'aaa.bbb.ccc' package. In this case, installer drops __init__.py not only for aaa/__init__.py but also for aaa/bbb/__init__.py. For 'aaa' this is correct since some one should provide __init__.py for 'aaa' before us.
You're still confused. In the RPM case, there will be *no* aaa/__init__.py.
But who will provide __init__.py for 'aaa.bbb' if we pack it as setuptool provided it for us?
It is not necessary. Look for a file like "mypackage-version-nspkg.pth" in the directory adjacent to aaa -- you will see that it contains code to make the package exist despite the nonexistence of the __init__.py
Think of it this way. You and I decide to have lunch together. I am not providing the lunch. You are not providing the lunch. But we are both participating in the lunch.
That is how namespace packages work. aaa.bbb participates in the aaa namespace, therefore it lists aaa as a namespace package. But what about 'aaa.bbb' that lists itself as a namespace package?
You are saying, "but what if I provide the lunch?" You can't provide the lunch. You can only *join* the lunch. You are trying to solve a problem that doesn't exist. Namespace packages are the names of a *group* of modules or packages. There is no "main" package for a namespace, and no "extensions". There is no single project that "provides" the namespace, just like there is no one person who "provides" the lunch. If you persist in thinking that there is a provider, then you will continue to be confused, and trying to solve a problem that doesn't actually exist.
Should it list itself or not? In case it should, the question is how to deal with __init__.py?
As it says in the setuptools documentation, you must provide an __init__.py in your project that ONLY contains the namespace registration. In installation environments that do not require an __init__.py (e.g. rpm installs), this file will NOT be included -- an nspkg.pth file will be used instead. For .egg files and directories, the __init__.py of every participating project is included, but only the first one found on sys.path will actually be used.
That is why I tried to draw such model with providers and extenders. One should make reservation if lunch is planned in some popular place.
Setuptools handles the reservations. The project simply says, "I'm with the aaa group", and setuptools says, "ah, then you're sitting over here". ;-) If there is nobody else arrived yet, then setuptools takes care of allocating the table, so to speak, and then seats all the newcomers there as well. But, just because one person arrives first, doesn't make them the head of the group. Members of the party may arrive in any order, and when others show up they will be seated in the same place.
Sorry for top-posting. Since my model was braked, I'll try to explain the problem from very beginning. I didn't expect that model is totally wrong :-) There is a package 'aaa.bbb' that uses setuptool only if it is installed on the system. This is useful for development, but this breaks rpm build process cause it expects __init__.py and doesn't expect egg and pth, more over it ignores them. So in my case there is no egg & pth. I started to dig if there is possibility to coexists development and rpmbuild processes on the same system without very messy spec file. I didn't find the strict requirements for the package to list itself in a namespace_package even if I know that there would be 'aaa.bbb.ccc'. But if I not list it there, I can ignore egg&pth files, and in my cases setuptool will behave compatible. So probably question does it break something if package 'aaa.bbb' wouldn't list itself in a namespace_package? And does it provide something useful if package 'aaa.bbb' would? Best regards, rik P.J. Eby wrote:
At 12:06 AM 2/8/2011 +0300, Roman Kurakin wrote:
but one is initial provider of 'aaa' share and other consumer of 'aaa' share.
This statement is why you're confused. A namespace is not "provided" by anyone, only participated in. No I am confused by practice :-) . The problem I am thinking about is __init__.py. So __init__.py is my namespace provider I thinking of beside the scene. There is no such thing as a namespace provider. That's why you're confused. A namespace package is just a namespace for other modules and packages. It does not have any existence of its own. The
P.J. Eby wrote: problem I hit was with rpm packages I was playing with. The package 'aaa.bbb' list itself in namespace_packages and developer states that this is needed cause there is 'aaa.bbb.ccc' package. In this case, installer drops __init__.py not only for aaa/__init__.py but also for aaa/bbb/__init__.py. For 'aaa' this is correct since some one should provide __init__.py for 'aaa' before us. You're still confused. In the RPM case, there will be *no* aaa/__init__.py. But who will provide __init__.py for 'aaa.bbb' if we pack it as setuptool provided it for us? It is not necessary. Look for a file like "mypackage-version-nspkg.pth" in the directory adjacent to aaa -- you will see that it contains code to make the package exist despite the nonexistence of the __init__.py
Think of it this way. You and I decide to have lunch together. I am not providing the lunch. You are not providing the lunch. But we are both participating in the lunch.
That is how namespace packages work. aaa.bbb participates in the aaa namespace, therefore it lists aaa as a namespace package. But what about 'aaa.bbb' that lists itself as a namespace package? You are saying, "but what if I provide the lunch?" You can't provide
At 01:07 PM 2/8/2011 +0300, Roman Kurakin wrote: the lunch. You can only *join* the lunch.
You are trying to solve a problem that doesn't exist. Namespace packages are the names of a *group* of modules or packages. There is no "main" package for a namespace, and no "extensions". There is no single project that "provides" the namespace, just like there is no one person who "provides" the lunch.
Should it list itself or not? In case it should, the question is how to deal with __init__.py? As it says in the setuptools documentation, you must provide an __init__.py in your project that ONLY contains the namespace registration. In installation environments that do not require an __init__.py (e.g. rpm installs), this file will NOT be included -- an nspkg.pth file will be used instead. For .egg files and directories,
That is why I tried to draw such model with providers and extenders. One should make reservation if lunch is planned in some popular place. Setuptools handles the reservations. The project simply says, "I'm with the aaa group", and setuptools says, "ah, then you're sitting over here". ;-) If there is nobody else arrived yet, then setuptools takes care of allocating the table, so to speak, and then seats all
If you persist in thinking that there is a provider, then you will continue to be confused, and trying to solve a problem that doesn't actually exist. the __init__.py of every participating project is included, but only the first one found on sys.path will actually be used. the newcomers there as well.
But, just because one person arrives first, doesn't make them the head of the group. Members of the party may arrive in any order, and when others show up they will be seated in the same place.
At 10:29 PM 2/8/2011 +0300, Roman Kurakin wrote:
There is a package 'aaa.bbb' that uses setuptool only if it is installed on the system.
In that case, it can't sensibly use setuptools namespace packages. At all. You need to either: 1. always use setuptools 2. use an alternate namespace package mechanism (such as pkgutil.extend_path()), and not declare the namespace 3. Not use namespace packages at all
This is useful for development, but this breaks rpm build process cause it expects __init__.py and doesn't expect egg and pth, more over it ignores them. So in my case there is no egg & pth.
The nspkg.pth file is for non-egg installs. If you have an egg, there is no nspkg.pth file for that package.
I started to dig if there is possibility to coexists development and rpmbuild processes on the same system without very messy spec file. I didn't find the strict requirements for the package to list itself in a namespace_package even if I know that there would be 'aaa.bbb.ccc'.
It is a requirement for consistent and correct runtime behavior across all installation targets. (The installation targets are .egg file, .egg-directory, and non-egg (e.g. rpm))
So probably question does it break something if package 'aaa.bbb' wouldn't list itself in a namespace_package? And does it provide something useful if package 'aaa.bbb' would?
To make sure we're clear: if aaa.bbb is actually a namespace -- that is, if it has separately-installable contents -- then you MUST declare it if you want consistently correct runtime behavior. If you do not declare it, then there are some environments where it will appear to work, and there will be other (superficially identical) environments where some part of the namespace will mysteriously fail to be importable. In other words, the rules are there for a reason. These limitations are the nature of Python 2.x's import machinery, which was not originally designed with namespace packages in mind. There is a namespace package PEP that (in principle) provides a better mechanism than what setuptools does, but implementation progress is currently stalled, and the PEP has not been accepted. So, your options at the present time for separately installed parts of a namespace package are as I have described. There is a pure distutils approach you can use that allows partially installed packages to exist, but it is somewhat fragile and is not setuptools compatible. It requires that there be a "provider" of a package, in the sense that exactly one project contains the __init__.py for the package. But it is fragile because it may not be supported on all platforms and build targets, and it requires that aaa.bbb be always installed before aaa.bbb.ccc.... but of course the distutils by themselves have no dependency mechanism to enforce this.
participants (2)
-
P.J. Eby -
Roman Kurakin