I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head. The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages. If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path. Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked? I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote:
I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']? Do they all ship an identical foo/__init__.py with import pkg_resources pkg_resources.declare_namespace(__name__) ? AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
yes, both of those statements are true.
However, with the namespace_packages = ['foo'], the
lib\site-packages\foo\__init__.py does not get installed (even though it is
in the source tree). Instead there's just a dir with "foo/bar/__init__.py"
and "foo/blah/__init__.py". I will try to look in the "wheel" side of
things next I guess. Perhaps pip is doing something since it seems to
install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote:
I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Just to recap:
1. if you don't put namespace_packages in the setup.py, then it will
uninstall the shared __init__.py when you uninstall any of the packages
2. If you put namespace_packages, then there is a pth file created for the
shared directory (site-packages/foo) and no foo/__init__.py is created
(even if it is in your package)
#2 - breaks things like : doing a source checkout that participates in this
namespace_package...If you do this then only the
lib/site-packages/foo/<modules> are importable
Solution appears to be:
On Tue, Dec 1, 2015 at 7:17 AM, KP
yes, both of those statements are true.
However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: base
(foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
(sorry for the stupid previous early send)
Just to recap:
1. if you don't put namespace_packages in the setup.py, then it will
uninstall the shared __init__.py when you uninstall any of the packages
2. If you put namespace_packages, then there is a pth file created for the
shared directory (site-packages/foo) and no foo/__init__.py is created
(even if it is in your package)
#2 - breaks things like : doing a source checkout that participates in this
namespace_package...If you do this then only the
lib/site-packages/foo/<modules> are importable
Solution appears to be:
create a standalone "foo" package that has ONLY the shared __init__.py, and
do NOT set "namespace_packages" . This seems to associate the __init__.py
with the "foo" tool, so that only when you uninstall the "foo" tool does
the __init__.py get uninstalled. It also has the shared __init__.py in
lib/site-packages, so it seems that enables the other namespace packages
that are source checkouts or added to path via other methods.
Hopefully this works long term, and maybe is useful to someone else out
there....
Thanks,
On Tue, Dec 1, 2015 at 4:29 PM, KP
Just to recap:
1. if you don't put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages 2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package) #2 - breaks things like : doing a source checkout that participates in this namespace_package...If you do this then only the lib/site-packages/foo/<modules> are importable
Solution appears to be:
On Tue, Dec 1, 2015 at 7:17 AM, KP
wrote: yes, both of those statements are true.
However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: base
(foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi KP,
On Dec 1, 2015, at 4:17 AM, KP
wrote: yes, both of those statements are true.
However, with the namespace_packages = [‘foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree).
This is exactly how it’s supposed to behave for namespace packages. The *.pth file will take care of providing info about your namespace to the python importer.
Instead there’s just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py".
These are regular packages. Hence they preserve their __init__.py. Hope this helps. Regards, Zvezdan
I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
mailto:marius@gedmin.as> wrote: On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Tue, Dec 01, 2015 at 07:17:58AM -0500, KP wrote:
However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
Can you show us your setup.py? Marius Gedminas -- I’m not big on predictions, but I do have one for 2011: HTML5 will continue to be popular, because anything popular will get labeled “HTML5.” -- Mark Pilgrim
It appears that the .pth file only adds the lib/site-packages/foo/ module
to the path, so that part _works_ but that seems to only work for modules
installed in to lib/site-packages. So if I do something like:
foo.bar having this setup.py
setup(name="foo.bar",
version=str(__version__),
packages=["foo","foo.bar"],
namespace_packages=['foo'],
)
To go with this package there's a couple of possible use cases:
1. --- simple pth file point to another participant in this name space
package --
But I also have a pth file with:
c:\test
And the directory contents of C:\Test are:
c:\test\foo\__init__.py # and this as the typical pkgutil.extend_path
c:\test\foo\more\__init__.py
The import mechanisms only find the "foo.bar" I installed and not the
foo.more
2. -- is similar to #1, but maybe slightly more accepted normal...assume
the same "foo.bar", and the setup.py provide. If I'm working on another
"foo.more" package that participates in this namespace package, then an pip
install from the source will _not_ work.
The workaround I posted earlier (an additional package that is NOT a
namespace package, and is needed to force an __init__.py to exist in
lib/site-packages/foo). This works, but it feels like a bit of a hack, and
that the the "-pkg.pth" created for foo.bar should have worked. (work =
allow namespace packages in multiple directories that are in sys.path)
On Wed, Dec 2, 2015 at 12:17 AM, Zvezdan Petkovic wrote: Hi KP, On Dec 1, 2015, at 4:17 AM, KP yes, both of those statements are true. However, with the namespace_packages = [‘foo'], the
lib\site-packages\foo\__init__.py does not get installed (even though it is
in the source tree). This is exactly how it’s supposed to behave for namespace packages.
The *.pth file will take care of providing info about your namespace to
the python importer. Instead there’s just a dir with "foo/bar/__init__.py" and
"foo/blah/__init__.py". These are regular packages. Hence they preserve their __init__.py.
Hope this helps. Regards, Zvezdan I will try to look in the "wheel" side of things next I guess. Perhaps pip
is doing something since it seems to install even source distributables by
first converting to a wheel. On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas I'm not sure where the issue is, but when I specify a namespace_package
in
the setup.py file, I can indeed have multiple packages with the same On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote:
base (foo.bar, foo.blah, etc...). The files all install in to the same
directory. It drops the foo/__init__.py that would be doing the
extend_path, and instead adds a ".pth" file that is a bit over my head. The problem is that it does not seem to traverse the entire sys.path to
find multiple foo packages. Does every foo.x package specify namespace_packages=['foo']? Do they all ship an identical foo/__init__.py with import pkg_resources
pkg_resources.declare_namespace(__name__) ? AFAIU you need both things in every package, if you want to use
namespace packages. If I do not specify namespace_packages and instead just use the
pkgutil.extend_path, then this seems to allow the packages to be in
multiple places in the sys.path. Is there something additional for the namespace_package that i need to
specify in order for all of the sys.path to be checked? I'm using 18.5 setuptools....but I am not sure if this somehow ties in
to
wheel/pip, since I'm using that for the actual install. Marius Gedminas
--
Give a man a computer program and you give him a headache, but teach him
to
program computers and you give him the power to create headaches for
others for
the rest of his life...
-- R. B. Forest _______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Hi KP, Maybe I didn’t follow the thread from the beginning, but your use case is not clear to me.
On Dec 1, 2015, at 1:31 PM, KP
wrote: (sorry for the stupid previous early send)
Just to recap:
1. if you don’t put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages
Right. You need to declare the namespace in setup.py.
2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package)
Correct. That is exactly the behavior that’s expected. There can be many packages in the namespace (e.g., zope.* or zope.app.*) and they all share the namespace, but none of them owns that __init__.py. That’s why it’s not installed there and is replaced with a *.pth file.
#2 - breaks things like : doing a source checkout that participates in this namespace_package…If you do this then only the lib/site-packages/foo/<modules> are importable
Again, it’s not clear what are you referring to? Are you doing a source checkout over your installed packages / virtual environment? I sure hope not. You can use pip —editable for adding your source checkouts to virtual environment or installed packages. If you use build tools (such as zc.buildout) and install your packages, they will end up over several egg / wheel directories “parallel” to each other, but the proper namespace declaration will ensure that they are all importable. You cannot even do the source checkout over such an installation. It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
Solution appears to be:
create a standalone “foo" package that has ONLY the shared __init__.py, and do NOT set "namespace_packages" .
No, that is not the solution. That’s a sure way to break the namespace in the egg/wheel based installations.
This seems to associate the __init__.py with the “foo” tool, so that only when you uninstall the "foo" tool does the __init__.py get uninstalled.
Why do you worry so much about uninstalling __init__.py. If it’s not there (and it isn’t with properly declared namespaces) why does it matter?
It also has the shared __init__.py in lib/site-packages, so it seems that enables the other namespace packages that are source checkouts or added to path via other methods.
What other methods? The key here is to make decision how do you want to install your packages? Pick one and use it instead fighting the tooling.
Hopefully this works long term, and maybe is useful to someone else out there….
I doubt it, It’s an incorrect advice. Zvezdan
Thanks,
On Tue, Dec 1, 2015 at 4:29 PM, KP
mailto:patter001@gmail.com> wrote: Just to recap: 1. if you don't put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages 2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package) #2 - breaks things like : doing a source checkout that participates in this namespace_package...If you do this then only the lib/site-packages/foo/<modules> are importable
Solution appears to be:
On Tue, Dec 1, 2015 at 7:17 AM, KP
mailto:patter001@gmail.com> wrote: yes, both of those statements are true. However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
mailto:marius@gedmin.as> wrote: On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
pip install foo.bar pip install -e svn+http://
Even if both tools have the namespace_package foo, the "foo.more" will not
properly import.
How is this going against standard approaches?
On Wed, Dec 2, 2015 at 9:54 PM, Zvezdan Petkovic
Hi KP,
Maybe I didn’t follow the thread from the beginning, but your use case is not clear to me.
On Dec 1, 2015, at 1:31 PM, KP
wrote: (sorry for the stupid previous early send)
Just to recap:
1. if you don’t put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages
Right. You need to declare the namespace in setup.py.
2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package)
Correct. That is exactly the behavior that’s expected. There can be many packages in the namespace (e.g., zope.* or zope.app.*) and they all share the namespace, but none of them owns that __init__.py. That’s why it’s not installed there and is replaced with a *.pth file.
#2 - breaks things like : doing a source checkout that participates in this namespace_package…If you do this then only the lib/site-packages/foo/<modules> are importable
Again, it’s not clear what are you referring to? Are you doing a source checkout over your installed packages / virtual environment? I sure hope not. You can use pip —editable for adding your source checkouts to virtual environment or installed packages.
If you use build tools (such as zc.buildout) and install your packages, they will end up over several egg / wheel directories “parallel” to each other, but the proper namespace declaration will ensure that they are all importable. You cannot even do the source checkout over such an installation.
It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
Solution appears to be:
create a standalone “foo" package that has ONLY the shared __init__.py, and do NOT set "namespace_packages" .
No, that is not the solution. That’s a sure way to break the namespace in the egg/wheel based installations.
This seems to associate the __init__.py with the “foo” tool, so that only when you uninstall the "foo" tool does the __init__.py get uninstalled.
Why do you worry so much about uninstalling __init__.py. If it’s not there (and it isn’t with properly declared namespaces) why does it matter?
It also has the shared __init__.py in lib/site-packages, so it seems that enables the other namespace packages that are source checkouts or added to path via other methods.
What other methods? The key here is to make decision how do you want to install your packages? Pick one and use it instead fighting the tooling.
Hopefully this works long term, and maybe is useful to someone else out there….
I doubt it, It’s an incorrect advice.
Zvezdan
Thanks,
On Tue, Dec 1, 2015 at 4:29 PM, KP
wrote: Just to recap:
1. if you don't put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages 2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package) #2 - breaks things like : doing a source checkout that participates in this namespace_package...If you do this then only the lib/site-packages/foo/<modules> are importable
Solution appears to be:
On Tue, Dec 1, 2015 at 7:17 AM, KP
wrote: yes, both of those statements are true.
However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same
(foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my
On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: base head.
The problem is that it does not seem to traverse the entire sys.path
to
find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi KP,
On Dec 2, 2015, at 7:00 PM, KP
wrote: It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
pip install foo.bar pip install -e svn+http://
Even if both tools have the namespace_package foo, the "foo.more" will not properly import.
How is this going against standard approaches?
I don’t know. Without seeing the source code for these packages (which Marius already asked for) everything is hypothetical. All I know is that I’m using namespaces successfully for over a decade now and never had the need to work around them. Show us the code. We may be able to help better if we can try and reproduce. Otherwise, it’s hard. Thanks! Zvezdan
On Wed, Dec 2, 2015 at 9:54 PM, Zvezdan Petkovic
mailto:zvezdanpetkovic@gmail.com> wrote: Hi KP, Maybe I didn’t follow the thread from the beginning, but your use case is not clear to me.
On Dec 1, 2015, at 1:31 PM, KP
mailto:patter001@gmail.com> wrote: (sorry for the stupid previous early send)
Just to recap:
1. if you don’t put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages
Right. You need to declare the namespace in setup.py.
2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package)
Correct. That is exactly the behavior that’s expected. There can be many packages in the namespace (e.g., zope.* or zope.app.*) and they all share the namespace, but none of them owns that __init__.py. That’s why it’s not installed there and is replaced with a *.pth file.
#2 - breaks things like : doing a source checkout that participates in this namespace_package…If you do this then only the lib/site-packages/foo/<modules> are importable
Again, it’s not clear what are you referring to? Are you doing a source checkout over your installed packages / virtual environment? I sure hope not. You can use pip —editable for adding your source checkouts to virtual environment or installed packages.
If you use build tools (such as zc.buildout) and install your packages, they will end up over several egg / wheel directories “parallel” to each other, but the proper namespace declaration will ensure that they are all importable. You cannot even do the source checkout over such an installation.
It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
Solution appears to be:
create a standalone “foo" package that has ONLY the shared __init__.py, and do NOT set "namespace_packages" .
No, that is not the solution. That’s a sure way to break the namespace in the egg/wheel based installations.
This seems to associate the __init__.py with the “foo” tool, so that only when you uninstall the "foo" tool does the __init__.py get uninstalled.
Why do you worry so much about uninstalling __init__.py. If it’s not there (and it isn’t with properly declared namespaces) why does it matter?
It also has the shared __init__.py in lib/site-packages, so it seems that enables the other namespace packages that are source checkouts or added to path via other methods.
What other methods? The key here is to make decision how do you want to install your packages? Pick one and use it instead fighting the tooling.
Hopefully this works long term, and maybe is useful to someone else out there….
I doubt it, It’s an incorrect advice.
Zvezdan
Thanks,
On Tue, Dec 1, 2015 at 4:29 PM, KP
mailto:patter001@gmail.com> wrote: Just to recap: 1. if you don't put namespace_packages in the setup.py, then it will uninstall the shared __init__.py when you uninstall any of the packages 2. If you put namespace_packages, then there is a pth file created for the shared directory (site-packages/foo) and no foo/__init__.py is created (even if it is in your package) #2 - breaks things like : doing a source checkout that participates in this namespace_package...If you do this then only the lib/site-packages/foo/<modules> are importable
Solution appears to be:
On Tue, Dec 1, 2015 at 7:17 AM, KP
mailto:patter001@gmail.com> wrote: yes, both of those statements are true. However, with the namespace_packages = ['foo'], the lib\site-packages\foo\__init__.py does not get installed (even though it is in the source tree). Instead there's just a dir with "foo/bar/__init__.py" and "foo/blah/__init__.py". I will try to look in the "wheel" side of things next I guess. Perhaps pip is doing something since it seems to install even source distributables by first converting to a wheel.
On Tue, Dec 1, 2015 at 1:38 AM, Marius Gedminas
mailto:marius@gedmin.as> wrote: On Mon, Nov 30, 2015 at 06:59:31PM -0500, KP wrote: I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head.
The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages.
Does every foo.x package specify namespace_packages=['foo']?
Do they all ship an identical foo/__init__.py with
import pkg_resources pkg_resources.declare_namespace(__name__)
?
AFAIU you need both things in every package, if you want to use namespace packages.
If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path.
Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked?
I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.
Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig https://mail.python.org/mailman/listinfo/distutils-sig
Zvezdan Petkovic
Hi KP,
On Dec 2, 2015, at 7:00 PM, KP
wrote: It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
pip install foo.bar pip install -e svn+http://
Even if both tools have the namespace_package foo, the "foo.more" will not properly import.
How is this going against standard approaches?
I don’t know. Without seeing the source code for these packages (which Marius already asked for) everything is hypothetical. All I know is that I’m using namespaces successfully for over a decade now and never had the need to work around them.
Show us the code. We may be able to help better if we can try and reproduce. Otherwise, it’s hard.
It seems that KP case is either https://github.com/pypa/pip/issues/3160 or https://github.com/pypa/pip/issues/3, isn't it? Both come with sample code that demonstrate the problem. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Yes, the https://github.com/pypa/pip/issues/3 definitely sounds like my
issue. Seems there is some concern over the nspkg.pth there as well. It
seems taht the nspkg.pth is a fine idea to replace the install of the
__init__.py, but that it just doesn't work to fully extend the locations in
which a namespace can reside.
Either way, there a few other workarounds posted there as well, I will
check them out to see if any of them are more palatable than the one I
posted here.
Thanks Lele for sending that link!
-Kevin
On Thu, Dec 3, 2015 at 3:19 AM, Lele Gaifax
Zvezdan Petkovic
writes: Hi KP,
On Dec 2, 2015, at 7:00 PM, KP
wrote: It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
pip install foo.bar pip install -e svn+http://
Even if both tools have the namespace_package foo, the "foo.more" will not properly import.
How is this going against standard approaches?
I don’t know. Without seeing the source code for these packages (which Marius already asked for) everything is hypothetical. All I know is that I’m using namespaces successfully for over a decade now and never had the need to work around them.
Show us the code. We may be able to help better if we can try and reproduce. Otherwise, it’s hard.
It seems that KP case is either https://github.com/pypa/pip/issues/3160 or https://github.com/pypa/pip/issues/3, isn't it? Both come with sample code that demonstrate the problem.
ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi Kevin, Sorry for not being able to respond to this earlier. I have time only in the evening.
On Dec 3, 2015, at 3:57 AM, KP
wrote: Yes, the https://github.com/pypa/pip/issues/3 https://github.com/pypa/pip/issues/3 definitely sounds like my issue. Seems there is some concern over the nspkg.pth there as well. It seems taht the nspkg.pth is a fine idea to replace the install of the __init__.py, but that it just doesn’t work to fully extend the locations in which a namespace can reside.
I now understand what the issue is and can help perhaps with some advice below.
Either way, there a few other workarounds posted there as well, I will check them out to see if any of them are more palatable than the one I posted here.
You need to make foo.bar-1.0.0-py2.7-nspkg.pth look like this: import sys, types, os, pkgutil; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('foo',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('foo',types.ModuleType('foo')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); mp[:] = m and pkgutil.extend_path(mp, 'foo') or mp Notice the difference from the default in use of pkgutil.extend_path() that helps with editable packages. How to make this happen automatically is explained below.
Thanks Lele for sending that link!
-Kevin
I also want to thank Lele for pointing out the possible cause of your issues. Now, I’d like to separate two things here: 1. Defining namespaces correctly — that should be done as we talked before in the standard way. 2. Issue with the tooling — in this case pip is the tool that causes issues. Let’s talk about it more. These are the reasons I didn’t recognize Kevin’s issue. :-) - The namespaces work great when defined correctly (#1 above) with zc.buildout and other tools. There are good open-source build systems out there that take care of namespaces properly. - When one makes a good custom build system, it can fix the pip deficiencies too. For example, the build system specifically used in a company, team, … So, my advice is to: 1. Make a custom distutils Distribution extension, for example, MyDistribution (or some better name) (a) make custom install_egg_info class that overrides either: - two templates for pth files (for newer versions of setuptools) or - install_namespaces method (for older versions of setuptools) (b) override get_command_class method in the MyDistribution class with your own install_egg_info 2. Build this package and install it. 3. Use distclass=MyDistribution in your foo package setup.py (import it from your package). Using this approach everything will work for every of your packages automatically. I attached your original code from foo_test.zip and added to it a minimal mydist package + a sample __init__.py file that I think you should use in your namespace packages (if you want). Kevin, you can try this by doing the following: - create a virtual environment - source bin/activate - go to mydist package and run “python setup.py sdist” - pip install /path/to/mydist-1.0.0.tar.gz - go to your foo_bar package and run “python setup.py sdist” - pip install /path/to/foo.bar-1.0.0.tar.gz - pip install -e /path/to/foo_more - start the python interpreter in your virtual environment - confirm that it works >>> import foo.bar foo.bar imported >>> import foo.more foo.more imported I know that this is not a fix for pip, but it is a generic solution that I hope helps you. Perhaps I should make a little better example (better names, better __init__.py, …) and post it as a little project on GitHub that people can refer to. All the best, Zvezdan
On Thu, Dec 3, 2015 at 3:19 AM, Lele Gaifax
mailto:lele@metapensiero.it> wrote: Zvezdan Petkovic mailto:zvezdanpetkovic@gmail.com> writes: Hi KP,
On Dec 2, 2015, at 7:00 PM, KP
mailto:patter001@gmail.com> wrote: It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
pip install foo.bar pip install -e svn+http://
Even if both tools have the namespace_package foo, the "foo.more" will not properly import.
How is this going against standard approaches?
I don’t know. Without seeing the source code for these packages (which Marius already asked for) everything is hypothetical. All I know is that I’m using namespaces successfully for over a decade now and never had the need to work around them.
Show us the code. We may be able to help better if we can try and reproduce. Otherwise, it’s hard.
It seems that KP case is either https://github.com/pypa/pip/issues/3160 https://github.com/pypa/pip/issues/3160 or https://github.com/pypa/pip/issues/3 https://github.com/pypa/pip/issues/3, isn't it? Both come with sample code that demonstrate the problem.
ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it mailto:lele@metapensiero.it | -- Fortunato Depero, 1929.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Thank you Zvezdan! I will try to make a branch of https://github.com/lelit/pipinstall-e_bugex injecting your solution to see if it works in that case too. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Lele Gaifax
I will try to make a branch of https://github.com/lelit/pipinstall-e_bugex injecting your solution to see if it works in that case too.
No, it doesn't appear making a difference, in that case. Thanks anyway! ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Hi Lele, I took a quick look at your GitHub demo project. For what it’s worth, I do not have issues with editable packages and package dir, but I also: - use setuptools.find_packages(’src’) to get my packages automatically (this shouldn’t be important) - use package_dir={‘’: ’src’} - have my namespace properly declared namespace_package=[‘…’] - have my package properly structured under src: src | bugex —> __init__.py (a namespace __init__.py) | foo —> __init__.py (empty __init__.py for a package) + other modules - use install_package_data=True See for example setup.py in https://github.com/zopefoundation/zope.minmax Did you try that? I’ll take a better look in the evening when I have more time. Zvezdan
On Dec 4, 2015, at 7:22 AM, Lele Gaifax
wrote: Lele Gaifax
writes: I will try to make a branch of https://github.com/lelit/pipinstall-e_bugex injecting your solution to see if it works in that case too.
No, it doesn't appear making a difference, in that case.
Thanks anyway!
ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Zvezdan Petkovic
Did you try that?
Yes, indeed that way it works, an all my packages do that. But, as explained by the https://github.com/pypa/pip/issues/3160 issue, I would like to use a shallower tree for some of them. Thanks for taking a look! ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Hi Lele, I experimented with your GitHub demo project. First, let’s state that the custom distclass I sent previously on this thread definitely helps Kevin (KP, the original poster) with the editable packages in a namespace. Unfortunately, the fundamental problem and the reason why it doesn’t help you is as follows: - The solution relies on the presence of a *.pth file for the namespace that extends the path to include the editable package which has only the *.egg-link file - When you install a package, it gets the correct file structure in site-packages as bugex/foo/__init__.py and you have a *.pth file so everything works fine (no need for custom distclass) - When python imports such a package it follows the file-structure in site-packages and succeeds. - When you install it as an editable package, there’s no *.pth file, only *.egg-link that leads to your source - However, the source code does not have a directory for bugex (the namespace) and it fails to import In essence, the Python importer expects a directory for each part separated with dot. Thus, if there’s import bugex.foo it expects a directory for bugex which contains a foo directory. I’m afraid your editable package must follow the file structure that gets installed into site-packages or you’ll have to write a custom importer for such packages. In conclusion: 1. The solution I sent helps with regularly structured packages installed editable as a part of a namespace. (the pip issue #3) 2. I don’t see the way to use the flatten file structure as editable. (the issue #3160 that you opened) Sorry for not being of too much help for your issue. Maybe someone else might help. Honestly, I do not have a compelling motive for not keeping the packages and namespaces hierarchically structured even with Python 3. We usually do not nest our namespaces deeper than 2 anyway (e.g., zope.app.session). In Java development it’s quite common to see file structure like this: src/java/main/com/somecompany/x/y/z/SomeClass.java src/java/test/com/somecompany/x/y/z/TestSomeClass.java src/scala/main/…. You get the idea. And all the directories until z would have no code in them. At least we don’t need to do that much nesting. :-) All the best, Zvezdan
On Dec 4, 2015, at 12:19 PM, Lele Gaifax
wrote: Zvezdan Petkovic
writes: Did you try that?
Yes, indeed that way it works, an all my packages do that. But, as explained by the https://github.com/pypa/pip/issues/3160 issue, I would like to use a shallower tree for some of them.
Thanks for taking a look!
ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I apologize for bringing up this old thread again, but I was working on
another namespace package, and going through this posted BKM again. First
off I noticed I did not give a good Thank You for posting the fix, my
apologies there.
I noticed the original issue (https://github.com/pypa/pip/issues/3) was
still open, and was wondering, what is the downfall for the new .pth file
that was posted. Any reason that the proposed fix could not be permanently
added to setuptools?
On Fri, Dec 4, 2015 at 12:31 AM, Zvezdan Petkovic wrote: Hi Kevin, Sorry for not being able to respond to this earlier. I have time only in
the evening. On Dec 3, 2015, at 3:57 AM, KP Yes, the https://github.com/pypa/pip/issues/3 definitely sounds like my
issue. Seems there is some concern over the nspkg.pth there as well. It
seems taht the nspkg.pth is a fine idea to replace the install of the
__init__.py, but that it just doesn’t work to fully extend the locations in
which a namespace can reside. I now understand what the issue is and can help perhaps with some advice
below. Either way, there a few other workarounds posted there as well, I will
check them out to see if any of them are more palatable than the one I
posted here. You need to make foo.bar-1.0.0-py2.7-nspkg.pth look like this: import sys, types, os, pkgutil; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('foo',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not
ie and sys.modules.setdefault('foo',types.ModuleType('foo')); mp = (m
or []) and m.__dict__.setdefault('__path__',[]); (p not in mp)
and mp.append(p); mp[:] = m and pkgutil.extend_path(mp, 'foo') or mp Notice the difference from the default in use of pkgutil.extend_path()
that helps with editable packages. How to make this happen automatically is explained below. Thanks Lele for sending that link! -Kevin I also want to thank Lele for pointing out the possible cause of your
issues. Now, I’d like to separate two things here: 1. Defining namespaces correctly — that should be done as we talked before
in the standard way. 2. Issue with the tooling — in this case pip is the tool that causes
issues. Let’s talk about it more. These are the reasons I didn’t recognize Kevin’s
issue. :-) - The namespaces work great when defined correctly (#1 above) with
zc.buildout and other tools.
There are good open-source build systems out there that take care of
namespaces properly. - When one makes a good custom build system, it can fix the pip
deficiencies too.
For example, the build system specifically used in a company, team, … So, my advice is to: 1. Make a custom distutils Distribution extension, for example,
MyDistribution (or some better name) (a) make custom install_egg_info class that overrides either: - two templates for pth files (for newer versions of
setuptools) or
- install_namespaces method (for older versions of setuptools) (b) override get_command_class method in the MyDistribution class with
your own install_egg_info 2. Build this package and install it. 3. Use distclass=MyDistribution in your foo package setup.py (import it
from your package). Using this approach everything will work for every of your packages
automatically. I attached your original code from foo_test.zip and added to it a minimal
mydist package + a sample __init__.py file that I think you should use in
your namespace packages (if you want). Kevin, you can try this by doing the following: - create a virtual environment
- source bin/activate
- go to mydist package and run “python setup.py sdist”
- pip install /path/to/mydist-1.0.0.tar.gz
- go to your foo_bar package and run “python setup.py sdist”
- pip install /path/to/foo.bar-1.0.0.tar.gz
- pip install -e /path/to/foo_more
- start the python interpreter in your virtual environment
- confirm that it works import foo.bar
foo.bar imported
import foo.more
foo.more imported I know that this is not a fix for pip, but it is a generic solution that I
hope helps you. Perhaps I should make a little better example (better names, better
__init__.py, …) and post it as a little project on GitHub that people can
refer to. All the best, Zvezdan On Thu, Dec 3, 2015 at 3:19 AM, Lele Gaifax Zvezdan Petkovic Hi KP, On Dec 2, 2015, at 7:00 PM, KP It looks like you are trying to find a workaround for the problem
that perhaps is not a problem at all if you use the standard approach
properly. I'm definitely _trying_ to use a standard approach...That is why I am
here posting. Put simply this seems like a valid use case: pip install foo.bar
pip install -e svn+http:// Even if both tools have the namespace_package foo, the "foo.more" will
not properly import. How is this going against standard approaches? I don’t know.
Without seeing the source code for these packages (which Marius already
asked for) everything is hypothetical.
All I know is that I’m using namespaces successfully for over a decade
now and never had the need to work around them. Show us the code. We may be able to help better if we can try and
reproduce.
Otherwise, it’s hard. It seems that KP case is either https://github.com/pypa/pip/issues/3160
or
https://github.com/pypa/pip/issues/3, isn't it? Both come with sample
code
that demonstrate the problem. ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it | -- Fortunato Depero, 1929. _______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
participants (4)
-
KP
-
Lele Gaifax
-
Marius Gedminas
-
Zvezdan Petkovic