how can I egt the packages and modules in a distribution?
Hi All, Given a distribution name (ie: one of those things setuptools uses, 'name' parameter to setup.py's setup call, etc) of a dsitrbution that has been installed with easy_install, buildout, pip, etc, how can I find out what python packages (ie: dotted names that I can import) are provided by that package? I'm trying to write a scanner than looks for SQLAlchemy table objects and it would be great to be able to pass a distribution name rather than a set of dotted package names... cheers, Chris
At 06:29 PM 8/19/2010 +0100, Chris Withers wrote:
Hi All,
Given a distribution name (ie: one of those things setuptools uses, 'name' parameter to setup.py's setup call, etc) of a dsitrbution that has been installed with easy_install, buildout, pip, etc, how can I find out what python packages (ie: dotted names that I can import) are provided by that package?
I'm trying to write a scanner than looks for SQLAlchemy table objects and it would be great to be able to pass a distribution name rather than a set of dotted package names...
get_distribution('projectname').get_metadata('top_level.txt').splitlines() will return you a list of top-level package or module names provided by the package. Subpackages you'll have to find yourself; the pkgutil module in 2.5+ has a walk_packages() that might be helpful. (Note that if there are packages installed in "flat" form (e.g. pip-installed or system-installed), you may see overlapping packages in the same directory, so simply walking packages in a distribution object's base directory may include packages and modules from another project.)
P.J. Eby wrote:
I'm trying to write a scanner than looks for SQLAlchemy table objects and it would be great to be able to pass a distribution name rather than a set of dotted package names...
get_distribution('projectname').get_metadata('top_level.txt').splitlines() will return you a list of top-level package or module names provided by the package.
Ah, okay. I assume this will only work for packages that use setuptools?
Subpackages you'll have to find yourself; the pkgutil module in 2.5+ has a walk_packages() that might be helpful.
(Note that if there are packages installed in "flat" form (e.g. pip-installed or system-installed), you may see overlapping packages in the same directory, so simply walking packages in a distribution object's base directory may include packages and modules from another project.)
I assume this is only a problem for namespace packages? (ie: if so packages each supply part of a namespace, they will both have the namespace listed in top_level.txt, right?) How do static resources returned with pkg_resources.resource_filename and friends fare when packages are installed in a "flat" form? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
At 11:32 PM 8/19/2010 +0100, Chris Withers wrote:
P.J. Eby wrote:
I'm trying to write a scanner than looks for SQLAlchemy table objects and it would be great to be able to pass a distribution name rather than a set of dotted package names... get_distribution('projectname').get_metadata('top_level.txt').splitlines() will return you a list of top-level package or module names provided by the package.
Ah, okay. I assume this will only work for packages that use setuptools?
Or distribute. Or pip (I assume). Anything but plain distutils, basically.
I assume this is only a problem for namespace packages? (ie: if so packages each supply part of a namespace, they will both have the namespace listed in top_level.txt, right?)
Correct.
How do static resources returned with pkg_resources.resource_filename and friends fare when packages are installed in a "flat" form?
They go in the same relative location as they would in an egg, it's just that in flat layouts everyone's files are in the same namespace. Sort of a "scrambled egg", if you will. ;-)
P.J. Eby wrote:
Ah, okay. I assume this will only work for packages that use setuptools?
Or distribute. Or pip (I assume). Anything but plain distutils, basically.
Well, the packages import setup from setuptools for all of the above, apart from plain distutils ;-)
How do static resources returned with pkg_resources.resource_filename and friends fare when packages are installed in a "flat" form?
They go in the same relative location as they would in an egg, it's just that in flat layouts everyone's files are in the same namespace.
Sort of a "scrambled egg", if you will. ;-)
Okay, so if we have two distributions, x.a and x.b both define template folders called 'templates', and each has a template called 'master', what ends up on disk? what does pkg_resources.resource_filename('x.a','templates/master') return? It should be different to pkg_resources.resource_filename('x.b','templates/master'), but will it be? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
At 12:20 AM 8/20/2010 +0100, Chris Withers wrote:
P.J. Eby wrote:
Ah, okay. I assume this will only work for packages that use setuptools? Or distribute. Or pip (I assume). Anything but plain distutils, basically.
Well, the packages import setup from setuptools for all of the above, apart from plain distutils ;-)
If you use any of those tools to do the installing, it doesn't matter whether the package itself imports setuptools or distutils.
How do static resources returned with pkg_resources.resource_filename and friends fare when packages are installed in a "flat" form? They go in the same relative location as they would in an egg, it's just that in flat layouts everyone's files are in the same namespace. Sort of a "scrambled egg", if you will. ;-)
Okay, so if we have two distributions, x.a and x.b both define template folders called 'templates', and each has a template called 'master', what ends up on disk? what does pkg_resources.resource_filename('x.a','templates/master') return? It should be different to pkg_resources.resource_filename('x.b','templates/master'), but will it be?
You seem to be confusing distributions and packages. If 'x.a' and 'x.b' are package names, then each will have its own directory ('x/a/templates' or 'x/b/templates'). If they used install_data to specify non-package data, OTOH, it'll be mixed. That's why the recommendation is to use package_data or include_package_data; if both projects included their data using one of those two setup() options, they'll be fine.
P.J. Eby wrote:
How do static resources returned with pkg_resources.resource_filename and friends fare when packages are installed in a "flat" form? They go in the same relative location as they would in an egg, it's just that in flat layouts everyone's files are in the same namespace. Sort of a "scrambled egg", if you will. ;-)
Okay, so if we have two distributions, x.a and x.b both define template folders called 'templates', and each has a template called 'master', what ends up on disk? what does pkg_resources.resource_filename('x.a','templates/master') return? It should be different to pkg_resources.resource_filename('x.b','templates/master'), but will it be?
You seem to be confusing distributions and packages.
Potentially, is the first argument to resource_filename a distributions name or a dotted package named? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
P.J. Eby wrote:
Okay, so if we have two distributions, x.a and x.b both define template folders called 'templates', and each has a template called 'master', what ends up on disk? what does pkg_resources.resource_filename('x.a','templates/master') return? It should be different to pkg_resources.resource_filename('x.b','templates/master'), but will it be?
You seem to be confusing distributions and packages. If 'x.a' and 'x.b' are package names, then each will have its own directory ('x/a/templates' or 'x/b/templates'). If they used install_data to specify non-package data, OTOH, it'll be mixed.
Now that I'm somewhere I can read the docs, I can be clearer. 'x.a' and 'x.b' were distribution names, such distributions names are quite common. However, for absolute clarity, assuming we have: - A distribution called 'XA', that contains an 'x.a' package but also a 'templates' directory at the root of the distribution, containing a 'master' template - A distribution called 'XB', that contains an 'x.b' package but also a 'templates' directory at the root of the distribution, containing a 'master' template. Which template do I get if I do: pkg_resources.resource_string(tuple(pkg_resources.parse_requirements('XA'))[0],'templates/master') ...in the context of pip or another flat-structure install? Would it work okay if each distribution was unpacked in its own folder (as buildout and easy_install do)? cheers, Chris
At 11:49 AM 8/20/2010 +0100, Chris Withers wrote:
P.J. Eby wrote:
Okay, so if we have two distributions, x.a and x.b both define template folders called 'templates', and each has a template called 'master', what ends up on disk? what does pkg_resources.resource_filename('x.a','templates/master') return? It should be different to pkg_resources.resource_filename('x.b','templates/master'), but will it be? You seem to be confusing distributions and packages. If 'x.a' and 'x.b' are package names, then each will have its own directory ('x/a/templates' or 'x/b/templates'). If they used install_data to specify non-package data, OTOH, it'll be mixed.
Now that I'm somewhere I can read the docs, I can be clearer. 'x.a' and 'x.b' were distribution names, such distributions names are quite common. However, for absolute clarity, assuming we have:
- A distribution called 'XA', that contains an 'x.a' package but also a 'templates' directory at the root of the distribution, containing a 'master' template
- A distribution called 'XB', that contains an 'x.b' package but also a 'templates' directory at the root of the distribution, containing a 'master' template.
Which template do I get if I do:
pkg_resources.resource_string(tuple(pkg_resources.parse_requirements('XA'))[0],'templates/master')
...in the context of pip or another flat-structure install?
You get whatever was installed last.
Would it work okay if each distribution was unpacked in its own folder (as buildout and easy_install do)?
Yep. And of course, it'll also work if they put their templates under x/a or x/b, like they really ought to be doing in the first place, for precisely this reason. (Alternatively, they can put their stuff under the project's egg-info directory, the way EggTranslations does.)
P.J. Eby wrote:
Which template do I get if I do:
pkg_resources.resource_string(tuple(pkg_resources.parse_requirements('XA'))[0],'templates/master')
...in the context of pip or another flat-structure install?
You get whatever was installed last.
Is pkg_resources.resource_string(<requirement>,..) *ever* useful or sane then? It doesn't sound like it, in which case, could it be deprecated and removed please?
(Alternatively, they can put their stuff under the project's egg-info directory, the way EggTranslations does.)
What's EggTranslations and where can I see its code? Chris
At 03:19 PM 8/20/2010 +0100, Chris Withers wrote:
P.J. Eby wrote:
Which template do I get if I do:
pkg_resources.resource_string(tuple(pkg_resources.parse_requirements('XA'))[0],'templates/master')
...in the context of pip or another flat-structure install? You get whatever was installed last.
Is pkg_resources.resource_string(<requirement>,..) *ever* useful or sane then?
Good point. The installation of data in that way was for backward-compatibility with packages using install_data; but such packages by definition are unlikely to be using the resource APIs in the first place.
It doesn't sound like it, in which case, could it be deprecated and removed please?
I'll give it some serious thought.
(Alternatively, they can put their stuff under the project's egg-info directory, the way EggTranslations does.)
What's EggTranslations and where can I see its code?
participants (2)
-
Chris Withers -
P.J. Eby