[Distutils] Problems with Python eggs/setuptools

Phillip J. Eby pje at telecommunity.com
Mon Aug 8 18:45:54 CEST 2005

At 06:28 PM 8/8/2005 +0200, Walter Dörwald wrote:
>Phillip J. Eby wrote:
>>At 07:29 PM 8/5/2005 +0200, Walter Dörwald wrote:
>>>Hello Philip!
>>>I've started to play around with your easy_install script, since 
>>>managing our packages gets more complicated with each version.
>>>After I've installed setuptools via ez_setup.py I've tried downloading 
>>>the ll-xist package via "python -m easy_install ll-xist" but got a stack 
>>>trace instead (see the attached stacktrace.txt). The problem seems to be 
>>>the umlaut in my name. As PyPI requires UTF-8 encoded strings now, I've put a
>>>    author=u"Walter Dörwald".encode("utf-8")
>>>in my setup.py when I registered the package (but a plain
>>>    author=u"Walter Dörwald"
>>>in the setup.py in the package).
>>>To work around this problem it's possible to set the system default 
>>>encoding to Latin-1. I don't know if this is a problem with setuptools 
>>>or distutils, but doing a simple "python setup.py install" works.
>>Note that this problem is a distutils problem writing PKG-INFO files; 
>>it's not anything specific to setuptools.  I would recommend you use the 
>>.encode('utf-8') workaround until there is an official policy for the 
>>distutils to deal with encodings in PKG-INFO.
>As PyPI espects UTF-8 now, I'd say it makes sense to use UTF-8 in package 
>info files too.

That's probably true, but this probably needs at least some distutils-sig 
discussion and maybe a PEP update.

>>>The other problem seems more severe: ll-xist is installed as the 
>>>packages ll.xist, but the package init file ll/__init__.py is *not* 
>>>provided by this package, but by the ll-core package instead. If I 
>>>correctly distributing subpackages via setuptools only works if the 
>>>package init file is empty, which it isn't in this case.
>>Technically, you can do it, it's just not guaranteed that the non-empty 
>>one will be executed, unless it's first on sys.path.
> >
>>>  Is there any workaround for this?
>>You can distribute an identical __init__.py with each distribution that 
>>shares that parent package.
>I tried to avoid that, because the Windows installer then asks the user, 
>whether files should be overwritten or not when installing the second 
>package, and for package managers like RPM, Debian .deb or Gentoo emerge 
>it's also problematic, if one file (even if it is empty) is provided by 
>several packages.

Maybe the best solution, then, is to arrange things such that these 
installers install the packages as eggs, since this would then avoid any 
overwrite issues.  This is pretty much how eggs are supposed to work with 
packaging systems in the long run.

>>and make sure that all the distributions that don't contain the _my_init 
>>module have declared a dependency to the one that does.
>OK, this would work, but the clearer alternative would be to simply move 
>everything from ll to ll.core and have ll as an empty package module 
>again. Nevertheless this would require that each package includes the 
>empty __init__.py again, with the problems mentioned above.

Hm.  Tricky.  I'll have to give this some thought, because setuptools *has* 
to have the __init__, since separately distributed packages are in separate 
directories or zipfiles, but it never occurred to me that you could hack 
the distutils to do partial distribution in this way.  So, I never came up 
with a strategy for it.

Ideally, it sounds like I should have a way to ensure that when setuptools 
builds an egg, it should create empty __init__.py files if they don't 
exist.  That way, you could leave out the __init__.py for other builds, but 
it would ensure the package exists for the egg build.  The tricky part is 
that I can't do it in the build directory, because then it might sneak into 
other distributions.  But it's probably doable.

>>If you want to try it, change this line in the _handle_ns routine of 
>>     module = sys.modules.get(packageName) or loader.load_module(packageName)
>>to read instead:
>>     module = loader.load_module(packageName)
>>and this will cause it to load *all* of the __init__ modules.  However, 
>>the order of execution is still not guaranteed, and I'm not certain that 
>>it might not cause an __init__ to be reloaded if its grandparent 
>>directory is included twice on sys.path.  Anyway, let me know if that 
>>change works for you, and I will think some more on the "load all 
>>__init__ modules" strategy.  Even if I do implement it, I will want to 
>>mainly advise people to use empty __init__ modules for any new namespace 
>>packages they create.
>If I've understoodd this correcty, it doesn't solve my problem, as I only 
>have *one* __init__.py. And as setuptools puts different subpackages in 
>different directories, that get added via .pth files, I'm not sure, if 
>there can be a simple solution.

Well, if I create the __init__ in the zipfile, and pkg_resources executes 
all the __init__ files, then it should work even with your current layout, 
as long as there's a dependency declared from the package without the 
__init__ to the package with it.

More information about the Distutils-SIG mailing list