Using code from a dependency in setup.py
I have a little web framework, let's call it Bread, which is used to build applications like Jam, Marmalade, PeanutButter, and other toppings. Bread both *builds* and *serves* these applications. I'm trying to figure out how to make the applications' `setup.py`s work, given the following requirements: * The apps depend upon Bread, via [setuptool's install_requires]( http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies) * To build an application at development time, Bread reads some config and then emits assets (HTML, JS, CSS, images, etc) to the application's `output` directory. In other words, `bread devserver` reads `Jam/bread.yaml` and assembles assets in `Jam/output`, then serves the application (via Flask, but that's not otherwise pertinent). * In order to build a *deployable* Jam application, I want to invoke Bread during `python setup.py install` of Jam, to build `Jam/output`. In production, Jam should not need to build anything. * I've defined a custom `bdist_egg` setup command where `initialize_options` imports Bread, invokes the builder, then sets `self.distribution.data_files` with the appropriate tuples, before calling the base class. (And *that* was no fun to figure out.) * Right now, the `bdist_egg` is defined in Jam's `setup.py`. I want to move this and other boilerplate code into `bread.setup`, so that I can reuse it in Marmalade, PeanutButter, etc. * Potentially, this means that I'm now importing Bread code before Bread has been installed. This will surely arise in a clean install, such as a fresh virtualenv on a build machine. Can this be done with Distutils / setuptools / Distribute? -- /George V. Reilly george@reilly.org Twitter: @georgevreilly http://www.georgevreilly.com/blog http://blogs.cozi.com/tech
On 30 May 2013 07:33, George V. Reilly
Can this be done with Distutils / setuptools / Distribute?
I believe that setup_requires can do what you want, in principle -
AIUI that describes what modules are needed to run setup correctly :
of course you have to bootstrap manually - but installers can examine
the EGG_INFO once it's created. I suspect only ezsetup does that
though - though I'd be delighted to be wrong.
-Rob
--
Robert Collins
Here is my explanation of why it can be problematic if you import Bread in Bread's setup.py: https://bugs.launchpad.net/nevow/+bug/812537/comments/3 If you do want to import dependencies of Bread in Bread's setup.py, and if your build tool (e.g. setuptools) supports it, then you could add the dependencies to the "setup_requires" list. If possible, maybe you could move some of this work from setup.py-time (whether build-time or install-time) to "first time I run" or a custom "initialize me now" command of Bread. I think setup.py is a sub-optimal place to do stuff, because there are a lot of complicated things going then and there, and because it is very inconvenient to write unit tests of setup.py behavior. Regards, Zooko
Actually, Bread's setup is fine; it's not doing anything interesting.
It's Jam's setup that's the problem.
Robert's mention of `setup_requires` led me to
http://stackoverflow.com/a/12061891/6364, which gave me the hint
I needed: create a separate `Distribution` object before calling `setup`
which defines the `setup_requires` entries.
Jam's `setup.py` now looks like:
from setuptools import setup, dist
dist.Distribution(dict(setup_requires='Bread'))
from bread.setup_topping import *
setup(
name='Jam',
version='0.2',
long_description=open('README.md').read(),
**topping_setup_options
)
# Remove *.egg left by bootstrapping Bread
cleanup_bread_bootstrap()
And `bread/setup_topping.py` looks like:
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
import os, fnmatch, glob, shutil
def recursive_data_files(treeroot, pattern):
results = []
for base, dirs, files in os.walk(treeroot):
goodfiles = fnmatch.filter(files, pattern)
if goodfiles:
results.append((base, [os.path.join(base, f) for f in goodfiles]))
return results
def make_data_files(output='output'):
return (
[('', ['bread.yaml'])]
+ recursive_data_files(output, '*')
)
class bdist_egg(_bdist_egg):
def initialize_options(self):
bake_bread() # build files to './output'
self.distribution.data_files = make_data_files()
_bdist_egg.initialize_options(self)
topping_setup_options = dict(
cmdclass={
'bdist_egg': bdist_egg,
},
install_requires=[
'Bread',
],
zip_safe=False,
)
def cleanup_bread_bootstrap(root='.'):
for f in glob.glob(os.path.join(os.path.abspath(root), '*.egg')):
if os.path.isdir(f):
shutil.rmtree(f) # Egg directory
else:
os.remove(f) # Zipped Egg
On Wed, May 29, 2013 at 2:48 PM, zooko
Here is my explanation of why it can be problematic if you import Bread in Bread's setup.py:
https://bugs.launchpad.net/nevow/+bug/812537/comments/3
If you do want to import dependencies of Bread in Bread's setup.py, and if your build tool (e.g. setuptools) supports it, then you could add the dependencies to the "setup_requires" list.
If possible, maybe you could move some of this work from setup.py-time (whether build-time or install-time) to "first time I run" or a custom "initialize me now" command of Bread. I think setup.py is a sub-optimal place to do stuff, because there are a lot of complicated things going then and there, and because it is very inconvenient to write unit tests of setup.py behavior.
Regards,
Zooko
participants (3)
-
George V. Reilly -
Robert Collins -
zooko