[Distutils] Enterprise Python - Multicomponent Projects

Tin Tvrtković tinchester at gmail.com
Wed Apr 16 18:18:16 CEST 2014

Hello packaging community,

I'm investigating ways of setting up Python projects at my workplace. 
We're predominantly a Java shop, but we might be dipping our toes in 
Python waters (finally!) due to a fortuitous project and my multi-year 
insistence, so I'm contemplating how to set up our Python build system 
to minimize workflow differences for other developers (well, and myself).

I've actually written uš a lengthy description of Maven and why we use 
it but I'll spare you for now. :) To keep the story short, I'm 
interested in options for setting up a multi-module Python project. By 
'multi-module' I don't mean a single setuptools-based project with 
several .py files inside, but a way of triggering a complex build 
process with a single command that would build all sub-modules 
(essentially sub-projects) and produce a number of end artifacts - just 
like Maven. Imagine a repository containing 30 separate Django apps, 
packaged independently, 10 utility libraries, 10 Django projects 
combining those app, and 10 RPM building projects for packaging it all 
up for deployment.

As far as I know, just using setuptools isn't adequate for a workflow 
like this - setuptools deals with the build process (testing, packaging, 
etc) of a single project only. Solutions that come to mind are: a 
hierarchy of Makefiles, shell scripts, or maybe Twitter's Pants, which 
sort of looks like Maven for Python but would probably need 
contributions to do what we want, and looks predisposed to building PEX 
files which, while very interesting, I'm not looking to do right now. 
None of these solutions are really ideal, especially if I want to 
support development on Windows (which I absolutely want).

I've even thought about actually using Maven, but that's just a 
Pandora's box of problems waiting to happen.

I'd appreciate insight on this from anyone who's thought about (and 
maybe solved) problems like this. I'm also willing to engage and 
contribute to improving the situation, especially if there's low hanging 
fruit to be picked. How do other companies handle large Python 
repositories with a lot of subcomponents?

Kind regards,

More information about the Distutils-SIG mailing list