-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/16/2014 12:57 PM, Jim Fulton wrote:

On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtković wrote:

...

Hello packaging community,

I'm investigating ways of setting up Python projects at my workplace. We're predominantly a Java shop, but we might be dipping our toes in Python waters (finally!) due to a fortuitous project and my multi-year insistence, so I'm contemplating how to set up our Python build system to minimize workflow differences for other developers (well, and myself).

I've actually written uš a lengthy description of Maven and why we use it but I'll spare you for now. :) To keep the story short, I'm interested in options for setting up a multi-module Python project. By 'multi-module' I don't mean a single setuptools-based project with several .py files inside, but a way of triggering a complex build process with a single command that would build all sub-modules (essentially sub-projects) and produce a number of end artifacts - just like Maven. Imagine a repository containing 30 separate Django apps, packaged independently, 10 utility libraries, 10 Django projects combining those app, and 10 RPM building projects for packaging it all up for deployment.

As far as I know, just using setuptools isn't adequate for a workflow like this - setuptools deals with the build process (testing, packaging, etc) of a single project only. Solutions that come to mind are: a hierarchy of Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks like Maven for Python but would probably need contributions to do what we want, and looks predisposed to building PEX files which, while very interesting, I'm not looking to do right now. None of these solutions are really ideal, especially if I want to support development on Windows (which I absolutely want).

I've even thought about actually using Maven, but that's just a Pandora's box of problems waiting to happen.

I'd appreciate insight on this from anyone who's thought about (and maybe solved) problems like this. I'm also willing to engage and contribute to improving the situation, especially if there's low hanging fruit to be picked. How do other companies handle large Python repositories with a lot of subcomponents?

Checkout Buildout, http://www.buildout.org

It addresses a lot of the same problems. Of course, for Python, compiling is (mostly) not all that important. For us (Zope Corporation) buildout is more about assembly/deployment, both for development and production, that building.

buildout also allow for reproducible builds of things that *do* require compilation (e.g., via the 'zc.reciple.cmmi' recipe for 'configure-make-make install' software). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlNOwLcACgkQ+gerLs4ltQ46aACgts47yE/ErKtqag0FyOEpfCZP 4pAAoI8d04UodGt3NQB0AzPPJbJITPb/ =vBpI -----END PGP SIGNATURE-----

Hi Brian, thanks for the reply. I'm definitely excited about Pants. I see there's a pants-users group now, but it's currently empty (as in zero posts - crickets and tumbleweeds :) ). Would that be a good place to ask questions when I get around to playing with Pants? It also looks like Pants development is very active right now. Would you suggest waiting a while before diving into the docs and code? I don't really need Pants to be super easy to use (your suggested one year period), just to be able to understand it and maybe write a few modules I need. I guess I'm asking if the core is stable enough for semi-serious plugin development. Cheers! On 16.04.2014 19:32, Brian Wickman wrote:

Pants can definitely do what you want, but you're probably right in that it requires significant up-front investment. It's not strictly tied to PEX files (it can recursively generate setup.py-based projects from a transitive closure of BUILD dependencies) but the learning curve is non-trivial and most attention is given to the Java/Scala backends. Python support will improve over time but it may be a year or more before this is really easy to adopt.

~brian

On Wed, Apr 16, 2014 at 12:18 PM, Tin Tvrtković mailto:tinchester@gmail.com> wrote:

Hello packaging community,

I'm investigating ways of setting up Python projects at my workplace. We're predominantly a Java shop, but we might be dipping our toes in Python waters (finally!) due to a fortuitous project and my multi-year insistence, so I'm contemplating how to set up our Python build system to minimize workflow differences for other developers (well, and myself).

I've actually written uš a lengthy description of Maven and why we use it but I'll spare you for now. :) To keep the story short, I'm interested in options for setting up a multi-module Python project. By 'multi-module' I don't mean a single setuptools-based project with several .py files inside, but a way of triggering a complex build process with a single command that would build all sub-modules (essentially sub-projects) and produce a number of end artifacts - just like Maven. Imagine a repository containing 30 separate Django apps, packaged independently, 10 utility libraries, 10 Django projects combining those app, and 10 RPM building projects for packaging it all up for deployment.

As far as I know, just using setuptools isn't adequate for a workflow like this - setuptools deals with the build process (testing, packaging, etc) of a single project only. Solutions that come to mind are: a hierarchy of Makefiles, shell scripts, or maybe Twitter's Pants, which sort of looks like Maven for Python but would probably need contributions to do what we want, and looks predisposed to building PEX files which, while very interesting, I'm not looking to do right now. None of these solutions are really ideal, especially if I want to support development on Windows (which I absolutely want).

I've even thought about actually using Maven, but that's just a Pandora's box of problems waiting to happen.

I'd appreciate insight on this from anyone who's thought about (and maybe solved) problems like this. I'm also willing to engage and contribute to improving the situation, especially if there's low hanging fruit to be picked. How do other companies handle large Python repositories with a lot of subcomponents?

Kind regards, Tin _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org mailto:Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 16 April 2014 13:32, Brian Wickman wrote:

Pants can definitely do what you want, but you're probably right in that it requires significant up-front investment. It's not strictly tied to PEX files (it can recursively generate setup.py-based projects from a transitive closure of BUILD dependencies) but the learning curve is non-trivial and most attention is given to the Java/Scala backends. Python support will improve over time but it may be a year or more before this is really easy to adopt.

It also depends heavily on whether cross-platform support is needed. Fedora uses koji for this kind of thing, for example, as that's a general purpose RPM build system, which handles Python along with everything else. The conda/binstar.org approach created by Continuum Analytics handles the same issue from a cross-platform perspective. Cheers, Nick. P.S. For folks that may sometimes wonder at the high level of complexity in the metadata 2.0 specs, this kind of use case has a lot to do with it. I want metadata 2.0 to work not only for the "direct consumption of upstream packages" model favoured by Software-as-a-Service developers, but also to integrate cleanly with downstream redistributors (whether Linux distros, redistributors like Enthought, ActiveState and Continuum Analytics or custom in-house deployment mechanisms). -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia