----- Original Message -----

On 6/26/12 8:56 AM, Bohuslav Kabrda wrote:

...

Hi all, I recently came upon Tarek's blogpost [1] about converting Python packages to rpm specfiles. I have pretty good knowledge of this, as I am a Fedora package maintainer and Python is one of my main responsibilities. I have always wanted to become a Python developer and becoming part of distutils-sig and trying to give a helping hand with RPM related stuff is probably a good entry point for me :)

To introduce myself, I work at Red Hat as a package maintainer for dynamic languages, I am author of the new Fedora-used pyp2rpm [2], [3], which tries to do exactly what Py2RPM is supposed to be. (Well, it's not perfect, but it can do 90 % of the work, usually. So far, it doesn't work with distutils2, but I'll probably be working on that when Python 3.3 gets released.) I also have some minor not-yet-released projects in Python, I work with Django and I'm just a Python enthusiast. In Fedora, I am a maintainer/comaintainer of both Python and Python3, Django and another 15+ (and growing) set of packages.

Thanks for reading this through :) I'm looking forward to working with - and learning from - all of you.

Welcome !

As I stated in the blog post, I think a separate project for all RPM -related stuff would make a lot of sense

I agree. RPM (and distro-packaging in general) things have different lifecycle, so it may be nice to be able to reflect that.

FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI projects, we use at Mozilla to deploy our apps, http://pypi.python.org/pypi/pypi2rpm It's just a glue script on the top of an isolated bdist_rpm command and distutils2.version (for sorting versions) but you can also pass your spec file for a given project and have Fedora/RHEL specific options (like adding python26- prefixes etc)

It sounds like your project is on the other side, e.g. generating spec files out of Python projects, so I'd be really interested to see how it may replace bdist_rpm in pypi2rpm

Exactly, my project is aimed at creating just the specfiles. In Fedora (and other distros as well), there is a review process that each package must go through and having a specfile written according to the current guidelines is a necessary part of that. "pypi2rpm" solves a different use case than distro-packagers need.

I have a pretty good idea about how distro-packagers would like things to work and combining it with the "pypi2rpm", I see it this way: 1) Starting point: we have a Python package. 2) We want to: 2a) directly convert it to binary RPM => we use "pypi2rpm" - good for Mozilla to deploy apps, not useful for distro-packagers - Producing binary RPM probably could be left in distutils, but looking at the bigger picture, RPM has different lifecycle than python standard library, so it might be good to pull this functionality out. 2b) produce a specfile/whatever that is called in other distributions => use Fedora's "pyp2rpm"/is there something similar for other distributions? - This should definitely be present in a library outside of Python - we may want to provide templates/distro-specific stuff here and be able to react to new releases of different distros (when they change their way of packaging).

So the best approach that currently comes to my mind is having a functionality in distutils, that can provide the package metadata (license, runtime/build requirements) and separate tool(s) that would be able to use these metadata to a) convert directly to binary packages (rpm, deb, etc.) or b) produce specfiles/whatever for the packagers' reviews

Some questions that pop immediately are - Should a) and b) be one library or two? They handle packages in completely different way. - Should every distribution provide its very own library to handle a) and b) stuff or should these be a general libraries with distro-specific modules+templates? ...

How does this sound? This is just a general idea that came to my mind some time ago - feel free to dump it, if you don't like it :)

Have a nice day, Slavek.

Cheers Tarek _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

----- Original Message -----

On 6/26/12 1:23 PM, Bohuslav Kabrda wrote:

...

[...] Exactly, my project is aimed at creating just the specfiles. In Fedora (and other distros as well), there is a review process that each package must go through and having a specfile written according to the current guidelines is a necessary part of that. "pypi2rpm" solves a different use case than distro-packagers need.

I have a pretty good idea about how distro-packagers would like things to work and combining it with the "pypi2rpm", I see it this way:

1. Starting point: we have a Python package.
2. We want to:

2a) directly convert it to binary RPM => we use "pypi2rpm" - good for Mozilla to deploy apps, not useful for distro-packagers

• Producing binary RPM probably could be left in distutils, but

looking at the bigger picture, RPM has different lifecycle than python standard library, so it might be good to pull this functionality out.

Yeah -- I had no intent to keep bdist_rpm in the stdlib. We removed it from distiutils2

Ah, I see. I didn't have that much time to get familiar with distutils2 yet - I only know them from the documentation :)

...

2b) produce a specfile/whatever that is called in other distributions => use Fedora's "pyp2rpm"/is there something similar for other distributions?

You mean for RPM that are not Fedora/RHEL ? I have no idea.

...

• This should definitely be present in a library outside of

Python - we may want to provide templates/distro-specific stuff here and be able to react to new releases of different distros (when they change their way of packaging).

So the best approach that currently comes to my mind is having a functionality in distutils, that can provide the package metadata (license, runtime/build requirements)

you can already have this, with the current Distutils code.

True. What I meant by this is, that providing the metadata is the thing that distutils should do. In my opinion, they shouldn't do anything more (no distro-specific stuff).

...

and separate tool(s) that would be able to use these metadata to a) convert directly to binary packages (rpm, deb, etc.) or b) produce specfiles/whatever for the packagers' reviews

Some questions that pop immediately are

• Should a) and b) be one library or two? They handle packages in

completely different way.]

In a different project I think. For debian there's 'stdeb' - http://pypi.python.org/pypi/stdeb

...

• Should every distribution provide its very own library to handle

a) and b) stuff or should these be a general libraries with distro-specific modules+templates? ...

As a user I want to pick up *any* project, and build a deb or a rpm, even it the author did not provide the deb/rpm integration, even if I eventually need to tweak the process.

What I meant here is - should there be one big tool producing the specfiles/debian stuff/... or one project for RPM specfiles, one project for debian stuff...

...

How does this sound? This is just a general idea that came to my mind some time ago - feel free to dump it, if you don't like it :)

So what are your plans ?

Since I'm new here, I first want to watch and listen and contribute to the discussions. I would certainly like to discuss the distro-specific stuff in more detail and participate on a solution. That means first create a clear vision with more distutils people here and then actually take my part implementing it. It would certainly be nice to create something flexible that could do both the specfiles and binary packages for more distributions - for me, that's the goal here (and of course, Python rocks).

...

Have a nice day, Slavek.

...

Cheers Tarek _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 7/1/12 8:50 PM, Marcus wrote:

FWIW I have started a "pypi2rpm" project that creates RPM out of
PyPI projects, we use at Mozilla to deploy our apps,
http://pypi.python.org/pypi/pypi2rpm
It's just a glue script on the top of an isolated bdist_rpm
command and distutils2.version (for sorting versions) but you can
also pass your spec file for a given
project and have Fedora/RHEL specific options (like adding
python26- prefixes etc)

Hello Tarek: if you have anything that details your rpm deployment workflow at mozilla, i'd love to see it. I work in a very active build environment, and repetitive"setup.py build" time is pain point. currently, we have a pretty fragile virtualenv caching strategy, and I've even considered patching distutils so that "setup.py build" would somehow work against a cache. currently, my mindset is to migrate to a modular rpm install strategy (managed by puppet) to prevent wasteful build time.

Marcus P.S. Daniel's "wheel" format is another possible to answer to my concern

To speed up deployments, we use pip's --download-cache option, so all builds on the box share the downloaded tarballs.

We also use our own local mirror, and an extra apache directory for packages that are not on pypi.

I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has.

Everything is driven from a Makefile. Example: https://github.com/mozilla-services/tokenserver/blob/master/Makefile The script to create RPMs is at https://github.com/mozilla-services/mopytools

HTH Tarek

On 7/3/12 9:44 AM, Donald Stufft wrote:

On Tuesday, July 3, 2012 at 3:42 AM, Tarek Ziadé wrote:

...

I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has.

I hate this particular misfeature of python packaging. http://restricted.crate.io/ gets rid of it.

Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error')

But the feature I need is to be able to do a full offline deployment without touching the net *but* my local index+directories, and by cutting all the crawling pip or easy+install do when following links.

That's something possible in packaging's crawler, and that should be a hard requirement imho for whatever tool is built

Cheers Tarek

On 7/3/12 9:51 AM, Donald Stufft wrote:

On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:

...

Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error')

Hmm that's odd, it works for me now and I don't see anything logged.

The 3rd hit worked.

All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)

That's all being refactored currently to work a lot nicer though on Crate's end.

But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?

Le mar. 03 juil. 2012 09:55:06 CEST, Tarek Ziadé a écrit :

On 7/3/12 9:51 AM, Donald Stufft wrote:

...

On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:

...

Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error')

Hmm that's odd, it works for me now and I don't see anything logged.

The 3rd hit worked.

...

All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)

That's all being refactored currently to work a lot nicer though on Crate's end.

But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?

the crawler follows the links present in the simple index before determining which distribution to download.

However, that's true that the metadata is buit from the information contained in the setup.py file, and reuses the url present in setup.py as the "url" metadata. In any case, that's *after* following the links in the index.

Le mar. 03 juil. 2012 10:08:07 CEST, Tarek Ziadé a écrit :

Yeah so even a 'cleaned up' mirror will not do the trick. For pip and easy_install, we need a lower-level barrier to avoid hitting the net

Well, if links are removed from the indexes, then the crawler will not try to follow the links, which sounds enough, isn't it? In which cases would an installer hit the external links if it doesn't have a way to discover them?

Additionally, distutils2 do have a "follow_links" options that can be set to False (I'm even wondering if thats not the default option)

Alexis

On Tuesday, July 3, 2012 at 4:15 AM, Tarek Ziadé wrote:

The scenario I had in mind was:

1. installer downloads Foo-1.2.tgz
2. installer runs Foo's setup.py
3. Foo's metadata mentions http://example.com
4. installer follow that link

But I did not check the code yet

The only time this would happen is if the metadata includes dependency links which isn't a very common bit of metadata.