Self Introduction and Getting Hands On
Hi all, I recently came upon Tarek's blogpost [1] about converting Python packages to rpm specfiles. I have pretty good knowledge of this, as I am a Fedora package maintainer and Python is one of my main responsibilities. I have always wanted to become a Python developer and becoming part of distutils-sig and trying to give a helping hand with RPM related stuff is probably a good entry point for me :) To introduce myself, I work at Red Hat as a package maintainer for dynamic languages, I am author of the new Fedora-used pyp2rpm [2], [3], which tries to do exactly what Py2RPM is supposed to be. (Well, it's not perfect, but it can do 90 % of the work, usually. So far, it doesn't work with distutils2, but I'll probably be working on that when Python 3.3 gets released.) I also have some minor not-yet-released projects in Python, I work with Django and I'm just a Python enthusiast. In Fedora, I am a maintainer/comaintainer of both Python and Python3, Django and another 15+ (and growing) set of packages. Thanks for reading this through :) I'm looking forward to working with - and learning from - all of you. -- Regards, Bohuslav "Slavek" Kabrda. [1] http://ziade.org/2011/03/25/bdist_rpm-is-dead-long-life-to-py2rpm/ [2] http://pypi.python.org/pypi/pyp2rpm [3] https://bitbucket.org/bkabrda/pyp2rpm
On 6/26/12 8:56 AM, Bohuslav Kabrda wrote:
Hi all, I recently came upon Tarek's blogpost [1] about converting Python packages to rpm specfiles. I have pretty good knowledge of this, as I am a Fedora package maintainer and Python is one of my main responsibilities. I have always wanted to become a Python developer and becoming part of distutils-sig and trying to give a helping hand with RPM related stuff is probably a good entry point for me :)
To introduce myself, I work at Red Hat as a package maintainer for dynamic languages, I am author of the new Fedora-used pyp2rpm [2], [3], which tries to do exactly what Py2RPM is supposed to be. (Well, it's not perfect, but it can do 90 % of the work, usually. So far, it doesn't work with distutils2, but I'll probably be working on that when Python 3.3 gets released.) I also have some minor not-yet-released projects in Python, I work with Django and I'm just a Python enthusiast. In Fedora, I am a maintainer/comaintainer of both Python and Python3, Django and another 15+ (and growing) set of packages.
Thanks for reading this through :) I'm looking forward to working with - and learning from - all of you.
Welcome ! As I stated in the blog post, I think a separate project for all RPM -related stuff would make a lot of sense FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI projects, we use at Mozilla to deploy our apps, http://pypi.python.org/pypi/pypi2rpm It's just a glue script on the top of an isolated bdist_rpm command and distutils2.version (for sorting versions) but you can also pass your spec file for a given project and have Fedora/RHEL specific options (like adding python26- prefixes etc) It sounds like your project is on the other side, e.g. generating spec files out of Python projects, so I'd be really interested to see how it may replace bdist_rpm in pypi2rpm Cheers Tarek
----- Original Message -----
On 6/26/12 8:56 AM, Bohuslav Kabrda wrote:
Hi all, I recently came upon Tarek's blogpost [1] about converting Python packages to rpm specfiles. I have pretty good knowledge of this, as I am a Fedora package maintainer and Python is one of my main responsibilities. I have always wanted to become a Python developer and becoming part of distutils-sig and trying to give a helping hand with RPM related stuff is probably a good entry point for me :)
To introduce myself, I work at Red Hat as a package maintainer for dynamic languages, I am author of the new Fedora-used pyp2rpm [2], [3], which tries to do exactly what Py2RPM is supposed to be. (Well, it's not perfect, but it can do 90 % of the work, usually. So far, it doesn't work with distutils2, but I'll probably be working on that when Python 3.3 gets released.) I also have some minor not-yet-released projects in Python, I work with Django and I'm just a Python enthusiast. In Fedora, I am a maintainer/comaintainer of both Python and Python3, Django and another 15+ (and growing) set of packages.
Thanks for reading this through :) I'm looking forward to working with - and learning from - all of you.
Welcome !
As I stated in the blog post, I think a separate project for all RPM -related stuff would make a lot of sense
I agree. RPM (and distro-packaging in general) things have different lifecycle, so it may be nice to be able to reflect that.
FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI projects, we use at Mozilla to deploy our apps, http://pypi.python.org/pypi/pypi2rpm It's just a glue script on the top of an isolated bdist_rpm command and distutils2.version (for sorting versions) but you can also pass your spec file for a given project and have Fedora/RHEL specific options (like adding python26- prefixes etc)
It sounds like your project is on the other side, e.g. generating spec files out of Python projects, so I'd be really interested to see how it may replace bdist_rpm in pypi2rpm
Exactly, my project is aimed at creating just the specfiles. In Fedora (and other distros as well), there is a review process that each package must go through and having a specfile written according to the current guidelines is a necessary part of that. "pypi2rpm" solves a different use case than distro-packagers need. I have a pretty good idea about how distro-packagers would like things to work and combining it with the "pypi2rpm", I see it this way: 1) Starting point: we have a Python package. 2) We want to: 2a) directly convert it to binary RPM => we use "pypi2rpm" - good for Mozilla to deploy apps, not useful for distro-packagers - Producing binary RPM probably could be left in distutils, but looking at the bigger picture, RPM has different lifecycle than python standard library, so it might be good to pull this functionality out. 2b) produce a specfile/whatever that is called in other distributions => use Fedora's "pyp2rpm"/is there something similar for other distributions? - This should definitely be present in a library outside of Python - we may want to provide templates/distro-specific stuff here and be able to react to new releases of different distros (when they change their way of packaging). So the best approach that currently comes to my mind is having a functionality in distutils, that can provide the package metadata (license, runtime/build requirements) and separate tool(s) that would be able to use these metadata to a) convert directly to binary packages (rpm, deb, etc.) or b) produce specfiles/whatever for the packagers' reviews Some questions that pop immediately are - Should a) and b) be one library or two? They handle packages in completely different way. - Should every distribution provide its very own library to handle a) and b) stuff or should these be a general libraries with distro-specific modules+templates? ... How does this sound? This is just a general idea that came to my mind some time ago - feel free to dump it, if you don't like it :) Have a nice day, Slavek.
Cheers Tarek _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Regards, Bohuslav "Slavek" Kabrda.
On 6/26/12 1:23 PM, Bohuslav Kabrda wrote:
[...] Exactly, my project is aimed at creating just the specfiles. In Fedora (and other distros as well), there is a review process that each package must go through and having a specfile written according to the current guidelines is a necessary part of that. "pypi2rpm" solves a different use case than distro-packagers need.
I have a pretty good idea about how distro-packagers would like things to work and combining it with the "pypi2rpm", I see it this way: 1) Starting point: we have a Python package. 2) We want to: 2a) directly convert it to binary RPM => we use "pypi2rpm" - good for Mozilla to deploy apps, not useful for distro-packagers - Producing binary RPM probably could be left in distutils, but looking at the bigger picture, RPM has different lifecycle than python standard library, so it might be good to pull this functionality out. Yeah -- I had no intent to keep bdist_rpm in the stdlib. We removed it from distiutils2
2b) produce a specfile/whatever that is called in other distributions => use Fedora's "pyp2rpm"/is there something similar for other distributions? You mean for RPM that are not Fedora/RHEL ? I have no idea. - This should definitely be present in a library outside of Python - we may want to provide templates/distro-specific stuff here and be able to react to new releases of different distros (when they change their way of packaging).
So the best approach that currently comes to my mind is having a functionality in distutils, that can provide the package metadata (license, runtime/build requirements) you can already have this, with the current Distutils code.
and separate tool(s) that would be able to use these metadata to a) convert directly to binary packages (rpm, deb, etc.) or b) produce specfiles/whatever for the packagers' reviews
Some questions that pop immediately are - Should a) and b) be one library or two? They handle packages in completely different way.] In a different project I think. For debian there's 'stdeb' - http://pypi.python.org/pypi/stdeb
- Should every distribution provide its very own library to handle a) and b) stuff or should these be a general libraries with distro-specific modules+templates? ...
As a user I want to pick up *any* project, and build a deb or a rpm, even it the author did not provide the deb/rpm integration, even if I eventually need to tweak the process.
How does this sound? This is just a general idea that came to my mind some time ago - feel free to dump it, if you don't like it :)
So what are your plans ?
Have a nice day, Slavek.
Cheers Tarek _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----- Original Message -----
On 6/26/12 1:23 PM, Bohuslav Kabrda wrote:
[...] Exactly, my project is aimed at creating just the specfiles. In Fedora (and other distros as well), there is a review process that each package must go through and having a specfile written according to the current guidelines is a necessary part of that. "pypi2rpm" solves a different use case than distro-packagers need.
I have a pretty good idea about how distro-packagers would like things to work and combining it with the "pypi2rpm", I see it this way: 1) Starting point: we have a Python package. 2) We want to: 2a) directly convert it to binary RPM => we use "pypi2rpm" - good for Mozilla to deploy apps, not useful for distro-packagers - Producing binary RPM probably could be left in distutils, but looking at the bigger picture, RPM has different lifecycle than python standard library, so it might be good to pull this functionality out. Yeah -- I had no intent to keep bdist_rpm in the stdlib. We removed it from distiutils2
Ah, I see. I didn't have that much time to get familiar with distutils2 yet - I only know them from the documentation :)
2b) produce a specfile/whatever that is called in other distributions => use Fedora's "pyp2rpm"/is there something similar for other distributions? You mean for RPM that are not Fedora/RHEL ? I have no idea. - This should definitely be present in a library outside of Python - we may want to provide templates/distro-specific stuff here and be able to react to new releases of different distros (when they change their way of packaging).
So the best approach that currently comes to my mind is having a functionality in distutils, that can provide the package metadata (license, runtime/build requirements) you can already have this, with the current Distutils code.
True. What I meant by this is, that providing the metadata is the thing that distutils should do. In my opinion, they shouldn't do anything more (no distro-specific stuff).
and separate tool(s) that would be able to use these metadata to a) convert directly to binary packages (rpm, deb, etc.) or b) produce specfiles/whatever for the packagers' reviews
Some questions that pop immediately are - Should a) and b) be one library or two? They handle packages in completely different way.] In a different project I think. For debian there's 'stdeb' - http://pypi.python.org/pypi/stdeb
- Should every distribution provide its very own library to handle a) and b) stuff or should these be a general libraries with distro-specific modules+templates? ...
As a user I want to pick up *any* project, and build a deb or a rpm, even it the author did not provide the deb/rpm integration, even if I eventually need to tweak the process.
What I meant here is - should there be one big tool producing the specfiles/debian stuff/... or one project for RPM specfiles, one project for debian stuff...
How does this sound? This is just a general idea that came to my mind some time ago - feel free to dump it, if you don't like it :)
So what are your plans ?
Since I'm new here, I first want to watch and listen and contribute to the discussions. I would certainly like to discuss the distro-specific stuff in more detail and participate on a solution. That means first create a clear vision with more distutils people here and then actually take my part implementing it. It would certainly be nice to create something flexible that could do both the specfiles and binary packages for more distributions - for me, that's the goal here (and of course, Python rocks).
Have a nice day, Slavek.
Cheers Tarek _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Regards, Bohuslav "Slavek" Kabrda.
FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI projects, we use at Mozilla to deploy our apps, http://pypi.python.org/pypi/**pypi2rpmhttp://pypi.python.org/pypi/pypi2rpm It's just a glue script on the top of an isolated bdist_rpm command and distutils2.version (for sorting versions) but you can also pass your spec file for a given project and have Fedora/RHEL specific options (like adding python26- prefixes etc)
Hello Tarek: if you have anything that details your rpm deployment workflow at mozilla, i'd love to see it. I work in a very active build environment, and repetitive"setup.py build" time is pain point. currently, we have a pretty fragile virtualenv caching strategy, and I've even considered patching distutils so that "setup.py build" would somehow work against a cache. currently, my mindset is to migrate to a modular rpm install strategy (managed by puppet) to prevent wasteful build time. Marcus P.S. Daniel's "wheel" format is another possible to answer to my concern
On 7/1/12 8:50 PM, Marcus wrote:
FWIW I have started a "pypi2rpm" project that creates RPM out of PyPI projects, we use at Mozilla to deploy our apps, http://pypi.python.org/pypi/pypi2rpm It's just a glue script on the top of an isolated bdist_rpm command and distutils2.version (for sorting versions) but you can also pass your spec file for a given project and have Fedora/RHEL specific options (like adding python26- prefixes etc)
Hello Tarek: if you have anything that details your rpm deployment workflow at mozilla, i'd love to see it. I work in a very active build environment, and repetitive"setup.py build" time is pain point. currently, we have a pretty fragile virtualenv caching strategy, and I've even considered patching distutils so that "setup.py build" would somehow work against a cache. currently, my mindset is to migrate to a modular rpm install strategy (managed by puppet) to prevent wasteful build time.
Marcus P.S. Daniel's "wheel" format is another possible to answer to my concern
To speed up deployments, we use pip's --download-cache option, so all builds on the box share the downloaded tarballs. We also use our own local mirror, and an extra apache directory for packages that are not on pypi. I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has. Everything is driven from a Makefile. Example: https://github.com/mozilla-services/tokenserver/blob/master/Makefile The script to create RPMs is at https://github.com/mozilla-services/mopytools HTH Tarek
On Tuesday, July 3, 2012 at 3:42 AM, Tarek Ziadé wrote:
I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has. I hate this particular misfeature of python packaging. http://restricted.crate.io/ gets rid of it.
On 7/3/12 9:44 AM, Donald Stufft wrote:
On Tuesday, July 3, 2012 at 3:42 AM, Tarek Ziadé wrote:
I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has. I hate this particular misfeature of python packaging. http://restricted.crate.io/ gets rid of it.
Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error') But the feature I need is to be able to do a full offline deployment without touching the net *but* my local index+directories, and by cutting all the crawling pip or easy+install do when following links. That's something possible in packaging's crawler, and that should be a hard requirement imho for whatever tool is built Cheers Tarek
On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:
Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error') Hmm that's odd, it works for me now and I don't see anything logged. All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)
That's all being refactored currently to work a lot nicer though on Crate's end.
On 7/3/12 9:51 AM, Donald Stufft wrote:
On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:
Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error') Hmm that's odd, it works for me now and I don't see anything logged.
The 3rd hit worked.
All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)
That's all being refactored currently to work a lot nicer though on Crate's end.
But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?
But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?
On Tuesday, July 3, 2012 at 3:55 AM, Tarek Ziadé wrote: pip at least, and I assume easy_install doesn't use the home url etc links from inside of the setup.py. PyPI takes that data and places it on the simple index page and pip looks at that simple index page and finds all the urls it can to spider. So removing urls from the simple index page effectively removes external linking. The only time that's not true is if there a dependency_links inside of setup.py but that is rarely used.
Le mar. 03 juil. 2012 09:55:06 CEST, Tarek Ziadé a écrit :
On 7/3/12 9:51 AM, Donald Stufft wrote:
On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:
Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error') Hmm that's odd, it works for me now and I don't see anything logged.
The 3rd hit worked.
All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)
That's all being refactored currently to work a lot nicer though on Crate's end.
But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?
the crawler follows the links present in the simple index before determining which distribution to download. However, that's true that the metadata is buit from the information contained in the setup.py file, and reuses the url present in setup.py as the "url" metadata. In any case, that's *after* following the links in the index.
On 7/3/12 10:05 AM, Alexis Métaireau wrote:
Le mar. 03 juil. 2012 09:55:06 CEST, Tarek Ziadé a écrit :
On 7/3/12 9:51 AM, Donald Stufft wrote:
On Tuesday, July 3, 2012 at 3:48 AM, Tarek Ziadé wrote:
Not sure how http://restricted.crate.io/ qorks (unreachable for me right now - 'internal server error') Hmm that's odd, it works for me now and I don't see anything logged.
The 3rd hit worked.
All it does is it doesn't list any external links on the simple pages so pip, easy_install etc never find any links to spider ;)
That's all being refactored currently to work a lot nicer though on Crate's end.
But IIRC when the package is downloaded, the metadata are built in-place, and the installer may visit the home url for example that's in setup.py, no ?
the crawler follows the links present in the simple index before determining which distribution to download.
However, that's true that the metadata is buit from the information contained in the setup.py file, and reuses the url present in setup.py as the "url" metadata. In any case, that's *after* following the links in the index.
Yeah so even a 'cleaned up' mirror will not do the trick. For pip and easy_install, we need a lower-level barrier to avoid hitting the net
Le mar. 03 juil. 2012 10:08:07 CEST, Tarek Ziadé a écrit :
Yeah so even a 'cleaned up' mirror will not do the trick. For pip and easy_install, we need a lower-level barrier to avoid hitting the net
Well, if links are removed from the indexes, then the crawler will not try to follow the links, which sounds enough, isn't it? In which cases would an installer hit the external links if it doesn't have a way to discover them? Additionally, distutils2 do have a "follow_links" options that can be set to False (I'm even wondering if thats not the default option) Alexis
On 7/3/12 10:12 AM, Alexis Métaireau wrote:
Le mar. 03 juil. 2012 10:08:07 CEST, Tarek Ziadé a écrit :
Yeah so even a 'cleaned up' mirror will not do the trick. For pip and easy_install, we need a lower-level barrier to avoid hitting the net
Well, if links are removed from the indexes, then the crawler will not try to follow the links, which sounds enough, isn't it? In which cases would an installer hit the external links if it doesn't have a way to discover them?
The scenario I had in mind was: 1. installer downloads Foo-1.2.tgz 2. installer runs Foo's setup.py 3. Foo's metadata mentions http://example.com 4. installer follow that link But I did not check the code yet
Additionally, distutils2 do have a "follow_links" options that can be set to False (I'm even wondering if thats not the default option)
Alexis
The scenario I had in mind was:
1. installer downloads Foo-1.2.tgz 2. installer runs Foo's setup.py 3. Foo's metadata mentions http://example.com 4. installer follow that link
But I did not check the code yet The only time this would happen is if the metadata includes dependency
On Tuesday, July 3, 2012 at 4:15 AM, Tarek Ziadé wrote: links which isn't a very common bit of metadata.
To speed up deployments, we use pip's --download-cache option, so all builds on the box share the downloaded tarballs.
We also use our own local mirror, and an extra apache directory for packages that are not on pypi.
I have also added a proxy that returns a 404 when pip tries to hit the net, to simulate the --allow-hosts option easy_install has.
Everything is driven from a Makefile. Example: https://github.com/mozilla-services/tokenserver/blob/master/Makefile The script to create RPMs is at https://github.com/mozilla-services/mopytools
thanks for the link to the makefile. we also use a download cache and a local mirror (although we're migrating to findlink dirs instead) for us, that still leaves the issue of wasteful "setup.py build/install" time (for many dependencies) in our repetitive test/build environment. that's my interest in modular binary build solutions like rpm. the idea would be to stop rebuilding a large virtualenv for every test build. but rather just rebuild a dev rpm the specific package(s) that's changing and a new dev master rpm that references those changes. *then* go install the new master rpm on the test box, which will find most of it's rpm dependencies already satisfied. but I'll admit I'm new to thinking about binary build solutions. thanks Marcus
HTH Tarek
participants (5)
-
Alexis Métaireau -
Bohuslav Kabrda -
Donald Stufft -
Marcus -
Tarek Ziadé