Autobuild packages using snakebite
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this? The biggest problem I see is security, but if people are really interested in this we could at least try it no? Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service. -- Leonardo Santagada santagada at gmail.com
I'm currently working on an automated build system with similar goals: http://pyvm.sourceforge.net Regards, Antonio
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
-- Leonardo Santagada santagada at gmail.com
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Jun 18, 2009, at 12:18 PM, A. Cavallo wrote:
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
I'm currently working on an automated build system with similar goals:
Regards, Antonio
What I proposed was a autobuild for pytho packages on pypi not cpython builds. The system could certanly use your tool as a way to have many python versions, but in the end I think people would want packages built by cpython binaries distributed at python.org and third party distros (like ubuntu, mac os x, macports, etc). -- Leonardo Santagada santagada at gmail.com
On Thu, Jun 18, 2009 at 5:02 PM, Leonardo Santagada <santagada@gmail.com>wrote:
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
I mentioned that need a while ago and work on a project see ( http://tarekziade.wordpress.com/2009/05/28/smoke-testing-in-distutils), and I made a buildbot doing smoke tests : I have it running for a few packages here http://buildbot.ziade.org/waterfall on one slave so far. I helps me on making sure I don't break distutils :) I know Titus wants to do something similar as well, and I think Antonio project is adressing another need.
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
I have a white list of packages so far. A complete system needs to run in a virtual environment and scratch it everytime a package is tested + cut the network access.
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
I am calling "python setup.py sdist" and "python setup.py bdist" so far and compare the output on various python versions, but any command would work. If you are interested in writing some scripts, you are welcome to join, it's an open project located here http://bitbucket.org/tarek/distutils-builbot/src/ Tarek -- Tarek Ziadé | http://ziade.org
Ok, I've been quiet for a little while fixing few things on the pyvm project... so sorry for this following very long email. Pyvm is my (personal at the moment) attempt to use what opensuse team has come up with: the opensuse build server. This service (free) provides a continuous integration system as many commercial projects are actively using: having some first hand experience I can tell you is quite an effective method. The basics are very simple: 1. the user/developer puts some files in a repository (actually a "svn" repository) 2. The scheduler builds a xen system image in a well known status (so guaranteed repeatable) 3. The scheduler then runs the build using the provided files including the smoke tests, collects the resulting rpms and publish them on line The advantage on buildbot are many fold: 1. There is a guarantee the resulting binaries are untampered (using the standard libraries provided by the linux distributions not the one the developer has installed on his box) 2. Hands off build, so no funny hacks that work on some machines/distributions but not on other (yes, it enforces a development discipline) 3. It supports a variety of linux distribution in parallel (so far I've addressed the lsb standard so using the rpm) 4. It make possible for an administrator to have many python interpreter versions side by side (believe or not there're companies still using python 2.3 because is the one shipped with red hat enterprise linux 4!) 5. The system images are network disconnected While build bot is fine for personal use, opensuse is a step ahead providing build that can be used by the users and, by the way, this system is the one that builds opensuse. Osc (the susebuild client) can provide also a local build in the same spirit of buildbot: the only missing bit (at the moment) is the reporting part. Now the pyvm project. -------------------------------- There are really two different subprojects: the intepreters and the modules. Now the interpreters are taken from the svn trunk and rebuild for a pletora of linux distributions and architectures. There's a minimal smoke test and few patches applied. A user can point its yum/yast package manager and download the python intepreter for each subversion checkout (or doing it by downloading the rpms). Then there is the more interesting part for developers: the modules. Basically they are few python packages build against each python subversion check out so the building chain is: opt-python-svn73450 -> setuptools-latest-svn73450 -> sphinx-0.6.1-svn73450 If something is broken everything will fail: this process is automatic(-ish) so I can manage a failry large set of package already. My use case for this is: A module developer creates his masterpiece then, if it is worth, gets an account on opensuse. He sends the tarball, a spec file and the eventual patch files and waits for the build to happen. If fails he has to repeat it again: the first time will be painful, but with time it will improve and the overall quality will be much better. There are many more use cases pyvm address, but it will bore people too much. I hope this clarifies a little bit what is my aim with pyvm. I wish this approach will be adopted because I can see clearly the benefits of it in terms of quality control (I'm not promoting here pyvm but the service opensuse is providing for free). Regards, Antonio
On Thu, Jun 18, 2009 at 5:02 PM, Leonardo Santagada <santagada@gmail.com>wrote:
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
I mentioned that need a while ago and work on a project see ( http://tarekziade.wordpress.com/2009/05/28/smoke-testing-in-distutils), and I made a buildbot doing smoke tests : I have it running for a few packages here http://buildbot.ziade.org/waterfall on one slave so far.
I helps me on making sure I don't break distutils :)
I know Titus wants to do something similar as well, and I think Antonio project is adressing another need.
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
I have a white list of packages so far. A complete system needs to run in a virtual environment and scratch it everytime a package is tested + cut the network access.
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
I am calling "python setup.py sdist" and "python setup.py bdist" so far and compare the output on various python versions, but any command would work. If you are interested in writing some scripts, you are welcome to join, it's an open project located here http://bitbucket.org/tarek/distutils-builbot/src/
Tarek
On Thu, 18 Jun 2009 12:02:24 -0300, Leonardo Santagada <santagada@gmail.com> wrote:
I remember when snakebite was announced someone had the idea to have a kind of build bot go into pypi and get all packages sent to it and automatically compile them on snakebite for all major platforms and automatically run tests and submit the compiled packages to pypi. Is anyone working on this?
I think I suggested that.. I followed it up with Snakebite. They said their facilities weren't operational and they suggested there might be a cost to doing it.. They gave me the impression that they wanted to control the coding...
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
It's not that. It's that the whole thing starts to get really huge. Really fast... And in my mind, it starts to look like a commercial service at some point. Because perphaps developers might find it worthwhile to donate $20 to test their apps on multiple platforms. I say this because not many mac users are happy to lend or run scripts from "those other people for free". And fair enough too because those machines are not cheap. Then you have the "free" side... yes.. I still believe that someone somewhere should be running tests on pypi on the packages. But that shouldn't cost anybody else. It should just be built in to the pypi facilities. Firebird, an open source dbms (http://www.firebirdsql.org/) do their open source project very well via a non-profit foundation. They just take donations and then direct the funds to paying for whatever they want done. Last but not least is the programming... which isn't such a big deal. But it takes time, effort and dedication. Even though I regard it as easy* it isn't trivial... :-)
Automatic testing in many python versions, conversion using 2to3 and building packages for windows and macosx seems like a very interesting service.
yep... sure is... David
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there. Stefan
Stefan Behnel wrote:
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there.
Since it would be inside a VM, the major risk would be running some kind of malicious server of some kind inside the setup script - but it should be relatively easy to make sure the vm prevents that from happening ? It is a major issue, but I would guess it has been solved by the build service system (which is really great BTW, I think it is a very significant advancement, under-rated project for open source software deployment). cheers, David
On Fri, 19 Jun 2009 07:24:21 +0200, Stefan Behnel <stefan_ml@behnel.de> wrote:
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there.
Doesn't a chroot jail stop this? (on unix anyway)
Hi, this is really getting off-topic, BTW. David Lyon wrote:
On Fri, 19 Jun 2009 07:24:21 +0200, Stefan Behnel wrote:
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there.
Doesn't a chroot jail stop this? (on unix anyway)
Stop it from running arbitrary code? How would it do that? Imagine your setup.py downloads Seti@Home jobs from a web site and calculates them. You can't really ban web access, as the setup.py script may use setuptools to download dependencies. You can't really stop it from doing calculations, as it might actually be doing real build stuff like source code generation. You can kill it after, say, an hour (or a half if you think that no build should take longer than that), but I guess I can do a lot of calculations in that time, on a lot of your computers. And if you ban my package, I'll just upload a different one with a different name. And if you ban my PyPI account, I'll grab a new one. And if you ban web access from your machines, well, I'll just upload a package that contains the tasks in the archive. And if you use hash codes to check for malicious packages, well, I'll just start obfuscating my code and my data. And be assured that what I'm really doing is not calculating Seti@Home jobs. There's really no way you can stop someone from misusing your infrastructure if you go for building any arbitrary package that gets uploaded to PyPI. Stefan
On Jun 19, 2009, at 11:28 AM, Stefan Behnel wrote:
Hi,
this is really getting off-topic, BTW.
David Lyon wrote:
On Fri, 19 Jun 2009 07:24:21 +0200, Stefan Behnel wrote:
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there.
Doesn't a chroot jail stop this? (on unix anyway)
Stop it from running arbitrary code? How would it do that?
Imagine your setup.py downloads Seti@Home jobs from a web site and calculates them. You can't really ban web access, as the setup.py script may use setuptools to download dependencies. You can't really stop it from doing calculations, as it might actually be doing real build stuff like source code generation. You can kill it after, say, an hour (or a half if you think that no build should take longer than that), but I guess I can do a lot of calculations in that time, on a lot of your computers. And if you ban my package, I'll just upload a different one with a different name. And if you ban my PyPI account, I'll grab a new one. And if you ban web access from your machines, well, I'll just upload a package that contains the tasks in the archive. And if you use hash codes to check for malicious packages, well, I'll just start obfuscating my code and my data. And be assured that what I'm really doing is not calculating Seti@Home jobs.
There's really no way you can stop someone from misusing your infrastructure if you go for building any arbitrary package that gets uploaded to PyPI.
I think you are completely right... so lets not do that. We should whitelist some pgp signatures from people we trust (and some of those people should whitelist other people) and then we can have simpler virtualization methods that only garantee isolation from one another. If we caught someone cheating we remove their signature and all the signatures from everyone this person permitted into the autobuild farm... and maybe we should ask the person that gave permission to this guy. Or something on these lines. -- Leonardo Santagada santagada at gmail.com
Stefan Behnel wrote:
Hi,
this is really getting off-topic, BTW.
David Lyon wrote:
On Fri, 19 Jun 2009 07:24:21 +0200, Stefan Behnel wrote:
Leonardo Santagada wrote:
The biggest problem I see is security, but if people are really interested in this we could at least try it no?
Security certainly is a major issue here. Anyone can upload packages to PyPI, so you can run arbitrary code on tons of machines, just by pushing some well-forged setup.py script there.
Doesn't a chroot jail stop this? (on unix anyway)
Stop it from running arbitrary code? How would it do that?
Imagine your setup.py downloads Seti@Home jobs from a web site and calculates them. You can't really ban web access, as the setup.py script may use setuptools to download dependencies. You can't really stop it from doing calculations, as it might actually be doing real build stuff like source code generation. You can kill it after, say, an hour (or a half if you think that no build should take longer than that), but I guess I can do a lot of calculations in that time, on a lot of your computers. And if you ban my package, I'll just upload a different one with a different name. And if you ban my PyPI account, I'll grab a new one. And if you ban web access from your machines, well, I'll just upload a package that contains the tasks in the archive. And if you use hash codes to check for malicious packages, well, I'll just start obfuscating my code and my data. And be assured that what I'm really doing is not calculating Seti@Home jobs.
There's really no way you can stop someone from misusing your infrastructure if you go for building any arbitrary package that gets uploaded to PyPI.
Forbidding any network access from the vm used to build would solve most of those problems. Controlling CPU/memory can be done from the host OS. I would be surprised if the openSuse build system worked in a fundamentally different way: rpm .spec files can also execute arbitrary code. cheers, David
On Sat, 20 Jun 2009 19:17:12 +0900, David Cournapeau
There's really no way you can stop someone from misusing your infrastructure if you go for building any arbitrary package that gets uploaded to PyPI.
But CPU cycles can be counted and likewise network usage...
Forbidding any network access from the vm used to build would solve most of those problems. Controlling CPU/memory can be done from the host OS. I would be surprised if the openSuse build system worked in a fundamentally different way: rpm .spec files can also execute arbitrary code.
I struggle with seeing how that would work. My easyinstall is always downloading additional packages (as it should). And how would you get packages onto the infastructure from pypi in the first place if there was no internet access? David
David Lyon wrote:
I struggle with seeing how that would work. My easyinstall is always downloading additional packages (as it should).
the answer is obvious: it should not download anything from the wild. That's one of the big reason why so many people literally hate this setuptools feature, BTW.
And how would you get packages onto the infastructure from pypi in the first place if there was no internet access?
that's the job of the infrastructure. All those problems have been solved by the opensuse build service, which is open source (in Perl for the server part, though). Before stating what is possible and not, it would be wise to study other similar services: http://en.opensuse.org/Build_Service cheers, David
On Sat, 20 Jun 2009 21:24:39 +0900, David Cournapeau
that's the job of the infrastructure. All those problems have been solved by the opensuse build service, which is open source (in Perl for the server part, though). Before stating what is possible and not, it would be wise to study other similar services:
Thanks David. I will look into it. Looks very fascinating at first glance... :-)
participants (6)
-
A. Cavallo -
David Cournapeau -
David Lyon -
Leonardo Santagada -
Stefan Behnel -
Tarek Ziadé