The problem with Setuptools on Python 3.
Executive summary: I have catch 22 problems in finishing the setuptools port to Python 3, and unless somebody can come up with alternatives, I may rip out setuptools dependency on itself completely, and only depend on distutils. I have as you know ported setuptools to Python 3. I'm in the process of trying to fix and clean up that porting so that it can be merged into trunk, but there is one obstacle left, and that's the question of installing. Currently there are several ways to install setuptools: 1) Run ez_install.py 2) Download / check-out source and run python setup.py install. 3) Download the correct egg and install that (which always confuses me, I usually fail, and end up doing 2) instead). Setuptools also helps with running tests by 4) python setup.py test Currently for the Python 3 version, I've made a script that copies the source tree and runs 2to3 on it. Then you can use that new tree to run tests, install, make eggs and releases etc. The problems with this is: a) The script is made for my computer only. It wouldn't work on Windows, and it requires you to have lib2to3 checked out in a particular place (although I think that's only if you use 3.0.0, and not 3.0.1, but I haven't tested). b) It's slow, as it copies all files, even those who hasn't changed when you fixed a bug. Making a script that only copies the changed files and runs 2to3 on them is possible, but I'd like not too, it takes time and is likely to break. It seems like a waste. Obviously, this is exactly the things distutils are meant to solve. And indeed, distutils in Python 3 has a command called build_2to3 which solves exactly these problems. And indeed, this is the technique I've used for the Python 3 branch of zope.interfaces. There I just run setup.py install, and if I'm doing that under Python 3, it will first run 2to3 on the code before installing. Same thing with running setup.py test. It will sync changes to build/, run 2to3 on changed files and then run the tests in build. It makes porting to Python 3 much easier, and it makes installing from source painless. So why don't I use that for setuptools? Well, because: c) The setup of setuptools requires setuptools. So to be able to do the 2to3 conversion in the setup, I need to first convert the source with 2to3. Yes, catch 22. I've tried to get around this problem, but failed. Solutions I tried was: I) Fall back to distutils if setuptools can't be imported. This means that I can with some effort make a setup that will work under Python 3 and install setuptools under Python 3. Problem solved? No. Because running setup.py again will just mean that it tries to import setuptools from the local Python2 location, and it will fail. So in this case I can't run tests of build eggs for setuptool. II) Moving the source into a src directory. This means that when setup.py runs, it will use the version of setuptools installed under I), and building eggs etc is possible (I think, I haven't tested that the eggs are correct). Problem solved? No, because you still can't test it. Which leads us to problem d: d) When running the tests, the setuptools module will already be imported. But it will have imported the one in site-packages, *not* the one in src, and it will therefore test the one in site-packages. And that one doesn't have the api_tests.txt copied in, so that will fail, and e) even if api_tests.txt was copied, this would mean you had to install setuptools before you test it, which is daft. I don't have a good solution to this, unless we can drop setuptools dependency on setuptools completely, and just use plain distutils for installing and testing setuptools. This will mean no more setuptools source eggs, but then I don't see the purpose of those, really. Arguments for not doing this and alternative solutions are highly welcome. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
2009/4/20 Lennart Regebro <regebro@gmail.com>:
I don't have a good solution to this, unless we can drop setuptools dependency on setuptools completely, and just use plain distutils for installing and testing setuptools. This will mean no more setuptools source eggs, but then I don't see the purpose of those, really. Arguments for not doing this and alternative solutions are highly welcome.
Personally, this type of circular dependency seems totally wrong. I'd argue that making setuptools not depend on itself is the right thing to do, unless someone can provide a compelling reason why it is necessary (and such a reason would hopefully then provide more concrete avenues for looking for other possible alternative approaches). Paul.
Paul Moore <p.f.moore@gmail.com> writes:
2009/4/20 Lennart Regebro <regebro@gmail.com>:
I don't have a good solution to this, unless we can drop setuptools dependency on setuptools completely, and just use plain distutils for installing and testing setuptools.
Personally, this type of circular dependency seems totally wrong. I'd argue that making setuptools not depend on itself is the right thing to do, unless someone can provide a compelling reason why it is necessary (and such a reason would hopefully then provide more concrete avenues for looking for other possible alternative approaches).
Yes, that seems crazy-making to me and I wasn't aware it was the current situation. +1 for building setuptools on a base of distutils only, especially if it's already been achieved. -- \ “A man's only as old as the woman he feels.” —Groucho Marx | `\ | _o__) | Ben Finney
On Mon, Apr 20, 2009 at 11:44, Ben Finney <ben+python@benfinney.id.au> wrote:
+1 for building setuptools on a base of distutils only, especially if it's already been achieved.
No, we are going to have to make special custom extensions, at least for running tests. It's not that much work, but it is code duplication. I don't have the full overview of what features we lose either. But it does seem to me that most of these features, like the source eggs, cause more trouble than they solve. We'll probably need a separate Python 3 version of easy_install and the virtual-python scripts as well, but I already did those. But I think getting easy_install to download the source tgz and run setup.py install on it are going to simplify the code and remove the need for the eggs. Setuptools seems to have a lot of clever code to fix non-problems. This may very well be because I don't know what the real problems are, and these problems are not documented. I guess we need Phillip's input on that. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
2009/4/20 Lennart Regebro <regebro@gmail.com>:
On Mon, Apr 20, 2009 at 11:44, Ben Finney <ben+python@benfinney.id.au> wrote:
+1 for building setuptools on a base of distutils only, especially if it's already been achieved.
No, we are going to have to make special custom extensions, at least for running tests. It's not that much work, but it is code duplication. I don't have the full overview of what features we lose either. But it does seem to me that most of these features, like the source eggs, cause more trouble than they solve.
Are you saying that you need to use setuptools (or at least the feaures of setuptools) to develop setuptools? That's crazy. To run the setuptools tests, just run the test.py (or whatever) script. The setuptools ability to type python setup.py test, while convenient, simply isn't available while you're developing setuptools. The same logic applies to *any* setuptools feature that is used in the development of setuptools itself. Trying to make it available adds lots of complexity for the benefit of very few people (ie, people writing the setuptools code). Bootstrapping like this should be reserved for people writing C compilers in C, and other equally major-league projects. Just my view, and I'm not a setuptools fan, so you can probably disregard it. (But this type of complexity for its own sake is the reason I dislike the whole setuptools philosophy, so maybe it's worth making explicit...) Paul.
On Mon, Apr 20, 2009 at 14:21, Paul Moore <p.f.moore@gmail.com> wrote:
Are you saying that you need to use setuptools (or at least the feaures of setuptools) to develop setuptools?
Currently, yes. The testsuite is run with the testrunner of setuptools. You can run them separately too, I'm sure, but that's a pain. There is no test.py to run. Maybe you can use nose or something to run the tests, but then again you would need to port *nose* to Python 3... :-/
That's crazy. To run the setuptools tests, just run the test.py (or whatever) script.
There isn't any. And to do that you need to convert it to Python 3 first, and we are back to the "run a porting script, then run a test command", which is what we want to avoid in the first place. distutils have code exactly to help this kind of situation, so I think we should depend on that.
setuptools ability to type python setup.py test, while convenient, simply isn't available while you're developing setuptools.
Sure it is. Maybe it *shouldn't*. But it *is*. Except under Python 3, of course, which is the problem.
logic applies to *any* setuptools feature that is used in the development of setuptools itself. Trying to make it available adds lots of complexity for the benefit of very few people (ie, people writing the setuptools code).
It's not so much "trying to". Since the structure of the package is such that setuptools end up directly on the path, it *is* available. So the question is if we should make sure it *isn't* available, by moving it all down into a src directory, and then modify the setup-script so that it no longer depends on setuptools.
Bootstrapping like this should be reserved for people writing C compilers in C, and other equally major-league projects.
Are you saying setuptools isn't major-league? ;-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Paul Moore wrote:
Are you saying that you need to use setuptools (or at least the feaures of setuptools) to develop setuptools? That's crazy.To run the setuptools tests, just run the test.py (or whatever) script. The setuptools ability to type python setup.py test, while convenient, simply isn't available while you're developing setuptools. The same logic applies to *any* setuptools feature that is used in the development of setuptools itself. Trying to make it available adds lots of complexity for the benefit of very few people (ie, people writing the setuptools code).
Bootstrapping like this should be reserved for people writing C compilers in C, and other equally major-league projects.
I don't know the details of setuptools, but it is generally quite tempting to develop a new build/distribution tool using the new build tool, with some bootstrap process. That's how most build tools I know work, actually. FWIW, we use this as well in numpy for numpy.distutils extensions. And as much as you can count me in the "not a setuptools fan camp", I think it is easy to say setuptools code is bad - that's the natural reaction, really, and I would be surprised if P.J.E would not agree. But I also think anyone who had to deal with distutils extensions will tell you the same story - that's inherent to how distutils was conceived (10 years ago) and implemented. The distutils codebase is pretty horrible - I find m4 macro and 100000 lines of shell code in autoconf easier to deal with, really. You can deal with it, but it will certainly never be pretty. David
On Mon, Apr 20, 2009 at 3:23 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Paul Moore wrote:
Are you saying that you need to use setuptools (or at least the feaures of setuptools) to develop setuptools? That's crazy.To run the setuptools tests, just run the test.py (or whatever) script. The setuptools ability to type python setup.py test, while convenient, simply isn't available while you're developing setuptools. The same logic applies to *any* setuptools feature that is used in the development of setuptools itself. Trying to make it available adds lots of complexity for the benefit of very few people (ie, people writing the setuptools code).
Bootstrapping like this should be reserved for people writing C compilers in C, and other equally major-league projects.
I don't know the details of setuptools, but it is generally quite tempting to develop a new build/distribution tool using the new build tool, with some bootstrap process. That's how most build tools I know work, actually. FWIW, we use this as well in numpy for numpy.distutils extensions.
And as much as you can count me in the "not a setuptools fan camp", I think it is easy to say setuptools code is bad - that's the natural reaction, really, and I would be surprised if P.J.E would not agree. But I also think anyone who had to deal with distutils extensions will tell you the same story - that's inherent to how distutils was conceived (10 years ago) and implemented. The distutils codebase is pretty horrible - I find m4 macro and 100000 lines of shell code in autoconf easier to deal with, really. You can deal with it, but it will certainly never be pretty.
this can change. I am working on it. I need feedback though. let me know how my extension code proposal fits your needs. http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
David _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
Hi Tarek, Tarek Ziadé wrote:
this can change. I am working on it. I need feedback though.
let me know how my extension code proposal fits your needs.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
I will fill in the examples (this pages does not concern manifest only, then ?) from my own experience. IMHO, adding plugins systems will not change the fundamental deficiencies of distutils, though. You may be able to mitigate some problems, but I don't think you will be able to solve the fundamental issues, because they are fundamental design issues. The only way to fix them would induce serious breakage, and in that case, given the quality of distutils code, starting from scratch would be easier - I have yet encountered a single major distutils abstraction which was not poorly conceived and implemented (my own experience is related to build issues, for which numpy has requirements which go much beyond the usual python package, though). Commands classes, compiler classes, distributions, all this is very wrong for someone who wants more than compiling a couple of C files and tight control. David
On Mon, Apr 20, 2009 at 4:00 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Hi Tarek,
Tarek Ziadé wrote:
this can change. I am working on it. I need feedback though.
let me know how my extension code proposal fits your needs.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
I will fill in the examples (this pages does not concern manifest only, then ?) from my own experience.
no it doesn't. it's a generic tool to extend a command using plugins .
IMHO, adding plugins systems will not change the fundamental deficiencies of distutils, though. You may be able to mitigate some problems, but I don't think you will be able to solve the fundamental issues, because they are fundamental design issues. The only way to fix them would induce serious breakage, and in that case, given the quality of distutils code, starting from scratch would be easier -
Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;) 1/ I am on Guido side here. I'd rather refactor the existing code. 2/ It's there, it works for a lot of people. let's improve it slowly step by step 3/ the quality is increasing, the test coverage too. compare the trunk with Python 2.5 one.. my proposal: Step 1 - check the extension mechanism and see if it fits your brain Step 2 - see where you would like to refactor distutils in order to make it configurable/extensible -- Tarek
Tarek Ziadé wrote:
no it doesn't. it's a generic tool to extend a command using plugins.
OK.
Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;)
Yes, it is easy to say it sucks, but I am not saying that out of thin air. Refactoring works if the general API is ok - but that's not the case with distutils. As a data point, several people have tried to add features to distutils in numpy extensions, and I went much further than anyone else by bypassing distutils entirely, replacing the build_* commands by scons. The core codebase is ten times smaller than the original code it replaces (numpy.distutils is around 10 000 LOC, like distutils itself). I agree that in the short term, distutils should be improved, but in the long term, I don't see anything which would be both a significant improvement while staying backward compatible with distutils, if only because so many setup.py files depend on implementation details of distutils. cheers, Davud
On Mon, Apr 20, 2009 at 4:36 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Tarek Ziadé wrote:
no it doesn't. it's a generic tool to extend a command using plugins.
OK.
Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;)
Yes, it is easy to say it sucks, but I am not saying that out of thin air. Refactoring works if the general API is ok - but that's not the case with distutils. As a data point, several people have tried to add features to distutils in numpy extensions, and I went much further than anyone else by bypassing distutils entirely, replacing the build_* commands by scons. The core codebase is ten times smaller than the original code it replaces (numpy.distutils is around 10 000 LOC, like distutils itself).
I agree that in the short term, distutils should be improved, but in the long term, I don't see anything which would be both a significant improvement while staying backward compatible with distutils, if only because so many setup.py files depend on implementation details of distutils.
In the short term, Distutils needs to be a better citizen when people wants to extend it, so they can do it without re-writing everything. You have this experience of "pain", share it through real use cases. In the long term Distutils needs to be *smaller* and to provide a playground for other tools. in particular for all the tools that need to build a binary package on a given platform. (We can't keep those platform specific tools in the scope of distutils) And the work done on various PEPs (all links at the top of http://wiki.python.org/moin/Distutils) are aiming to this goal. What about organizing a sprint soon btw, so we don't loose Pycon efforts ? -- Tarek Ziadé | http://ziade.org
Tarek Ziadé wrote:
In the short term, Distutils needs to be a better citizen when people wants to extend it, so they can do it without re-writing everything.
You have this experience of "pain", share it through real use cases.
I have added 5 usecases in the manifest plugin page, in order of difficulty/pain. The first ones should be solvable in distutils with your proposal, I am not sure about the last ones. David
On Mon, Apr 20, 2009 at 16:36, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
I agree that in the short term, distutils should be improved, but in the long term, I don't see anything which would be both a significant improvement while staying backward compatible with distutils, if only because so many setup.py files depend on implementation details of distutils.
I think the answer to this can be only one: Since this is open source, you can't expect somebody to do it. It will only be done if you do it yourself, "you" being the people who experience the pain, in this case, well, you. :-D I sure as heck do not have the knowledge to do it. So the answer I think, is the cold and brutal "Yes! Great Idea! When are you done?" :-) What I'm worried about with distutils is both what you mention: Missing flexibility internally, and also that the plans are to make it *smaller*, which inevitably will break backwards compatibility, which will cause pains. And I still haven't gotten any answer on how to make a backport of a new distutils and have that automatically installed when you install a package that uses the new features. (Although maybe just failing with an error asking you to install distutils vX.Y might be acceptable). It's quite possible that it's better to make something that at least has another name, which totally avoids all the above problems. That doesn't mean the code has to be all new and shiny. So I sure wouldn't mind something new, that avoids the distutil problems. But wanting it is something else than doing it. I can't. Can you? And it would be needed rather quickly too. :-D At least somebody could maybe make some sort of rough plan of what this golden distutils should contain and how it should work, and we can get a feeling if it's even remotely feasible. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
2009/4/20 Lennart Regebro <regebro@gmail.com>:
At least somebody could maybe make some sort of rough plan of what this golden distutils should contain and how it should work, and we can get a feeling if it's even remotely feasible.
And, as I frequently run into walls that make me thing setuptools should be completely ignored, and then after fiddling about quite a bit, find a way around it, and then run into the next wall, etc, etc, etc. And these walls are getting more and more frequent... I'm beginning to think that indeed rewrites are the only way. We need a plan. Somebody please make a plan, maybe I can help. But probably not. But I can try. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
And, as I frequently run into walls that make me thing setuptools should be completely ignored, and then after fiddling about quite a bit, find a way around it, and then run into the next wall, etc, etc, etc. And these walls are getting more and more frequent... I'm beginning to think that indeed rewrites are the only way.
We need a plan. Somebody please make a plan, maybe I can help. But probably not. But I can try.
I second that especially because I think setuptools looks like an over engineering excercise: I have many doubts about about setuptools, especially from a system administrative point of view. Why a build system (and setup.py is) should have a scm support in it? Why the plugin entry points should be part of such a system? Why the dependecied should be part of it? The scm support is part of a "development" stage, possibly useful to the developers only: that'd be better covered by a python script not tighted to the "building system". The plugin entry points (belonging to a "deployment" stage) are a bit of a problem because the paths might not be in their "final" form (think of DESTDIR temporary staging area while building a rpm): this mean the onlyreliable way to define entry points would be relative to modulename.__file__ variable. The dependencies part (belonging to a "deployment" stage) is a bit of a system wide problem. Rpm defines the same fields and then this introduce a (not so) hidden duplication: will the package respect the python dependencies or the rpm ones? I would recomend having a look at what rpm (and especially the spec files) does because it had embedded a lot of knoweledge about building packages across the years: and it still has its own problems.... Moving forward, itwould be beneficial a flow chart of what distutils should do (and shouldn't) and a possibly a usage cases study. Regards, Antonio
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Antonio Cavallo wrote:
And, as I frequently run into walls that make me thing setuptools should be completely ignored, and then after fiddling about quite a bit, find a way around it, and then run into the next wall, etc, etc, etc. And these walls are getting more and more frequent... I'm beginning to think that indeed rewrites are the only way.
We need a plan. Somebody please make a plan, maybe I can help. But probably not. But I can try.
I second that especially because I think setuptools looks like an over engineering excercise: I have many doubts about about setuptools, especially from a system administrative point of view.
Why a build system (and setup.py is) should have a scm support in it?
Setuptools builds on distutils, which is primarily aimed at distributing software; "building" is a minor part of the exercise (a huge supermajority of distutils packages require no real "build" effort at all).
Why the plugin entry points should be part of such a system?
Distributing plugins for Chandler was PJE's driving use case in the original development of setuptools.
Why the dependecied should be part of it?
Because distributing interrelated software sanely requires it.
The scm support is part of a "development" stage, possibly useful to the developers only: that'd be better covered by a python script not tighted to the "building system".
I don't know where you get the idea that setuptools / distutils is primarily about "building" software. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ7cRp+gerLs4ltQ4RAikNAJ43+6Cs8aCE0WufUsR/FiG3Xk8Z2gCgkIR1 46oMc5ShfytLkqppE2jN4do= =Vhj/ -----END PGP SIGNATURE-----
Lennart Regebro wrote:
We need a plan. Somebody please make a plan
Whatever plan you come up with, I'd like to see the basic architecture built around a dependency graph, at least for the parts concerned with building extension modules from sources. My use case for handling Pyrex sources. I'd like to be able to plug in a module that knows how to turn .pyx files into .c files, without requiring arcane internal knowledge and without interfering with any other extensions I might want to use at the same time. -- Greg
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution. http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft On Tue, Apr 21, 2009 at 7:19 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Lennart Regebro wrote:
We need a plan. Somebody please make a plan
Whatever plan you come up with, I'd like to see the basic architecture built around a dependency graph, at least for the parts concerned with building extension modules from sources.
My use case for handling Pyrex sources. I'd like to be able to plug in a module that knows how to turn .pyx files into .c files, without requiring arcane internal knowledge and without interfering with any other extensions I might want to use at the same time.
-- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org
On Tue, Apr 21, 2009 at 08:44, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
I realized another usecase, which may or may not be covered already, should I add it to the page? == Optional extensions == Some modules, such as zope.interface, have C-extensions which are only performance enhancing and have pure-python fallbacks. It should be easy to write an a command that if it fails doesn't stop the rest of the build, and also includes the right files if it succeeds, but doesn't fail when these files can't be built. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
I can identify three major areas of concern: 1) Platform testing/builds (via cloud computing) 2) External build tools 3) too many versions of python One thing you guys should also consider is automated building/testing of packages. On different platforms. Let me give you a practical example... package win32... latest version.. no longer works on windows 2000... only works on xp... or vista... Bad for me.. takes me time to figure out whats wrong... Only solution.. go back and find an older release.. yes.. i know.. I can do that... that works... With cloud computing... you can get test environments for every platform Amazon web services as one example offer this Perl does nightly "builds" and tests.... their automated test system "tests everything" Another big issue i dislike with many packages... they depend on having msvc installed to be able to build them. In many cases because I don't have MSC installed... this prevents me from building some packages. I don't like that... I don't believe building "everything" on the 5-10 major platforms we have (or however many there are) is too much. Plus now, we have all these different versions of python.. and instead of getting better it is only seeming to get more worse... David
One thing you guys should also consider is automated building/testing of packages. On different platforms.
Let me give you a practical example...
package win32...
latest version.. no longer works on windows 2000...
only works on xp... or vista...
Bad for me.. takes me time to figure out whats wrong...
Only solution.. go back and find an older release.. yes.. i know.. I can do that... that works...
With cloud computing... you can get test environments for every platform
Amazon web services as one example offer this
Perl does nightly "builds" and tests....
their automated test system "tests everything"
Another big issue i dislike with many packages... they depend on having msvc installed to be able to build them.
In many cases because I don't have MSC installed... this prevents me from building some packages. I don't like that...
I don't believe building "everything" on the 5-10 major platforms we have (or however many there are) is too much.
Plus now, we have all these different versions of python.. and instead of getting better it is only seeming to get more worse...
I had this idea of running a test server that would listen to PyPI and get every new package to build it and test it on various platform/python. I think this could be done right now, with how distutils works, It requires creating virtual machines and destroying them every time for security reasons. But it's quite a work... Cheers Tarek -- Tarek Ziadé | http://ziade.org
On Tue, 21 Apr 2009 09:21:12 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
I had this idea of running a test server that would listen to PyPI and get every new package to build it and test it on various platform/python.
I think this could be done right now, with how distutils works,
It requires creating virtual machines and destroying them every time for security reasons.
But it's quite a work...
I think it is very important to put this software infrastructure into place. or at least include it in some sort of plan going forward.... have the server farm report build bugs back to the maintainers... I am prepared to assist with this... and perphaps cover some of the costs. Amazon web services provide the very same virtual machines that you are describing at extremely low cost. Otherwise it is just two much work. Google also offer a similar thing. David
On Tue, Apr 21, 2009 at 11:17 AM, David Lyon <david.lyon@preisshare.net> wrote:
But it's quite a work...
I think it is very important to put this software infrastructure into place.
or at least include it in some sort of plan going forward....
have the server farm report build bugs back to the maintainers...
I am prepared to assist with this... and perphaps cover some of the costs.
Amazon web services provide the very same virtual machines that you are describing at extremely low cost. Otherwise it is just two much work.
Google also offer a similar thing.
Wonderful ! Money-wise, I think the PSF would probably cover these expenses if we come up with a good, well-defined project. I have tried to make it a GSoc topic but I didn't find any student picking it up because I didn't find the time to describe it deeply enough. I have created a page here : http://wiki.python.org/moin/Distutils/TestingInfrastructure If you want to start there on the topic, that could be great. Tarek
David
-- Tarek Ziadé | http://ziade.org
Maybe we should stop talking about how to improve distutils under the topic of what the problem is with setuptools under Python 3? It's pretty unrelated. ;-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
SuSE runs a build service for free and support automatic rebuild of packages from sources: https://build.opensuse.org For anyone interested you can find the lates svn python snapshot under: http://download.opensuse.org/repositories/home:/cavallo71:/python-opt/ Each subdirectory (CentOS_5, RHEL_5 etc) contains a yum repository to download the packages opt-python-XXXX These python packages install under their own separate directory tree (/opt/opt-python-2.7a0 is the latest snapshot). In order to use one of them all it is needed is: $> . /opt/opt-python-2.7a0/opt-python-env.sh $> python Python 2.7a0 (trunk, Apr 20 2009, 12:45:44) [GCC 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import os; print os.__file__ /opt/opt-python-2.7a0/lib/python2.7/os.pyc
I hope this helps in testing, Regards, Antonio On Tue, Apr 21, 2009 at 10:30 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Tue, Apr 21, 2009 at 11:17 AM, David Lyon <david.lyon@preisshare.net> wrote:
But it's quite a work...
I think it is very important to put this software infrastructure into place.
or at least include it in some sort of plan going forward....
have the server farm report build bugs back to the maintainers...
I am prepared to assist with this... and perphaps cover some of the costs.
Amazon web services provide the very same virtual machines that you are describing at extremely low cost. Otherwise it is just two much work.
Google also offer a similar thing.
Wonderful ! Money-wise, I think the PSF would probably cover these expenses if we come up with a good, well-defined project. I have tried to make it a GSoc topic but I didn't find any student picking it up because I didn't find the time to describe it deeply enough.
I have created a page here : http://wiki.python.org/moin/Distutils/TestingInfrastructure
If you want to start there on the topic, that could be great.
Tarek
David
-- Tarek Ziadé | http://ziade.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Tarek Ziadé wrote:
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
Could the people adding the use cases put their names by them, so if we have questions later we know who to ask?
Eric Smith wrote:
Tarek Ziadé wrote:
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
Could the people adding the use cases put their names by them, so if we have questions later we know who to ask?
Good idea, I have added my names to 'my' examples, David
Tarek Ziadé wrote:
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
Is this really the best place for it? My use case doesn't have anything to do with manifests. We need a page like this concerning overall distutils reorganisation. -- Greg
Greg Ewing wrote:
Tarek Ziadé wrote:
Please Greg, add your use case here with David's one, at the end of the page, since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft
Is this really the best place for it? My use case doesn't have anything to do with manifests.
Mine neither. I agree the title is confusing, but OTOH, moving things in wiki is easy, whereas waiting for things to be at the right place increases the risk of losing the information. That's how wiki work. We need more people who care about the building issues (vs the distribution issues) if we want to be taken in account for distutils improvements :) David
I really second this... after all.. we're only talking about a file copying program.. Surely after so many years... we must have a clear handle on what files need to be copied where with what permissions.... On Mon, 20 Apr 2009 23:00:39 +0900, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
IMHO, adding plugins systems will not change the fundamental deficiencies of distutils, though. You may be able to mitigate some problems, but I don't think you will be able to solve the fundamental issues, because they are fundamental design issues. The only way to fix them would induce serious breakage, and in that case, given the quality of distutils code, starting from scratch would be easier - I have yet encountered a single major distutils abstraction which was not poorly conceived and implemented (my own experience is related to build issues, for which numpy has requirements which go much beyond the usual python package, though). Commands classes, compiler classes, distributions, all this is very wrong for someone who wants more than compiling a couple of C files and tight control.
Hi Lennart, Sorry for my late posting... I'm working on a gui python package manager on sourceforge...
Executive summary: I have catch 22 problems in finishing the setuptools port to Python 3, and unless somebody can come up with alternatives, I may rip out setuptools dependency on itself completely, and only depend on distutils.
... It's not easy I agree... I am a GUI junky myself... :-) I don't use chainsaw (code) unless i need to I'm trying out all the installers on one machine... einstaller won't install because I already installed setuptools.... ez_installer won't de-install setuptools... pip doesn't want to go on either... I will figure it out... I am sympathising saying this package stuff is hard going....
I don't have a good solution to this, unless we can drop setuptools dependency on setuptools completely, and just use plain distutils for installing and testing setuptools. This will mean no more setuptools source eggs, but then I don't see the purpose of those, really. Arguments for not doing this and alternative solutions are highly welcome.
:-) me too
participants (10)
-
Antonio Cavallo -
Ben Finney -
David Cournapeau -
David Lyon -
Eric Smith -
Greg Ewing -
Lennart Regebro -
Paul Moore -
Tarek Ziadé -
Tres Seaver