Mailman 3 April 2009 - Distutils-SIG

Metadata-Version in PKG-INFO
by Carlos Tejo Alonso 22 Apr '09

22 Apr '09

Hello, This is my first email in distutils-sig mailing list. So I may introduce myself. I am Carlos Tejo, a member of the Semantic Web activity line at CTIC Foundation, a non-profit organization, where I work with a team of research fellows. We are developing an small library [1] in python language, and now we want to release it properly :-) I wonder how I should "write" setup.py in order that setuptools use the parameters of setup() to create a proper PKG-INFO file. Right now, I have a couple of troubles: - How can be configured the version of the metadata (AFAIK, rigth it is always "Metadata-Version: 1.0")? I would like to use version 1.1 (PEP 314. status: final) [2] or version 1.2 (PEP 345. status: draft) [3] - How should be written the "platform" parameter? Regards, ------------------------------------- Carlos Tejo Alonso Research & Development Department CTIC Foundation [Asturias, Spain] www.fundacionctic.org ------------------------------------- [1] http://rest-in-py.sourceforge.net/ [2] http://www.python.org/dev/peps/pep-0314/ [3] http://www.python.org/dev/peps/pep-0345/

4 7

Distutils work, roadmap
by Tarek Ziadé 22 Apr '09

22 Apr '09

Remainder about Distutils: - we did a summit at Pycon about distutils - some people started to work on various tasks (PEPs, etc) Can everyone here who says "distutils sucks" "setuptools sucks", "let's rewrite them from scratch" hear that: Just join the work in progress, help on the PEP, uses cases, and stop whining about the situation !!!! Thank you -- Tarek Ziadé | http://ziade.org

4 8

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby 22 Apr '09

22 Apr '09

At 02:26 PM 4/21/2009 +0200, Lennart Regebro wrote: >So why don't I use that for setuptools? Well, because: > >c) The setup of setuptools requires setuptools. So to be able to do >the 2to3 conversion in the setup, I need to first convert the source >with 2to3. Yes, catch 22. What I still don't get is why it can't work in a 2-stage process, running one setup.py with distutils to do the build_2to3, and then running a different setup.py to do the tests. I imagine that, if I were trying to support Python 3, what I would do first is make a Python 2 setuptools command that ran 2to3 on a setuptools-based Python 2 project, and generated a new source tree -- with all sdist-targeted content copied over, and all .py files converted (including setup.py itself)... and then ran whatever extra commands you gave, running Python 3 on the resulting setup.py, such that: python2 setup.py 2to3 test would automatically do the equivalent of cd build/2to3; python3 setup.py test after creating a converted distribution in build/2to3. That way, you could also do things like: python2 setup.py test 2to3 test to run the tests in Python 2 before converting and running them in Python 3. If somebody wants to create this command, perhaps that would be a good idea. It can of course be implemented as a plugin, so a change to setuptools itself is not required. In the simplest case, the command could just derive from sdist and build an sdist tree in build/2to3 each time, and then run 2to3 in place. Or it could reuse sdist and unpack the sdist into the build tree. This would be slower, but easier to code. A more advanced version could check for changes in SOURCES.txt between the original and the build/2to3 directory in order to find files to add/remove, and only run 2to3 on changed .py files. Something like: if not os.path.exists(target_sources_txt): # wipe build tree, build sdist and unpack target_manifest = load_manifest(target_sources_txt) for filename in target_manifest: if filename not in original_manifest: # delete file continue if is_older_or_nonexistent(filename): # copy or run 2to3 for filename in original_manifest: if filename not in target_manifest: # copy or run 2to3 ...but with a lot more os.path.join operations and a lot less handwaving. ;-) And the initial version of this could just always do the wipe-and-unpack step, although it'd still need to loop over the files to run 2to3 anyway, I suppose. Anyway, I know this is a fair amount of work; It just seems to me that it has more uses than converting setuptools; i.e., it'd be a useful rig for anybody doing 2-to-3 porting work.

4 4

What's missing from easy_install
by Neal Becker 22 Apr '09

22 Apr '09

1. easy_remove! 2. Various utilities to provide query package management. - easy_install --list (list files installed)

12 18

The problem with Setuptools on Python 3.
by Lennart Regebro 22 Apr '09

22 Apr '09

Executive summary: I have catch 22 problems in finishing the setuptools port to Python 3, and unless somebody can come up with alternatives, I may rip out setuptools dependency on itself completely, and only depend on distutils. I have as you know ported setuptools to Python 3. I'm in the process of trying to fix and clean up that porting so that it can be merged into trunk, but there is one obstacle left, and that's the question of installing. Currently there are several ways to install setuptools: 1) Run ez_install.py 2) Download / check-out source and run python setup.py install. 3) Download the correct egg and install that (which always confuses me, I usually fail, and end up doing 2) instead). Setuptools also helps with running tests by 4) python setup.py test Currently for the Python 3 version, I've made a script that copies the source tree and runs 2to3 on it. Then you can use that new tree to run tests, install, make eggs and releases etc. The problems with this is: a) The script is made for my computer only. It wouldn't work on Windows, and it requires you to have lib2to3 checked out in a particular place (although I think that's only if you use 3.0.0, and not 3.0.1, but I haven't tested). b) It's slow, as it copies all files, even those who hasn't changed when you fixed a bug. Making a script that only copies the changed files and runs 2to3 on them is possible, but I'd like not too, it takes time and is likely to break. It seems like a waste. Obviously, this is exactly the things distutils are meant to solve. And indeed, distutils in Python 3 has a command called build_2to3 which solves exactly these problems. And indeed, this is the technique I've used for the Python 3 branch of zope.interfaces. There I just run setup.py install, and if I'm doing that under Python 3, it will first run 2to3 on the code before installing. Same thing with running setup.py test. It will sync changes to build/, run 2to3 on changed files and then run the tests in build. It makes porting to Python 3 much easier, and it makes installing from source painless. So why don't I use that for setuptools? Well, because: c) The setup of setuptools requires setuptools. So to be able to do the 2to3 conversion in the setup, I need to first convert the source with 2to3. Yes, catch 22. I've tried to get around this problem, but failed. Solutions I tried was: I) Fall back to distutils if setuptools can't be imported. This means that I can with some effort make a setup that will work under Python 3 and install setuptools under Python 3. Problem solved? No. Because running setup.py again will just mean that it tries to import setuptools from the local Python2 location, and it will fail. So in this case I can't run tests of build eggs for setuptool. II) Moving the source into a src directory. This means that when setup.py runs, it will use the version of setuptools installed under I), and building eggs etc is possible (I think, I haven't tested that the eggs are correct). Problem solved? No, because you still can't test it. Which leads us to problem d: d) When running the tests, the setuptools module will already be imported. But it will have imported the one in site-packages, *not* the one in src, and it will therefore test the one in site-packages. And that one doesn't have the api_tests.txt copied in, so that will fail, and e) even if api_tests.txt was copied, this would mean you had to install setuptools before you test it, which is daft. I don't have a good solution to this, unless we can drop setuptools dependency on setuptools completely, and just use plain distutils for installing and testing setuptools. This will mean no more setuptools source eggs, but then I don't see the purpose of those, really. Arguments for not doing this and alternative solutions are highly welcome. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

10 32

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby 21 Apr '09

21 Apr '09

At 04:06 PM 4/21/2009 +0200, Lennart Regebro wrote: >On Tue, Apr 21, 2009 at 15:03, P.J. Eby <pje(a)telecommunity.com> wrote: > > python2 setup.py 2to3 test > >Well, yes, but it should be > > python 3 setup.py 2to3 test > >Otherwise it can't reasonably have any idea of which python to use. Why not? The 2to3 command could simply take an option for the python3 executable, and be set from the standard config files (e.g. setup.cfg). >And when you run it with python3, the 2to3 command isn't needed, as >the 2to3 step can be automatically inserted through version detection. Um... I'm still not following you. You sound like you're describing something very different from what I outlined. >In fact, this is exactly what I'm doing for zope.interface, as >explained. For zope.interface, you run > python2.5 setup.py test >and it will test it with python2.5 You run > python3.0 setup.py test >and it will do exactly what you say: copy the files to build, run >2to3, and then run the test on that in-place. > python3.0 setup.py install >will copy the files to build, run 2to3 and then install it. Right -- but this is totally *not* what I'm saying. Because the strategy that I outlined doesn't require setuptools to be ported to python 3 first. >Yes. That's is exactly what I'm trying to achieve. And I aim to extend >setuptools with build_2to3 and test_2to3 commands to do exactly that. >But to do that I need a good setuptools distribution that supports >Python3. No, you don't. You only need one command, implemented as a plugin for the Python2 version of setuptools, and you drive everything from python2. Once you get to the point where setuptools is ported enough to be able to have the Python3 version actually run its tests without crashing, you can go self-hosted. But before that point, you need Python2 as the bootstrap/host environment for the build process. >It explains why the test cases >could run even though setuptools still couldn't install itself. I >thought it was badly tested, now I know it's by design. No, it's also badly tested; that's kind of inherited from the distutils. And by the time I added the 'sandbox' virtualization tool, there was already tons of stuff that wasn't tested, such that it didn't seem worth it to do something else.

1 0

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby 21 Apr '09

21 Apr '09

At 09:11 PM 4/20/2009 +0200, Lennart Regebro wrote: >"Isolated"? What do you mean? Making a separate setup script for Python 3, at least for setuptools itself, if not having a general convention for that, since other packages may want to ship 2+3 stuff in the same package. Or, in the alternative, using version testing in setup.py to run an alternate script for Python 3. >If you have suggestions on how to solve >the problems I'm describing, can you please come forward with them, >I've asked several times already. All you do is say "no, no, no, no, >no", and "becasue". It isn't helpful. I'm sorry you feel that way, as I've been *trying* to help. I just still don't get what the problem is. If I were porting setuptools to Python 3, I would *want* it to be circular, even if I had to hack on it a little at first. So I have a hard time understanding why you don't. But I'm not you -- I understand the code base and I'm not trying to port it. Maybe if I were trying to port it, I would get what problem you're having, or maybe I would just keep right on going and not notice. I don't know. I've been trying to find out what exactly is stopping you and just can't seem to wrap my brain around it, any more than you've been able to about the reverse. > > For end-user convenience. A large number of people installing setuptools > > are not installing it because they are personally interested in it, but > > because something else they want uses it. > >Yes? And again you don't explain how this leads to you conclusion. Mostly because your questions aren't pinpointing what you want to know. You ask general questions, so I give general answers. We seem to both be suffering from a surplus of assumptions of what should be "obvious" to the other person. For example, this: >Eggs are not easier to install, on the contrary, I have tried and >failed a couple of times, and ended up using the source install >instead. ...seems to indicate that your question was actually about eggs, not other-than-source distributions in general. I was mostly talking about the wininst installers and source RPM. The eggs are there, on the other hand, for ez_setup.py to download. (Not to mention buildout's bootstrap script, and other tools that depend on setuptools and want to have an automated overall install process.)

4 5

mirror plone buildout
by Mihamina Rakotomandimby (R12y) 21 Apr '09

21 Apr '09

Hi, I use a quite bad internet connection: - Small bandwidth (128KBps) - Many micro stalls - 1s latency to the world... I would like to first download the Plone (+deps) locally, with a dedicated FTP client, that can handle more or less correctly the stalls. Have you got some related documentation somewhere in your bookmarks? -- Chef de projet chez Vectoris Phone: +261 33 11 207 36 System: xUbuntu 8.10 with almost all from package install http://www.google.com/search?q=mihamina+rakotomandimby

2 1

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby 21 Apr '09

21 Apr '09

At 06:31 PM 4/20/2009 +0200, Lennart Regebro wrote: >Let me reformulate that: > > > Because that's the one that generates the metadata setuptools needs to run, > > test itself, etc. > >Why do I need setuptools to do that? Why is not distutils enough? Because distutils doesn't have an egg_info command, or SVN-based manifest building (for making sdists).

2 1

Help making setuptools install more like plain distutils one
by Christian Hudon 20 Apr '09

20 Apr '09

Hello, We have a setup where a central /usr/local is copied to all our machines. The packages installed in said central /usr/local are managed via stow. (Basically, each package is installed in a separate directory under /usr/local/stow, and invoking the stow command creates symlinks to make the package appear under /usr/local.) This works very well for a wide range of packages: autoconf packages, CRAN packages for R, etc. This includes plain distutils python packages (install via "python setup.py install --prefix /usr/local/stow/some-package-name", then run stow). The only exception is setuptools-based python packages. Is there a way to ask setuptools to do an install that looks more like a standard distutils install? I don't care if I lose some the advanced setuptools features. Basically, I need an install that's done via just copying new files. Problems like setuptoosl checking that the destination directory is in the PYTHONPATH I can work around (although if there's a switch to disable that check, I'd be happy to learn about it). The main problem is the file that's edited on each install to add a new line for each package install via setuptools. Is there a way with setuptools of getting just a directory tree (or a tarball, etc.) that either setuptools or myself can just copy somewhere to have an installed python module? Thanks in advance for any help, Christian

4 5