[Distutils] The problem with Setuptools on Python 3.

[P.J. Eby]

At 02:26 PM 4/21/2009 +0200, Lennart Regebro wrote:
>So why don't I use that for setuptools? Well, because:
>
>c) The setup of setuptools requires setuptools. So to be able to do
>the 2to3 conversion in the setup, I need to first convert the source
>with 2to3. Yes, catch 22.

What I still don't get is why it can't work in a 2-stage process, 
running one setup.py with distutils to do the build_2to3, and then 
running a different setup.py to do the tests.

I imagine that, if I were trying to support Python 3, what I would do 
first is make a Python 2 setuptools command that ran 2to3 on a 
setuptools-based Python 2 project, and generated a new source tree -- 
with all sdist-targeted content copied over, and all .py files 
converted (including setup.py itself)...  and then ran whatever extra 
commands you gave, running Python 3 on the resulting setup.py, such that:

   python2 setup.py 2to3 test

would automatically do the equivalent of

   cd build/2to3; python3 setup.py test

after creating a converted distribution in build/2to3.  That way, you 
could also do things like:

   python2 setup.py test 2to3 test

to run the tests in Python 2 before converting and running them in Python 3.

If somebody wants to create this command, perhaps that would be a 
good idea.  It can of course be implemented as a plugin, so a change 
to setuptools itself is not required.

In the simplest case, the command could just derive from sdist and 
build an sdist tree in build/2to3 each time, and then run 2to3 in 
place.  Or it could reuse sdist and unpack the sdist into the build 
tree.  This would be slower, but easier to code.  A more advanced 
version could check for changes in SOURCES.txt between the original 
and the build/2to3 directory in order to find files to add/remove, 
and only run 2to3 on changed .py files.

Something like:

    if not os.path.exists(target_sources_txt):
        # wipe build tree, build sdist and unpack

    target_manifest = load_manifest(target_sources_txt)
    for filename in target_manifest:
        if filename not in original_manifest:
            # delete file
            continue
        if is_older_or_nonexistent(filename):
            # copy or run 2to3
    for filename in original_manifest:
        if filename not in target_manifest:
            # copy or run 2to3

...but with a lot more os.path.join operations and a lot less 
handwaving.  ;-)  And the initial version of this could just always 
do the wipe-and-unpack step, although it'd still need to loop over 
the files to run 2to3 anyway, I suppose.

Anyway, I know this is a fair amount of work; It just seems to me 
that it has more uses than converting setuptools; i.e., it'd be a 
useful rig for anybody doing 2-to-3 porting work.