[ python-Bugs-1539295 ] Long command lines don't work on Windows

Wed Aug 16 15:15:45 CEST 2006

Bugs item #1539295, was opened at 2006-08-12 21:28
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1539295&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Distutils
Group: None
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Submitted By: Albert Strasheim (albertstrasheim)
Assigned to: Nobody/Anonymous (nobody)
Summary: Long command lines don't work on Windows

Initial Comment:
Windows has a 32k command line limit, described here:

http://blogs.msdn.com/oldnewthing/archive/2003/12/10/56028.aspx

The SciPy project is running into problems when using
distutils to compile LAPACK from source, because the
large number of source files results in a command line
of more than 70,000 characters.

Windows has a way of dealing with this problem, namely
putting the command line arguments in a file and
passing that file to executable with prepended with an
@, i.e. you want to run something like

lib.exe @tempdir\tempargs.lnk

SCons takes care of this through it's TEMPFILE
mechanism which uses the TempFileMunge class which can
be perused here:

http://scons.tigris.org/source/browse/scons/trunk/src/engine/SCons/Platform/__init__.py?rev=1582&view=markup

I think this problem can easily be addressed in
distutils by modifying the _spawn_nt method in spawn.py
to switch to this temp file method when the command
line length exceeds some threshold.

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2006-08-16 15:15

Message:
Logged In: YES 
user_id=21627

As you just found out, this isn't a Windows feature. It's a
feature of certain command line tools; primarily, it's a
feature of the Visual Studio and PSDK command line tools.
Each tool has to implement this machinery for itself. So I'm
closing this as "won't fix"; alternatively, it could be
closed as "third-party bug" (i.e. Windows limitation).

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-13 04:16

Message:
Logged In: YES 
user_id=945096

On further testing it seems this patch is a bit overzealous.
The @ magic doesn't work for Windows commands in general. :-(

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-13 03:51

Message:
Logged In: YES 
user_id=945096

New patch.

Included a test case that fails without the changes and
passes with the changes. Modified the code to incorporate
some of nnorwitz's suggestions, except for the mkstemp change.

The test case has a minor bug: the output of the build is
still created relative to the the current working directory.
I don't know distutils very well, so I'm not quite sure
which option to mangle to fix this.

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-13 02:44

Message:
Logged In: YES 
user_id=945096

Okay, I'm going to try to implement some of your suggestions.

A comment at the top of spawn.py says:

# This module should be kept compatible with Python 2.1.

but mkstemp was only added in Python 2.3. Is this comment
still valid?

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-08-13 02:17

Message:
Logged In: YES 
user_id=33168

Ah, our msgs crossed.  That explains the 2k.  It would be
nice to add a comment about that.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-08-13 02:16

Message:
Logged In: YES 
user_id=33168

Thanks for the patch!  It looks pretty good, but there are a
few changes (and additions) I would like to see.

I don't understand why you use 2k in the patch, but mention
the limit is 32k.

You should use mkstemp() instead of mktemp().  That will
open the file for you, so you can just do os.write(fd,
data).  You will need to close the file.

I would prefer to see a single os.spawnv() call.  You would
need tmp defined (but possibly None) and in the finally, you
would have to check if tmp: os.remove().

It would also be nice to see test and doc updates.  The
tests are in Lib/distutils/tests.

Thanks again for the patch.

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-13 02:09

Message:
Logged In: YES 
user_id=945096

Looks like the max length on Windows 95 might be as low as
256 characters, but I'm not sure about this. It seems
Windows 98 and Windows Me have a limit of 2048 characters.

SCons assumes 2048 characters on any Windows system, so I
did the same.

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-13 02:03

Message:
Logged In: YES 
user_id=945096

Attached a first try at a patch.

Some things that could still be changed:

- We might want to print a different log info message when
using a temp file instead of calling the command directly

- Restructure the code so that the spawnv call only appears
in one place

The setup.py I attached should allow anyone to duplicate
this problem.

----------------------------------------------------------------------

Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-08-12 23:56

Message:
Logged In: YES 
user_id=945096

Okay, I'll see what I can do.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-08-12 23:53

Message:
Logged In: YES 
user_id=33168

Albert, could you try to work up a patch implementing this?
 It would be much more likely to get fixed that way.  It
should also work with older Windows (9x/ME).

Cheers,
n

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1539295&group_id=5470