[Python-bugs-list] [ python-Bugs-478339 ] Linking/compiling with DEBUG
noreply@sourceforge.net
noreply@sourceforge.net
Sun, 11 Nov 2001 06:18:29 -0800
Bugs item #478339, was opened at 2001-11-05 07:45
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470
Category: Windows
Group: Python 2.1.1
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Mark Hammond (mhammond)
Summary: Linking/compiling with DEBUG
Initial Comment:
I have an embedded use of the Python interpreter. I
want to compile and link the encapsulating application with
and without DEBUG. I have disabled the use of the
pragma in config.h to always use the library specified
in the link statement, which is always the standard
python library. I have ensured that all invocations of
Python.h undefine the DEBUG symbol before including them.
Compiling the application without debug: works fine if
you link non-debug. However, linking with debug causes
some change in the way python modules are looked up and
I'm getting undefined local modules. I have checked the
PYTHONPATH environment variable and it is correct, and
the .py module exists in the correct directory. (This
is a common scenario where some libraries are compiled
without DEBUG, some with DEBUG, and then the whole
application linked DEBUG.)
Compiling the application with debug: Link errors
result. Undefined -- _Py_RefTotal, _Py_Dealloc,
_PyInitModule4TraceRefs.
Python is making too many hidden assumptions about the
build environment when it is being used embedded. This
is a critical problem as it makes it impossible to
build the encapsulating application for debug purposes.
----------------------------------------------------------------------
>Comment By: Martin v. Löwis (loewis)
Date: 2001-11-11 06:18
Message:
Logged In: YES
user_id=21627
On Linux, it is common to include debug information into
every library and executable; in particular, it is common
to install python with debugging information. This is
possible since debugging information can be combined with
optimization almost seamless in gcc (distributors often
strip the debugging information, though).
So you use Py_DEBUG *only* when you want to do the refcount
debugging. You have to configure --with-pydebug for that,
and you never do that in the wild. *If* you would install a
pydebug binary, you would use a different --prefix, so that
it would not interfere with the standard installation.
Depending on which pyconfig.h extensions pick up, they would
then be suitable for use with one or the other installation.
----------------------------------------------------------------------
Comment By: Mark Hammond (mhammond)
Date: 2001-11-11 02:37
Message:
Logged In: YES
user_id=14198
*sigh* - forgot that the object layout changed :(
Re malloc issues: they are indeed still a problem - but
Python will not *force* you to do this. It used to be the
case that an extension module would malloc an object, and
whatever module did the final Py_DECREF (often the core)
would perform the free(). I believe this is no longer
true. It *is* still true with FILE objects and the
tp_print slot IIRC.
My point with the "_d" was that iff we can get things
working so that the standard Python extensions
are "safe", "_d" need not imply anything other than "the
preferred extension for debug builds of the core" - and
even that is only for b/w compat and convenience reasons.
How has Linux survived so long? I assume Py_DEBUG builds
are simply never made in the "wild".
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-11 02:27
Message:
Logged In: YES
user_id=21627
I thought the problem with memory blocks still exists: If
you malloc with the debug CRT, and free with the nodebug
CRT, you get a leak. At least Q140584 claims the problem
still exists in 6.0sp5. In addition to memory and files,
they also notice that setting the locale in one CRT doesn't
affect the other.
I still cannot tell whether Anonymous builds a DLL wrapper
or a static library around pythonxy.dll.If it was a DLL, it
would be indeed equivalent to the pyexpat case. However, in
that case, linking Python wouldn't matter at all to the
clients of his or her DLL, since the reference to
pythonxy.dll/pythonxy_d.dll would be only in the wrapper DLL.
Re: decoupling _DEBUG and Py_DEBUG. How exactly would that
work? Would a _d extension imply _DEBUG, or Py_DEBUG, or
would we stop using _d?
If it would imply _DEBUG, I cannot understand Mark's comment
"Individual extensions would be responsible for setting
Py_DEBUG via their makefiles in debug builds. This should
be stable." You have to use Py_DEBUG consistently throughout
all modules because it changes the struct _object layout.
Mixing them certainly isn't stable.
Perhaps we could try to completely avoid changing
PyObject_HEAD depending on Py_TRACE_REFS, and make it a type
flag instead. In that case, you could, at run-time, mix
objects that do trace_refs, and objects that don't.
----------------------------------------------------------------------
Comment By: Mark Hammond (mhammond)
Date: 2001-11-11 01:15
Message:
Logged In: YES
user_id=14198
The problem only occurs when the 2 DLLs must cooperate WRT
CRT resources. Eg, if a "FILE *" was ever passed and used
across the expat->Python boundary, or a memory block
allocated in one to be freed in the other, then we would be
in the same situation. This probably doesn't happen for
expat and Tk. For Python and its extension modules (and
even embedders), it may.
IIRC, the problem with memory blocks was solved some time
ago, leaving FILE* the only real problem. File objects are
used in only 1 or 2 cases, and these could be solved by
moving to "PyFile_Write()" style APIs.
So, I guess I could agree a good solution is to ensure that
Python C API does not require CRT resource sharing, and
that Py_DEBUG is defined independently of _DEBUG or DEBUG.
Individual extensions would be responsible for setting
Py_DEBUG via their makefiles in debug builds. This should
be stable.
This would blur the _d convention. I quite like it, but I
agree it is a PITA for many people. We could try giving
pythonxx_d.dll the ability to load non _d versions, in the
hope that the extensions have been "fixed" so they don't
directly share the CRT.
Tim - never quite understood your feelings on _d - what are
your thoughts?
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-11-10 22:15
Message:
Logged In: YES
user_id=31435
Martin and Mark, I want to play devil's advocate here just
a bit: How do we get away with, e.g., using expat.dll in
Windows Python? That is, we use the same expat.dll with
the release and debug Python builds. Nothing bad happens
that I've seen, and we don't get warnings about mixing CRT
types either. IOW, we use expat.dll the way Anonymous
would like to use the Python DLL. How do we get away with
that?
If that one is easy <wink>, ee don't use different versions
of the Tcl/Tk libraries either.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-10 02:13
Message:
Logged In: YES
user_id=21627
The second rule should read
Linking with python_d -> compile client code with _DEBUG
since Python does not care whether DEBUG is defined or not;
it only cares about _DEBUG (i.e. /MDd, as you point out).
Furthermore, the second dependcy is also thanks to
Microsoft: If you compile your library with /MDd, you must
also compile all client code with /MDd, see MSKB Q140584).
Python currently supports two configurations on Windows:
1. Neither of _DEBUG, Py_DEBUG is defined; link with
pythonxy.dll and msvcrt.dll (i.e. /MD); extensions end with
.pyd or .dll
2. Both _DEBUG, Py_DEBUG are defined, link with
pythonxy_d.dll and msvcrtd.dll, extensions end with _d.pyd
or _d.dll.
It would be possible to create a third configuration:
_DEBUG is defined, but Py_DEBUG isn't. Link with a no-debug
pythonxy.dll, but with msvcrtd.dll. You couldn't use the
existing pythonxy.dll and existing extension modules, since
those already link with msvcrt.dll. Adding such a
configuration would increase the mess, so it won't happen.
I'd advise you to pursue a fourth configuration: Do not use
the debug CRT (i.e. compile with /MD instead of /MDd), even
if you compile your own library for debugging. With that
configuration, you can use the standard Python distribution
with no changes.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 11:30
Message:
Logged In: NO
Turns out the _DEBUG is defined when using the /MDd
compiler option.
We have painstakingly worked out our build system to
deal with the library conflicts you mention. If I
understand what you're saying, these are the rules:
Linking with debug CRT -> link with python_d
Linking with python_d -> compile client code with DEBUG
The first rule is thanks to MS. The second rule is to
get internal object layouts to match up.
Is this right?
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-09 10:17
Message:
Logged In: YES
user_id=21627
I'm glad the PYTHONPATH issue is resolved.
On the _DEBUG issue: Somebody *must* be defining _DEBUG
somewhere, or else the #ifdef would not trigger. Please open
the MSVC++ project file of your project (the .dsp file) in a
text editor, and search for _DEBUG.
If you cannot find it there, please try to activate full
command line printing in your build process, so that you see
what options are passed to the compiler.
If you are absolutely sure that _DEBUG is not passed in the
command line, it must be defined in one of the header files.
In this case, please put the #ifdef _DEBUG block of the
previous test after each #include, e.g. in a form
#include <stdio.h>
#ifdef _DEBUG
#error After stdio.h
#endif
#include <nextfile.h>
#ifdef _DEBUG
#error After nextfile.h
#endif
I'm sure there are better ways to find out where _DEBUG is
defined; don't hesitate to apply them if you are aware of any.
Once you found the location of _DEBUG, you may remove it to
be able to use the standard Python .DLL.
Please make sure you understand Mark's comment on CRTs,
though: Many problems will occur if you mix the debug and
the nodebug version of the MSVC libraries (e.g. msvcrt.dll
and msvcrtd.dll). Please see the Microsoft documentation for
details: If you use pythonxy.dll, you absolutely *must* make
sure that you do not link the debug versions of the CRT,
even when performing a debug build. Unfortunately, you won't
get a compilation error if you do not follow this guideline;
instead, very hard-to-debug problems will occur. You can use
depends.exe on the final executable to analyse the set of
libraries used.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 08:16
Message:
Logged In: NO
Compiling without debug worked fine with the additional
#ifdefs. Compiling with DEBUG resulted in the _DEBUG
message and the Py_Debug message.
sys.path seems to be ok now in all configurations. One
of the (many) experimental changes to the build I did
must have bypassed the problem.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-09 07:51
Message:
Logged In: YES
user_id=21627
In your code that includes Python.h, can you please add the
lines
#ifdef _DEBUG
#error You must not define _DEBUG
#endif
#ifdef Py_DEBUG
#error For some reason Py_DEBUG is still defined
#endif
immediately after the
#include <Python.h>
line? Then please compile your library with DEBUG, and
report whether it compiles correctly; if not, please report
the errors it gives.
Furthermore, could you please evaluate the string
"import sys;print sys.path"
in your embedded code and report the exact output that this
gives when running with and without DEBUG?
Without the results of these specific experiments, I do not
think we can help much.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 07:16
Message:
Logged In: NO
I guess I'm not making myself very clear. Let me try again...
I have a library that completely encapsulates all use of
Python.h from my clients. I need to be able to debug this
library by compiling it with DEBUG. I do not want to use
the debug Python library. Unfortunately, just compiling
the Python headers with DEBUG seems to change the object
layout and require the use of the debug library.
BTW, the sys.path values include my PYTHONPATH values when
I link non-debug, and they are the default relative path
values when I link with debug. I know the code looks like
it doesn't ignore PYTHONPATH (I've looked through
getpathp.c too) but something is screwing up sys.path.
----------------------------------------------------------------------
Comment By: Mark Hammond (mhammond)
Date: 2001-11-08 17:09
Message:
Logged In: YES
user_id=14198
There is not much I can add here. I understand and
appreciate your concerns. However, as Tim and Martin have
stated, something is missing from the puzzle. PYTHONPATH
is *not* ignored in debug builds.
The reason for the _d is 2-fold:
* As Martin said, the object layout may change.
* The Python core and extensions sometimes share CRT
objects - notably file handles, and in some cases memory
blocks. It is critical that these come from the same CRT.
I could possibly agree that the "_d" convention is dumb and
should be dropped. However, it is unclear to me that this
bug can be simply summarised as this. This is probably
because it is unclear to me how this bug can be summarised
at all :)
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 15:28
Message:
Logged In: YES
user_id=21627
AFAICT, PYTHONPATH is *not* ignored when running a debug
build. If you think it is, please run the debug and the
nodebug interpreter (i.e. python.exe and python_d.exe), and
report the values of sys.path for either case.
More likely, the cause of your problems is that the import
code looks for different DLL name *in sys.path*. I.e. when
you do
import foo
in a nodebug build, it will try to find foo.pyd or foo.dll.
In a debug build, it will try to find foo_d.pyd or
foo_d.dll. So if you only have foo.pyd, then importing foo
will fail if Python was compiled with _DEBUG.
Again, this is by design: foo.pyd might crash the debug
interpreter, since the object layout is different.
If you merely want to single-step through the Python
runtime, without making use of the Py_DEBUG features, you
should build the release version of Python, but activate the
generation of debug symbols.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 12:03
Message:
Logged In: NO
You'll have to excuse the editorializing: I've found working
with the C API immensely frustrating in a variety of ways,
and then to have my empirical observations dismissed...
Let me step back and explain where I am right now.
I've built the 2.1 debug version of the python library using
the source distribution. When doing a final link without
debug (regardless of how all my client code was compiled)
I use the non-debug version of the Python library. When
doing a final link with debug (again, regardless of how all
of my client code is compiled) I use the debug version of
the python library. To get the latter scenario to work, I
edit the registry -- as well as the PYTHONPATH -- before
initializing python through the C API. For these two
cases, my unit tests run correctly to completion.
However, this means that our developers must compile the
library I wrote that wraps up the Python C API to match
how they are going to link it. I would like to find a way
around this constraint so that any combination of compiling
and linking my wrapper library works correctly.
(Let me also mention that I am aware that you are all
volunteers and I do appreciate your efforts. I am hopeful
that by resolving issues such as this Python can be made
more easy to use in a wider variety of circumstances.)
Thanks.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-11-08 11:49
Message:
Logged In: YES
user_id=31435
Anonymous, editorializing doesn't really help. I
understand that you want Python to work in a way other than
it does, and repeating that isn't necessary. I'm hoping
Mark Hammond will comment on that.
In the meantime, I'm trying to determine why you're seeing
the symptoms you are seeing. I believe you're seeing them,
but your analysis doesn't make sense, therefore something
is going on you haven't told us about. What *should*
happen in your ideal world isn't relevant to determining
what's actually happening. Take this one step at a time,
please.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:20
Message:
Logged In: NO
It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)
Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)
Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.
I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.
Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:19
Message:
Logged In: NO
It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)
Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)
Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.
I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.
Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-11-08 09:48
Message:
Logged In: YES
user_id=31435
Well, Martin is right, you know: if you're not defining
_DEBUG, and not letting MSVC define it either, then neither
will pyconfig.h define Py_DEBUG, and it gets harder and
harder to believe that what you're reporting is the full
truth.
Perhaps you could attach a tiny example, along with the
MSVC project file, or Makefile, you use. It may well be
academic, though, as it's *normal* for _DEBUG to get
defined under MSVC in a debug build (and whether or not you
add it explicitly -- as Martin implied, you would have had
to explicitly remove _DEBUG from the MSVC auto-generated
list of preprocesser debug-build #defines).
In answer to one of your earlier questions, PYTHONPATH is
not ignored after a debug link. I understand that you
believe it is, but I expect that's another illusion due to
something you haven't told us-- or not discovered --yet.
Perhaps you're linking with the wrong system libraries?
Python requires the multithreaded libraries.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:33
Message:
Logged In: YES
user_id=21627
What Python version are you using? You mentioned that you've
changed the pragma in config.h; atleast since Python 1.5.2,
this macro depends on _DEBUG, not on DEBUG. So if you didn't
define _DEBUG, you wouldn't need to change the pragma.Python
never changes its behaviour based on whether DEBUG is
defined, AFAICT.
Please check the project settings for the debug target in
your application again.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 09:22
Message:
Logged In: NO
The only debug-type symbol I am defining when compiling my
code is DEBUG. I'm not deliberately defining _DEBUG or
Py_Debug.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:09
Message:
Logged In: YES
user_id=21627
AFAICT, even on Windows, defining DEBUG does not change
anything whatsoever. It is only when you define _DEBUG that
things start to change.
I'm not sure whether this is a bug or a feature: If you just
want to enable debugging in your application, just don't
define _DEBUG. Notice that MSVC does that for you unless you
tell it not to. AFAICT, _DEBUG is what Microsoft uses in the
libraries to indicate "enable all debugging code that you
have". So Python enables Py_DEBUG.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-11-08 08:43
Message:
Logged In: YES
user_id=31435
Changed Category to Windows, since I'm almost certain
that's the missing info here.
Assigned to MarkH hoping he has a slick <wink> reply. This
kind of complaint comes up rarely but regularly, and I'm
never sure what to say: since I build Python from source,
and always did, our Windows debug scheme is fine by me.
But what are pure end-users expected to do?
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 06:10
Message:
Logged In: NO
I'm not recompiling the Python libraries. I am only
recompiling my personal libraries which use the Python
C API. I want to be able to compile my libraries with
debug and use the standard Python library.
This is not a "flawed" approach. It is extremely common
when using 3rd-party libraries to do this. Python should
be using a separate #define for its ref-counting debug
stuff so that clients can use the standard library and
still debug their own stuff.
Refer to my second comment regarding the use of PYTHONPATH
for another example of non-transparent substitution of
the debug and non-debug libraries.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2001-11-07 14:11
Message:
Logged In: YES
user_id=21627
I cannot understand what bug you are reporting. Of the
things you report, which of them did you not expect?
For example, that Py_RefTotal is undefined when you compile
with DEBUG, but link with the no-debug library is not a bug,
but by design: those functions are used in every debug
module, but available only in the debug library.
Your complete compilation approach is flawed: If you compile
with pydebug, you absolutely *must* link with the debug
library: the object layout of every object will change, so
crashes are likely if you link with the wrong library.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-05 09:29
Message:
Logged In: NO
More info: I've now narrowed it down to the fact that when
the final link is done with debug, the PYTHONPATH
environment variable is ignored altogether. In my case, I
cannot rely on the registry being set up for Python, and the
system is resorting to the default relative path names
(which, in my case, are of no use).
How does the behavior of the (non-debug) python library know
to change based on the final link? And why is PYTHONPATH
ignored if it is linked in debug?
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470