[Python-bugs-list] [ python-Bugs-478339 ] Linking/compiling with DEBUG

[noreply@sourceforge.net]

Bugs item #478339, was opened at 2001-11-05 07:45
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470

Category: Windows
Group: Python 2.1.1
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Mark Hammond (mhammond)
Summary: Linking/compiling with DEBUG

Initial Comment:
I have an embedded use of the Python interpreter. I
want to compile and link the encapsulating application with
and without DEBUG. I have disabled the use of the
pragma in config.h to always use the library specified
in the link statement, which is always the standard
python library. I have ensured that all invocations of
Python.h undefine the DEBUG symbol before including them.

Compiling the application without debug: works fine if
you link non-debug. However, linking with debug causes
some change in the way python modules are looked up and
I'm getting undefined local modules. I have checked the
PYTHONPATH environment variable and it is correct, and
the .py module exists in the correct directory. (This
is a common scenario where some libraries are compiled
without DEBUG, some with DEBUG, and then the whole
application linked DEBUG.)

Compiling the application with debug: Link errors
result. Undefined -- _Py_RefTotal, _Py_Dealloc,
_PyInitModule4TraceRefs.

Python is making too many hidden assumptions about the
build environment when it is being used embedded. This
is a critical problem as it makes it impossible to
build the encapsulating application for debug purposes.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 11:30

Message:
Logged In: NO 

Turns out the _DEBUG is defined when using the /MDd
compiler option.

We have painstakingly worked out our build system to
deal with the library conflicts you mention. If I
understand what you're saying, these are the rules:

Linking with debug CRT -> link with python_d
Linking with python_d -> compile client code with DEBUG

The first rule is thanks to MS. The second rule is to
get internal object layouts to match up.

Is this right?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-09 10:17

Message:
Logged In: YES 
user_id=21627

I'm glad the PYTHONPATH issue is resolved.

On the _DEBUG issue: Somebody *must* be defining _DEBUG
somewhere, or else the #ifdef would not trigger. Please open
the MSVC++ project file of your project (the .dsp file) in a
text editor, and search for _DEBUG. 

If you cannot find it there, please try to activate full
command line printing in your build process, so that you see
what options are passed to the compiler. 

If you are absolutely sure that _DEBUG is not passed in the
command line, it must be defined in one of the header files.
In this case, please put the #ifdef _DEBUG block of the
previous test after each #include, e.g. in a form

#include <stdio.h>
#ifdef _DEBUG
#error After stdio.h
#endif

#include <nextfile.h>
#ifdef _DEBUG
#error After nextfile.h
#endif

I'm sure there are better ways to find out where _DEBUG is
defined; don't hesitate to apply them if you are aware of any.

Once you found the location of _DEBUG, you may remove it to
be able to use the standard Python .DLL. 

Please make sure you understand Mark's comment on CRTs,
though: Many problems will occur if you mix the debug and
the nodebug version of the MSVC libraries (e.g. msvcrt.dll
and msvcrtd.dll). Please see the Microsoft documentation for
details: If you use pythonxy.dll, you absolutely *must* make
sure that you do not link the debug versions of the CRT,
even when performing a debug build. Unfortunately, you won't
get a compilation error if you do not follow this guideline;
instead, very hard-to-debug problems will occur. You can use
depends.exe on the final executable to analyse the set of
libraries used.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 08:16

Message:
Logged In: NO 

Compiling without debug worked fine with the additional
#ifdefs. Compiling with DEBUG resulted in the _DEBUG
message and the Py_Debug message.


sys.path seems to be ok now in all configurations. One
of the (many) experimental changes to the build I did
must have bypassed the problem.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-09 07:51

Message:
Logged In: YES 
user_id=21627

In your code that includes Python.h, can you please add the
lines

#ifdef _DEBUG
#error You must not define _DEBUG
#endif
#ifdef Py_DEBUG
#error For some reason Py_DEBUG is still defined
#endif

immediately after the
#include <Python.h>

line? Then please compile your library with DEBUG, and
report whether it compiles correctly; if not, please report
the errors it gives.

Furthermore, could you please evaluate the string

"import sys;print sys.path"

in your embedded code and report the exact output that this
gives when running with and without DEBUG?

Without the results of these specific experiments, I do not
think we can help much.


----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-09 07:16

Message:
Logged In: NO 

I guess I'm not making myself very clear. Let me try again...

I have a library that completely encapsulates all use of
Python.h from my clients. I need to be able to debug this
library by compiling it with DEBUG. I do not want to use
the debug Python library. Unfortunately, just compiling
the Python headers with DEBUG seems to change the object
layout and require the use of the debug library.

BTW, the sys.path values include my PYTHONPATH values when
I link non-debug, and they are the default relative path
values when I link with debug. I know the code looks like
it doesn't ignore PYTHONPATH (I've looked through
getpathp.c too) but something is screwing up sys.path.

----------------------------------------------------------------------

Comment By: Mark Hammond (mhammond)
Date: 2001-11-08 17:09

Message:
Logged In: YES 
user_id=14198

There is not much I can add here.  I understand and 
appreciate your concerns.  However, as Tim and Martin have 
stated, something is missing from the puzzle.  PYTHONPATH 
is *not* ignored in debug builds.

The reason for the _d is 2-fold:
* As Martin said, the object layout may change.
* The Python core and extensions sometimes share CRT 
objects - notably file handles, and in some cases memory 
blocks.  It is critical that these come from the same CRT.

I could possibly agree that the "_d" convention is dumb and 
should be dropped.  However, it is unclear to me that this 
bug can be simply summarised as this.  This is probably 
because it is unclear to me how this bug can be summarised 
at all :)

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 15:28

Message:
Logged In: YES 
user_id=21627

AFAICT, PYTHONPATH is *not* ignored when running a debug
build. If you think it is, please run the debug and the
nodebug interpreter (i.e. python.exe and python_d.exe), and
report the values of sys.path for either case.

More likely, the cause of your problems is that the import
code looks for different DLL name *in sys.path*. I.e. when
you do 
import foo
in a nodebug build, it will try to find foo.pyd or foo.dll.
In a debug build, it will try to find foo_d.pyd or
foo_d.dll. So if you only have foo.pyd, then importing foo
will fail if Python was compiled with _DEBUG. 

Again, this is by design: foo.pyd might crash the debug
interpreter, since the object layout is different.

If you merely want to single-step through the Python
runtime, without making use of the Py_DEBUG features, you
should build the release version of Python, but activate the
generation of debug symbols.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 12:03

Message:
Logged In: NO 

You'll have to excuse the editorializing: I've found working
with the C API immensely frustrating in a variety of ways,
and then to have my empirical observations dismissed...

Let me step back and explain where I am right now.

I've built the 2.1 debug version of the python library using
the source distribution. When doing a final link without
debug (regardless of how all my client code was compiled)
I use the non-debug version of the Python library. When
doing a final link with debug (again, regardless of how all
of my client code is compiled) I use the debug version of
the python library. To get the latter scenario to work, I
edit the registry -- as well as the PYTHONPATH -- before
initializing python through the C API. For these two
cases, my unit tests run correctly to completion.

However, this means that our developers must compile the
library I wrote that wraps up the Python C API to match
how they are going to link it. I would like to find a way
around this constraint so that any combination of compiling
and linking my wrapper library works correctly.

(Let me also mention that I am aware that you are all
volunteers and I do appreciate your efforts. I am hopeful
that by resolving issues such as this Python can be made
more easy to use in a wider variety of circumstances.)

Thanks.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 11:49

Message:
Logged In: YES 
user_id=31435

Anonymous, editorializing doesn't really help.  I 
understand that you want Python to work in a way other than 
it does, and repeating that isn't necessary.  I'm hoping 
Mark Hammond will comment on that.

In the meantime, I'm trying to determine why you're seeing 
the symptoms you are seeing.  I believe you're seeing them, 
but your analysis doesn't make sense, therefore something 
is going on you haven't told us about.  What *should* 
happen in your ideal world isn't relevant to determining 
what's actually happening.  Take this one step at a time, 
please.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:20

Message:
Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).


----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:19

Message:
Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 09:48

Message:
Logged In: YES 
user_id=31435

Well, Martin is right, you know:  if you're not defining 
_DEBUG, and not letting MSVC define it either, then neither 
will pyconfig.h define Py_DEBUG, and it gets harder and 
harder to believe that what you're reporting is the full 
truth.

Perhaps you could attach a tiny example, along with the 
MSVC project file, or Makefile, you use.  It may well be 
academic, though, as it's *normal* for _DEBUG to get 
defined under MSVC in a debug build (and whether or not you 
add it explicitly -- as Martin implied, you would have had 
to explicitly remove _DEBUG from the MSVC auto-generated 
list of preprocesser debug-build #defines).

In answer to one of your earlier questions, PYTHONPATH is 
not ignored after a debug link.  I understand that you 
believe it is, but I expect that's another illusion due to 
something you haven't told us-- or not discovered --yet.  
Perhaps you're linking with the wrong system libraries?  
Python requires the multithreaded libraries.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:33

Message:
Logged In: YES 
user_id=21627

What Python version are you using? You mentioned that you've
changed the pragma in config.h; atleast since Python 1.5.2,
this macro depends on _DEBUG, not on DEBUG. So if you didn't
define _DEBUG, you wouldn't need to change the pragma.Python
never changes its behaviour based on whether DEBUG is
defined, AFAICT.

Please check the project settings for the debug target in
your application again.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 09:22

Message:
Logged In: NO 

The only debug-type symbol I am defining when compiling my
code is DEBUG. I'm not deliberately defining _DEBUG or
Py_Debug.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:09

Message:
Logged In: YES 
user_id=21627

AFAICT, even on Windows, defining DEBUG does not change
anything whatsoever. It is only when you define _DEBUG that
things start to change.

I'm not sure whether this is a bug or a feature: If you just
want to enable debugging in your application, just don't
define _DEBUG. Notice that MSVC does that for you unless you
tell it not to. AFAICT, _DEBUG is what Microsoft uses in the
libraries to indicate "enable all debugging code that you
have". So Python enables Py_DEBUG.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 08:43

Message:
Logged In: YES 
user_id=31435

Changed Category to Windows, since I'm almost certain 
that's the missing info here.

Assigned to MarkH hoping he has a slick <wink> reply.  This 
kind of complaint comes up rarely but regularly, and I'm 
never sure what to say:  since I build Python from source, 
and always did, our Windows debug scheme is fine by me.  
But what are pure end-users expected to do?

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 06:10

Message:
Logged In: NO 

I'm not recompiling the Python libraries. I am only
recompiling my personal libraries which use the Python
C API. I want to be able to compile my libraries with
debug and use the standard Python library.

This is not a "flawed" approach. It is extremely common
when using 3rd-party libraries to do this. Python should
be using a separate #define for its ref-counting debug
stuff so that clients can use the standard library and
still debug their own stuff.

Refer to my second comment regarding the use of PYTHONPATH
for another example of non-transparent substitution of
the debug and non-debug libraries.


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-07 14:11

Message:
Logged In: YES 
user_id=21627

I cannot understand what bug you are reporting. Of the
things you report, which of them did you not expect?

For example, that Py_RefTotal is undefined when you compile
with DEBUG, but link with the no-debug library is not a bug,
but by design: those functions are used in every debug
module, but available only in the debug library.

Your complete compilation approach is flawed: If you compile
with pydebug, you absolutely *must* link with the debug
library: the object layout of every object will change, so
crashes are likely if you link with the wrong library.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-05 09:29

Message:
Logged In: NO 

More info: I've now narrowed it down to the fact that when
the final link is done with debug, the PYTHONPATH
environment variable is ignored altogether. In my case, I
cannot rely on the registry being set up for Python, and the
system is resorting to the default relative path names
(which, in my case, are of no use).

How does the behavior of the (non-debug) python library know
to change based on the final link? And why is PYTHONPATH
ignored if it is linked in debug?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470