[Python-Dev] Windows x64 & bsddb 4.4.20 woes

[Gregory P. Smith]

On 3/13/08, Trent Nelson <tnelson at onresolve.com> wrote:
>
> Hey Greg,
>
> I'm at PyCon indeed, staying through the sprints 'til next Thursday.  I'll
> drop you a note offline re catching up.
>
> The other query I had was whether or not I should try a later version of
> BerkeleyDB -- are we committed to 4.4.20 (or 4.4.x?) for 2.6/3.0 or is it
> worth investigating newer versions?  Martin/Jesus, any thoughts on this?
>

Python 2.6/3.0 should be built on Windows using BerkeleyDB 4.5.x for now.
4.6.x is out but has some bugs on some platforms so i don't recommend
shipping our release using it; 4.7.x is in beta and some bugs are being
worked on; if its out and shows no signs of obvious issues before the 2.6/3.0
beta period is over I recommend we build our binary releases using it.
Otherwise 4.5 it will be.  There is no reason to use 4.4.x.

Regarding the db_static build and conflicting compile/link options -- I'm
> going to bring the db_static source directly into the _bsddb project (for
> now) which should make this a lot easier to debug.
>
>     Trent.
>
>
> From: Gregory P. Smith [greg at krypto.org]
> Sent: 13 March 2008 22:00
> To: Trent Nelson
> Cc: python-dev at python.org; Jesus Cea
> Subject: Re: Windows x64 & bsddb 4.4.20 woes
>
>
>
> I haven't built the bsddb stuff on windows myself in a few years and have
> never had access to a windows x64 system so I'm no silver bullet.  Making
> the BerkeleyDB compile and link options match with those of python is the
> first place I'd start.  Also you should be able to make a debug build of
> BerkeleyDB (though it sounds like you may have tried that already?).  Next
> off in the debugging i'd take a look at the assembly to see what exactly it
> was failing to do.
>
> If you're at PyCon right now we should meet up and try to figure it out (I
> just arrived).
>
>
> On 3/13/08, Trent Nelson <tnelson at onresolve.com> wrote:
> I've been trying to give the Windows x64 builds a bit of TLC the past few
> evenings.  I managed to get a successful build with all external modules
> last night (Tcl/Tk required about a half a dozen code/configuration changes
> each in order to build in a Windows x64 environment with Visual Studio 9,
> I'll deal with that in a separate thread or roundup issue).
>
> Unfortunately, though, we're back to more bsddb issues.  I got about 15
> tests in without error before test_whichdb ran, which results in the
> following being called in dbhash.py:
>
>         return bsddb.hashopen(file, flag, mode)
>
> I can trace that call to DBEnv_open() in _bsddb.c:
>
> static PyObject*
> DBEnv_open(DBEnvObject* self, PyObject* args)
> {
>     int err, flags=0, mode=0660;
>     char *db_home;
>
>     if (!PyArg_ParseTuple(args, "z|ii:open", &db_home, &flags, &mode))
>         return NULL;
>
>     CHECK_ENV_NOT_CLOSED(self);
>
>     MYDB_BEGIN_ALLOW_THREADS;
>     err = self->db_env->open(self->db_env, db_home, flags, mode);
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Placing a breakpoint at the line above and stepping in results in Visual
> Studio reporting: " A buffer overrun has occurred in python_d.exe which has
> corrupted the program's internal state. Press Break to debug the program or
> Continue to terminate the program.".  FWIW, the exception is being raised as
> part of the /GS buffer overflow checks (implemented in gs_result.c, which is
> provided in my VS9 installation).
>
> This has been annoyingly awkward to debug.  I can't break down that call
> into multiple steps in order to try place breakpoints in the db_static
> module.  The callstack isn't that useful either:
>
> _bsddb_d.pyd!__crt_debugger_hook()
> _bsddb_d.pyd!__report_gsfailure(unsigned __int64 StackCookie=2211040)
> _bsddb_d.pyd!__GSHandlerCheckCommon(void *
> EstablisherFrame=0x000000000021bce0, ...)
> _bsddb_d.pyd!__GSHandlerCheck(_EXCEPTION_RECORD *
> ExceptionRecord=0x000000000021bbc0, ...)
> ntdll.dll!00000000773ae13d()
> [Frames below may be incorrect and/or missing, no symbols loaded for
> ntdll.dll]
> ntdll.dll!00000000773aea57()
> ntdll.dll!00000000773b59f8()
> _bsddb_d.pyd!__os_strdup()  + 0x18 bytes
> _bsddb_d.pyd!__os_tmpdir()  + 0x281 bytes
>
> You'd think placing breakpoints in db 4.4.20's __os_strdup and __os_tmpdir
> methods would do something, but alas, the bufferoverflow exception is raised
> before any breakpoints are set.  This makes me suspect there's something
> funky going on with the entire build and linking of db_static (VS should
> honour those breakpoints if the code is being entered, I even added
> db_static to pcbuild.sln and rebuilt but no dice).  I've noticed that
> they're not using consistent compiler flags by default (we use /GS, they use
> /GS-, we allow function level linking, they don't -- note that I did change
> db_static's options to align with _bsddb's but the bufferoverflow exception
> is still being thrown).
>
> Greg, Jesús, I'm CC'ing you guys as stfw'ing seems to bring back you two
> the most when it comes to bsddb issues.  I've still got a list of things to
> try with regarding to debugging this x64 issue, but I wanted to reach out
> now to see if anyone else had encountered it before.  Has bsddb ever been
> built successfully on Win64 and passed all tests or am I venturing into new
> ground?
>
> Martin, you've changed externals/bsddb-4.4.20 with regards to x64 builds
> recently -- have you been able to get things working in your x64
> environments?
>
> Regards,
>
>
>         Trent.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20080313/6b6ff2b3/attachment-0001.htm