Please give this patch for building bsddb a try
If you build the bsddb module on a Unix-like system (that is, you use configure and setup.py to build the interpreter and it attempts to build the bsddb module), please give the new patch attached to
a try. Ignore the subject of the patch. I just tacked my patch onto this item and assigned it to myself. If/when the issue is settled I'll track down and close other patches and bug reports related to building the bsddb module.
Briefly, it attempts the following:
1. Makes it inconvenient (though certainly not impossible) to build/link with version 1 of the Berkeley DB library by commenting out the relevant part of the db_try_this dictionary in setup.py.
2. Links the search for a DB library and corresponding include files so you don't find a version 2 include file and a version 3 library (for example).
3. Attempts to do the same for the dbm module when it decides to link with the Berkeley DB library for compatibility (this is stuff under "development" and will almost certainly require further changes). (You can ignore the debug print I forgot to remove before creating the patch. ;-)
I asked on c.l.py about where people have the Berkeley DB stuff installed so I could tune the locations listed in db_try_this, but the thread almost immediately went off into the weeds arguing about berkdb license issues. I therefore humbly request your more rational input on this topic. If you have a Unix-ish system and Berkeley DB is installed somewhere not listed in the db_try_this dictionary in setup.py, please let me know.
Thx,
Skip
Hello!
On Tue, Jun 11, 2002 at 11:22:08AM -0500, Skip Montanaro wrote:
http://python.org/sf/553108
- Makes it inconvenient (though certainly not impossible) to build/link with version 1 of the Berkeley DB library by commenting out the relevant part of the db_try_this dictionary in setup.py.
Can I have two different modules simultaneously? For example, a module linked with db.1.85 plus a module linked with db3.
- Links the search for a DB library and corresponding include files so you don't find a version 2 include file and a version 3 library (for example).
After compiling bsddb-3.2 from sources I have got /usr/local/BerkeleyDB.3.2/ directory, with lib/include being its subdirectories. The patch didn't look into this, as I understand it.
Oleg.
>> 1. Makes it inconvenient (though certainly not impossible) to >> build/link with version 1 of the Berkeley DB library by commenting >> out the relevant part of the db_try_this dictionary in setup.py.
Oleg> Can I have two different modules simultaneously? For example, a Oleg> module linked with db.1.85 plus a module linked with db3.
Nope. I don't believe you can do that today (at least not without some build-time gymnastics), and I have no plans to support that. For one thing, you'd have to compile and link bsddmodule.c twice. To allow multiple versions to be loaded into the interpreter you'd also have to name them differently. This would require source code changes to keep global symbols (at least the module init functions) from clashing.
>> 2. Links the search for a DB library and corresponding include files >> so you don't find a version 2 include file and a version 3 library >> (for example).
Oleg> After compiling bsddb-3.2 from sources I have got Oleg> /usr/local/BerkeleyDB.3.2/ directory, with lib/include being its Oleg> subdirectories. The patch didn't look into this, as I understand Oleg> it.
Thanks, I'll add that. I also notice that /usr/local/BerkeleyDB.4.0 is the default install directory for the 4.0 source.
Skip
Skip Montanaro skip@pobox.com writes:
This would require source code changes to keep global symbols (at least the module init functions) from clashing.
It actually only requires different init functions. To support that with distutils, you need to tell distutils to generate different object files from the same source file, which is probably not supported out of the box.
Regards, Martin
On Tue, Jun 11, 2002 at 09:00:24PM +0200, Martin v. Loewis wrote:
Skip Montanaro skip@pobox.com writes:
This would require source code changes to keep global symbols (at least the module init functions) from clashing.
It actually only requires different init functions. To support that with distutils, you need to tell distutils to generate different object files from the same source file, which is probably not supported out of the box.
I know. Once I thought about sed/awk magic to generate two different modules from one template.
Oleg.
* Oleg Broytmann phd@phd.pp.ru [2002-06-11 23:48 +0400]:
On Tue, Jun 11, 2002 at 09:00:24PM +0200, Martin v. Loewis wrote:
Skip Montanaro skip@pobox.com writes:
This would require source code changes to keep global symbols (at least the module init functions) from clashing.
It actually only requires different init functions. To support that with distutils, you need to tell distutils to generate different object files from the same source file, which is probably not supported out of the box.
I know. Once I thought about sed/awk magic to generate two different modules from one template.
What about symlinks, like:
bsd18module.c -> bsd30module.c bsd30module.c
and using a few #ifdefs in the C sources?
Gerhard
Gerhard Häring haering_python@gmx.de writes:
What about symlinks, like:
That can't work on Windows.
Regards, Martin
I know. Once I thought about sed/awk magic to generate two different modules from one template.
What about symlinks, like:
bsd18module.c -> bsd30module.c bsd30module.c
and using a few #ifdefs in the C sources?
Instead of symlinks, how about one .h file containing most of the code and two or three .c files that set a few #defines and then #include the .h file? Similar to what we do for Python/threads.c
--Guido van Rossum (home page: http://www.python.org/~guido/)
>> This would require source code changes to keep global symbols (at >> least the module init functions) from clashing.
Martin> It actually only requires different init functions. To support Martin> that with distutils, you need to tell distutils to generate Martin> different object files from the same source file, which is Martin> probably not supported out of the box.
Thanks for the clarification Martin. Even though this seems possible with minimal changes to the source, I still think supporting this is not worth it.
Oleg, at your end I suspect you could fairly easily copy a bit of code in setup.py, copy bsddbmodule.c to bsddb1module.c, and rename the module init function.
Skip
On 11 June 2002, Martin v. Loewis said:
It actually only requires different init functions. To support that with distutils, you need to tell distutils to generate different object files from the same source file, which is probably not supported out of the box.
In theory:
setup(... ext_modules=[Extension("foo1", ["foo.c"], define_macros=[('FOOSTYLE', 1)]), Extension("foo2", ["foo.c"], define_macros=[('FOOSTYLE', 2)])])
should work. See http://www.python.org/doc/current/dist/setup-script.html#SECTION000330000000...
Untested, YMMV, etc.
Greg
From: "Greg Ward" gward@python.net
On 11 June 2002, Martin v. Loewis said:
It actually only requires different init functions. To support that with distutils, you need to tell distutils to generate different object files from the same source file, which is probably not supported out of the box.
In theory:
setup(... ext_modules=[Extension("foo1", ["foo.c"], define_macros=[('FOOSTYLE', 1)]), Extension("foo2", ["foo.c"], define_macros=[('FOOSTYLE', 2)])])
should work.
But not in practice, IIRC.
Because the build process for foo2 will see that foo.o is up to date already.
Thomas
"OB" == Oleg Broytmann phd@phd.pp.ru writes:
OB> Can I have two different modules simultaneously? For OB> example, a module linked with db.1.85 plus a module linked OB> with db3.
I still think we may want to pull PyBSDDB into the standard distro, as a way to provide BDB api's > 1.85. The question is, what would this new module be called? I dislike "bsddb3" -- which I think PyBSDDB itself uses -- because it links against BDB 4.0.
OTOH, PyBSDDB seems to be quite solid, so I think it's mature enough to migrate into the Python distro. I'm cc'ing pybsddb-users for feedback.
-Barry
barry@zope.com (Barry A. Warsaw) writes:
I still think we may want to pull PyBSDDB into the standard distro, as a way to provide BDB api's > 1.85. The question is, what would this new module be called? I dislike "bsddb3" -- which I think PyBSDDB itself uses -- because it links against BDB 4.0.
If this is just a question of naming, I recommend bsddb2 - not indicating the version of the database, but the version of the Python module.
Regards, Martin
On Wed, Jun 19, 2002 at 07:30:11AM +0200, Martin v. Loewis wrote:
barry@zope.com (Barry A. Warsaw) writes:
I still think we may want to pull PyBSDDB into the standard distro, as a way to provide BDB api's > 1.85. The question is, what would this new module be called? I dislike "bsddb3" -- which I think PyBSDDB itself uses -- because it links against BDB 4.0.
If this is just a question of naming, I recommend bsddb2 - not indicating the version of the database, but the version of the Python module.
If I hadn't made the initial mistake of naming pybsddb's module bsddb3 when i first extended robin's berkeleydb 2.x module to work with 3.0 I would agree with that name. I worry that having a module named bsddb2 might cause endless confusion as bsddb and bsddb3 already exist and did correlate to the version number. How about 'berkeleydb'?
"Gregory P. Smith" greg@electricrain.com writes:
If I hadn't made the initial mistake of naming pybsddb's module bsddb3 when i first extended robin's berkeleydb 2.x module to work with 3.0 I would agree with that name. I worry that having a module named bsddb2 might cause endless confusion as bsddb and bsddb3 already exist and did correlate to the version number. How about 'berkeleydb'?
What does it have to do with the city of Berkeley (CA)? Perhaps "sleepycat"?
Regards, Martin
What does it have to do with the city of Berkeley (CA)? Perhaps "sleepycat"?
The company Sleepycat calls this particular product Berkeley DB, that's enough reason for me. They (may) have other products too, so Sleepycat is not sufficiently distinctive.
--Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, Jun 19, 2002 at 10:43:01PM +0200, Martin v. Loewis wrote:
What does it have to do with the city of Berkeley (CA)? Perhaps "sleepycat"?
from sleepycat import berkeleydb from sleepycat import bsddb2 from sleepycat import bsddb3 from sleepycat import bsddb4
???
Oleg.
I still think we may want to pull PyBSDDB into the standard distro, as a way to provide BDB api's > 1.85. The question is, what would this new module be called? I dislike "bsddb3" -- which I think PyBSDDB itself uses -- because it links against BDB 4.0.
Good idea. Maybe call it berkeleydb? That's what Sleepycat calls it (there's no connection with the BSD Unix distribution AFAICT).
--Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, Jun 19, 2002 at 08:39:37AM -0400, Guido van Rossum wrote:
Good idea. Maybe call it berkeleydb? That's what Sleepycat calls it (there's no connection with the BSD Unix distribution AFAICT).
+1 on berkeleydb
Oleg.
"GvR" == Guido van Rossum guido@python.org writes:
GvR> Maybe call it berkeleydb?
"GPS" == Gregory P Smith greg@electricrain.com writes:
GPS> How about 'berkeleydb'?
Sounds like consensus. Greg, how do you feel about moving project management from the pybsddb project to the Python project?
Maybe we should plan a transition on the pybsddb-users list. I'm willing to help.
-Barry
On Wed, Jun 19, 2002 at 02:44:34PM -0400, Barry A. Warsaw wrote:
"GvR" == Guido van Rossum guido@python.org writes:
GvR> Maybe call it berkeleydb?"GPS" == Gregory P Smith greg@electricrain.com writes:
GPS> How about 'berkeleydb'?Sounds like consensus. Greg, how do you feel about moving project management from the pybsddb project to the Python project?
Maybe we should plan a transition on the pybsddb-users list. I'm willing to help.
That sounds like a good idea to me, though I don't know what moving it entails. (i assume creating a berkeleydb module directory in the python project and maintaining the code and documentation from there?).
Technically Robin Dunn is the only project administrator on the pybsddb sourceforge project but i've got access to at least modify pybsddb.sf.net, cvs and file releases. The project has been relatively idle recently; i've tried to give it a little of my time every month or two (basic 4.0 support, some bugfixes, accepting patches, etc). As it is quite stable I don't believe he's actively working on it anymore. Robin?
-G
BAW> I still think we may want to pull PyBSDDB into the standard distro, BAW> as a way to provide BDB api's > 1.85. The question is, what would BAW> this new module be called? I dislike "bsddb3" -- which I think BAW> PyBSDDB itself uses -- because it links against BDB 4.0.
Guido> Good idea. Maybe call it berkeleydb? That's what Sleepycat Guido> calls it (there's no connection with the BSD Unix distribution Guido> AFAICT).
Why can't it just be called bsddb? As far as I could tell tell, it provides a bsddb-compatible interface at the module level. The only change at the bsddb level is the addition of an extra object (db? I can't recall right now and have to get offline soon for the credit card machine so I can't pause to check ;-) which gives the programmer access to all the PyBSDDB magic.
Skip
On Wed, Jun 19, 2002 at 04:15:16PM -0500, Skip Montanaro wrote:
BAW> I still think we may want to pull PyBSDDB into the standard distro, BAW> as a way to provide BDB api's > 1.85. The question is, what would BAW> this new module be called? I dislike "bsddb3" -- which I think BAW> PyBSDDB itself uses -- because it links against BDB 4.0. Guido> Good idea. Maybe call it berkeleydb? That's what Sleepycat Guido> calls it (there's no connection with the BSD Unix distribution Guido> AFAICT).Why can't it just be called bsddb? As far as I could tell tell, it provides a bsddb-compatible interface at the module level. The only change at the bsddb level is the addition of an extra object (db? I can't recall right now and have to get offline soon for the credit card machine so I can't pause to check ;-) which gives the programmer access to all the PyBSDDB magic.
Skip
Modern berkeleydb uses much different on disk database formats, glancing at the docs on sleepycat.com i don't even think it can read bsddb (1.85) files. Existing code using bsddb (1.85) should not automatically start using a different database library even if we provide a compatibility interface. That upgrade can be done to code manually using:
import berkeleydb bsddb = berkeleydb
(and creating a single bsddb module that used the old 1.85 library for the old interface and the 3.3/4.0 library for the modern interface would add bloat to many applications that don't need both if it were even possible to link that in such a way as to avoid the symbol conflicts)
Greg
>> Why can't it just be called bsddb?
Greg> Modern berkeleydb uses much different on disk database formats, Greg> glancing at the docs on sleepycat.com i don't even think it can Greg> read bsddb (1.85) files.
That's never stopped us before. ;-) The current bsddb module works with versions 1, 2, 3, and 4 of Berkeley DB using the 1.85-compatible API that Sleepycat provides. It's always been the user's responsibility to run the appropriate db_dump or db_dump185 commands before using the next version of Berkeley DB. Using the library from Python never removed that requirement.
Skip
On Wed, Jun 19, 2002 at 05:27:06PM -0500, Skip Montanaro wrote:
>> Why can't it just be called bsddb? Greg> Modern berkeleydb uses much different on disk database formats, Greg> glancing at the docs on sleepycat.com i don't even think it can Greg> read bsddb (1.85) files.That's never stopped us before. ;-) The current bsddb module works with versions 1, 2, 3, and 4 of Berkeley DB using the 1.85-compatible API that Sleepycat provides. It's always been the user's responsibility to run the appropriate db_dump or db_dump185 commands before using the next version of Berkeley DB. Using the library from Python never removed that requirement.
Good point. I was ignorant of the original bsddb 1.85 module workings as i never used it. Pybsddb backwards compatibility was implemented (not be me) by with the intention that it could be used as a replacement for the existing bsddb module. It passes the simplistic test_bsddb.py that is included with python today as well as pybsddb's own test_compat.py to test the compatibility layer.
If we replace the existing bsddb with pybsddb (bsddb3), it should work. If there are hidden bugs that's what the alpha/beta periods are for. However linking against berkeleydb versions less than 3.2 will no longer be supported; should we keep the existing bsddb around as oldbsddb for users in that situation?
-G
"Gregory P. Smith" greg@electricrain.com writes:
However linking against berkeleydb versions less than 3.2 will no longer be supported; should we keep the existing bsddb around as oldbsddb for users in that situation?
I don't think so; users could always extract the module from older distributions if they want to.
Instead, if there are complaints, I think we should try to extend source support a little further back.
Regards, Martin
Greg> should we keep the existing bsddb around as oldbsddb for users in Greg> that situation?
Martin> I don't think so; users could always extract the module from Martin> older distributions if they want to.
I would prefer the old version be moved to lib-old (or Modules-old?). For people still running DB 2.x it shouldn't be a major headache to retrieve.
Skip
On Fri, Jun 21, 2002 at 09:26:35AM -0500, Skip Montanaro wrote:
Greg> should we keep the existing bsddb around as oldbsddb for users in Greg> that situation? Martin> I don't think so; users could always extract the module from Martin> older distributions if they want to.I would prefer the old version be moved to lib-old (or Modules-old?). For people still running DB 2.x it shouldn't be a major headache to retrieve.
This sounds good. Here's what i see on the plate to be done so far:
1) move the existing Modules/bsddbmodule.c to a new Modules-old or directory. 2) create a new Lib/bsddb directory containing bsddb3/bsddb3/*.py from the pybsddb project. 3) create a new Modules/bsddb directory containing bsddb3/src/* from the pybsddb project (the files should probably be renamed to _bsddbmodule.c and bsddbmoduleversion.h for consistent naming) 4) place the pybsddb setup.py in the Modules/bsddb directory, modifying it as needed. OR modify the top level setup.py to understand how to build the pybsddb module. (there is code in pybsddb's setup.py to locate the berkeleydb install and determine appropriate flags that should be cleaned up and carried on) 5) modify the top level python setup.py to build the bsddb module as appropriate. 6) "everything else" including integrating documentation and pybsddb's large test suite.
Sound correct?
How do we want future bsddb module development to proceed? I envision it either taking place 100% under the python project, or taking place as it is now in the pybsddb project with patches being fed to the python project as desired? Any preferences? [i prefer to not maintain the code in two places myself (ie: do it all under the python project)]
Greg
"Gregory P. Smith" greg@electricrain.com writes:
Sound correct?
Yes, please go ahead.
How do we want future bsddb module development to proceed? I envision it either taking place 100% under the python project, or taking place as it is now in the pybsddb project with patches being fed to the python project as desired? Any preferences? [i prefer to not maintain the code in two places myself (ie: do it all under the python project)]
It's your choice. If people want to maintain pybsddb3 for older Python releases, it would be necessary to synchronize the two code bases regularly. That would be the task of whoever is interested in providing older Python releases with newer code. From the viewpoint of the Python distribution, this support is not interesting.
Regards, Martin
"GPS" == Gregory P Smith greg@electricrain.com writes:
GPS> This sounds good. Here's what i see on the plate to be done GPS> so far:
GPS> 1) move the existing Modules/bsddbmodule.c to a new GPS> Modules-old or directory.
mkdir Modules/old (or Modules/extensions-old) mv Modules/bsddbmodule.c Modules/old
GPS> 2) create a new Lib/bsddb GPS> directory containing bsddb3/bsddb3/*.py from the pybsddb GPS> project.
+1
GPS> 3) create a new Modules/bsddb directory containing GPS> bsddb3/src/* from the pybsddb project (the files should GPS> probably be renamed to _bsddbmodule.c and GPS> bsddbmoduleversion.h for consistent naming)
I don't think you need to create a new directory under Modules for this; it's just two files. Probably Modules/_bsddbmodule.c and Modules/_bsddbversion.h are fine.
Also, for backwards compatibility, won't Lib/bsddb/__init__.py need to do "from _bsddb import btopen, error, hashopen, rnopen"?
GPS> 4) place the pybsddb setup.py in the Modules/bsddb directory, GPS> modifying it as needed. OR modify the top level setup.py GPS> to understand how to build the pybsddb module. (there is GPS> code in pybsddb's setup.py to locate the berkeleydb install GPS> and determine appropriate flags that should be cleaned up and GPS> carried on)
How much of Skip's recent changes to setup.py can be retargeted for pybsddb?
GPS> 5) modify the top level python setup.py to build the bsddb GPS> module as appropriate. 6) "everything else" including GPS> integrating documentation and pybsddb's large test suite.
What to do about the test suite is a good question. pybsddb's is /much/ more extensive, and I wouldn't want to lose that, but I'm also not sure I'd want it to run during a normal regrtest.
Here's an idea: leave test_bsddb as is and add pybsddb's as test_all_bsddb.py. Then add a "-u all_bsddb" to regrtest's resource flags.
GPS> How do we want future bsddb module development to proceed? I GPS> envision it either taking place 100% under the python GPS> project, or taking place as it is now in the pybsddb project GPS> with patches being fed to the python project as desired? Any GPS> preferences? [i prefer to not maintain the code in two GPS> places myself (ie: do it all under the python project)]
I'd like to see one more official release from the pybsddb project, since its cvs has some very useful additions (important bug fixes plus support for BerkeleyDB 4). Then move all development over to the Python project and let interested volunteers port critical patches back to the pybsddb project. If you add me as a developer on pybsddb.sf.net, I'll volunteer to help.
-Barry
"SM" == Skip Montanaro skip@pobox.com writes:
Greg> should we keep the existing bsddb around as oldbsddb for Greg> users in that situation?
Martin> I don't think so; users could always extract the module Martin> from older distributions if they want to.
SM> I would prefer the old version be moved to lib-old (or SM> Modules-old?). For people still running DB 2.x it shouldn't SM> be a major headache to retrieve.
Modules/old/ probably. We wouldn't do anything with that directory except use it as a placeholder for old extension source, right?
Do we care about preserving the cvs history for the current bsddbmodule.c? If so, we'll have to ask SF to do a cvs dance for us. It may not be worth it.
-Barry
SM> I would prefer the old version be moved to lib-old (or SM> Modules-old?). For people still running DB 2.x it shouldn't be a SM> major headache to retrieve.
BAW> Modules/old/ probably. We wouldn't do anything with that directory BAW> except use it as a placeholder for old extension source, right?
Sounds good to me.
BAW> Do we care about preserving the cvs history for the current BAW> bsddbmodule.c? If so, we'll have to ask SF to do a cvs dance for BAW> us. It may not be worth it.
I think it would be worthwhile. Alternatively, you could cvs remove it, the add it to Modules/old with a note to check the Attic for older revision notes.
Skip
"SM" == Skip Montanaro skip@pobox.com writes:
BAW> Do we care about preserving the cvs history for the current BAW> bsddbmodule.c? If so, we'll have to ask SF to do a cvs dance BAW> for us. It may not be worth it.
SM> I think it would be worthwhile. Alternatively, you could cvs SM> remove it, the add it to Modules/old with a note to check the SM> Attic for older revision notes.
That would be fine with me. -Barry
"Gregory P. Smith" greg@electricrain.com writes:
Modern berkeleydb uses much different on disk database formats, glancing at the docs on sleepycat.com i don't even think it can read bsddb (1.85) files. Existing code using bsddb (1.85) should not automatically start using a different database library even if we provide a compatibility interface.
The Python bsddb module never guaranteed that you can use it to read bsddb 1.85 data files. In fact, on many installation, the bsddb module links with bsddb 2.x or bsddb 3.x, using db_185.h.
So this is no reason not to call the module bsddb.
Regards, Martin
Skip Montanaro skip@pobox.com writes:
Why can't it just be called bsddb?
If full compatibility is guaranteed, I'm all for it.
Regards, Martin
"SM" == Skip Montanaro skip@pobox.com writes:
SM> If you build the bsddb module on a Unix-like system (that is, SM> you use configure and setup.py to build the interpreter and it SM> attempts to build the bsddb module), please give the new patch SM> attached to
SM> http://python.org/sf/553108
Skip,
Apologies for taking so long to respond to this thread. I'm still digging out from my move.
Basically what you have in cvs works great, except for one small necessary addition. If you build Berkeley DB from source, it's going to install it in something like /usr/local/BerkeleyDB.3.3 by default. Why they choose such a bizarre location, I don't know.
The problem is that unless your sysadmin hacks ld.so.conf to add /usr/local/BerkeleyDB.X.Y/lib onto your standard ld run path, bsddbmodule.so won't be linked in such a way that it can actually resolve the symbols at run time. I don't think it's reasonable to require such system hacking to get the bsddb module to link properly, and I think we can do better.
Here's a small patch to setup.py which should fix things in a portable way, at least for *nix systems. It sets the envar LD_RUN_PATH to the location that it found the Berkeley library, but only if that envar isn't already set. I've tested this on RH7.3 and MD8.1 -- all of which I have a from-source install of BerkeleyDB 3.3.11. Seems to work well for me.
I'm still having build trouble on my RH6.1 system, but maybe it's just too old to worry about (I /really/ need to upgrade one of these days ;).
-------------------- snip snip -------------------- building 'bsddb' extension gcc -g -Wall -Wstrict-prototypes -fPIC -DHAVE_DB_185_H=1 -I/usr/local/BerkeleyDB.3.3/include -I. -I/home/barry/projects/python/./Include -I/usr/local/include -I/home/barry/projects/python/Include -I/home/barry/projects/python -c /home/barry/projects/python/Modules/bsddbmodule.c -o build/temp.linux-i686-2.3/bsddbmodule.o In file included from /home/barry/projects/python/Modules/bsddbmodule.c:25: /usr/local/BerkeleyDB.3.3/include/db_185.h:171: parse error before `*' /usr/local/BerkeleyDB.3.3/include/db_185.h:171: warning: type defaults to `int' in declaration of `__db185_open' /usr/local/BerkeleyDB.3.3/include/db_185.h:171: warning: data definition has no type or storage class /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbhashobject': /home/barry/projects/python/Modules/bsddbmodule.c:74: warning: assignment from incompatible pointer type /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbbtobject': /home/barry/projects/python/Modules/bsddbmodule.c:124: warning: assignment from incompatible pointer type /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbrnobject': /home/barry/projects/python/Modules/bsddbmodule.c:182: warning: assignment from incompatible pointer type -------------------- snip snip --------------------
Sigh.
Anyway thanks! You've improved the situation immensely. -Barry
-------------------- snip snip -------------------- Index: setup.py =================================================================== RCS file: /cvsroot/python/python/dist/src/setup.py,v retrieving revision 1.94 diff -u -r1.94 setup.py --- setup.py 17 Jun 2002 17:55:30 -0000 1.94 +++ setup.py 19 Jun 2002 01:01:19 -0000 @@ -507,6 +507,13 @@ dblibs = [dblib] raise found except found: + # A default source build puts Berkeley DB in something like + # /usr/local/Berkeley.3.3 and the lib dir under that isn't + # normally on LD_RUN_PATH, unless the sysadmin has hacked + # /etc/ld.so.conf. Setting the envar should be portable across + # compilers and platforms. + if 'LD_RUN_PATH' not in os.environ: + os.environ['LD_RUN_PATH'] = dblib_dir if dbinc == 'db_185.h': exts.append(Extension('bsddb', ['bsddbmodule.c'], library_dirs=[dblib_dir],
* Barry A. Warsaw barry@zope.com [2002-06-18 22:34 -0400]:
The problem is that unless your sysadmin hacks ld.so.conf to add /usr/local/BerkeleyDB.X.Y/lib onto your standard ld run path, bsddbmodule.so won't be linked in such a way that it can actually resolve the symbols at run time. [...] os.environ['LD_RUN_PATH'] = dblib_dir
I may be missing something here, but AFAIC that's what the library_dirs parameter in the Extension constructor of distutils is for. It basically sets the runtime library path at compile time using the "-R" linker option.
Gerhard
"GH" == Gerhard Häring gerhard@bigfoot.de writes:
GH> * Barry A. Warsaw barry@zope.com [2002-06-18 22:34 -0400]: >> The problem is that unless your sysadmin hacks ld.so.conf to >> add /usr/local/BerkeleyDB.X.Y/lib onto your standard ld run >> path, bsddbmodule.so won't be linked in such a way that it can >> actually resolve the symbols at run time. [...] >> os.environ['LD_RUN_PATH'] = dblib_dir
GH> I may be missing something here, but AFAIC that's what the GH> library_dirs parameter in the Extension constructor of GH> distutils is for. It basically sets the runtime library path GH> at compile time using the "-R" linker option.
Possibly (Greg?), but without setting that envar (or doing a less portable -Xlinker -Rblah trick) I'd end up with a build/lib.linux-i686-2.3/bsddb_failed.so which, if you ran ldd over it would show no resolution for libdb-3.3.so. Also, no -R or equivalent option showed up in the compilation output.
-Barry
On 19 June 2002, Gerhard H?ring said:
- Barry A. Warsaw barry@zope.com [2002-06-18 22:34 -0400]:
The problem is that unless your sysadmin hacks ld.so.conf to add /usr/local/BerkeleyDB.X.Y/lib onto your standard ld run path, bsddbmodule.so won't be linked in such a way that it can actually resolve the symbols at run time. [...] os.environ['LD_RUN_PATH'] = dblib_dir
I may be missing something here, but AFAIC that's what the library_dirs parameter in the Extension constructor of distutils is for. It basically sets the runtime library path at compile time using the "-R" linker option.
No, library_dirs is for good old -L. AFAIK it works fine.
For -R (or equivalent) you need runtime_library_dirs. I'm not sure if it works (or ever did). I think it's a question of knowing what magic options to supply to each compiler. Probably it works (worked) on Solaris, since for once Sun got things right and supplied a simple, obvious, working command-line option -- namely -R.
Greg
"GW" == Greg Ward gward@python.net writes:
GW> No, library_dirs is for good old -L. AFAIK it works fine.
GW> For -R (or equivalent) you need runtime_library_dirs. I'm not GW> sure if it works (or ever did). I think it's a question of GW> knowing what magic options to supply to each compiler. GW> Probably it works (worked) on Solaris, since for once Sun got GW> things right and supplied a simple, obvious, working GW> command-line option -- namely -R.
runtime_library_dirs works perfectly for Linux and gcc, thanks.
-Barry
barry@zope.com (Barry A. Warsaw) writes:
Basically what you have in cvs works great, except for one small necessary addition. If you build Berkeley DB from source, it's going to install it in something like /usr/local/BerkeleyDB.3.3 by default. Why they choose such a bizarre location, I don't know.
The problem is that unless your sysadmin hacks ld.so.conf to add /usr/local/BerkeleyDB.X.Y/lib onto your standard ld run path, bsddbmodule.so won't be linked in such a way that it can actually resolve the symbols at run time. I don't think it's reasonable to require such system hacking to get the bsddb module to link properly, and I think we can do better.
Here's a small patch to setup.py which should fix things in a portable way, at least for *nix systems. It sets the envar LD_RUN_PATH to the location that it found the Berkeley library, but only if that envar isn't already set.
I dislike that change. Setting LD_RUN_PATH is the jobs of whoever is building the compiler, and should not be done by Python automatically. So far, the Python build process avoids adding any -R linker options, since it requires quite some insight into the specific installation to determine whether usage of that option is the right thing.
If setup.py fails to build an extension correctly, it is the adminstrator's job to specify a correct build procedure in Modules/Setup. For that reason, I rather recommend to remove the magic that setup.py looks in /usr/local/Berkeley*, instead of adding more magic.
Regards, Martin
On Wed, Jun 19, 2002, Martin v. Loewis wrote:
barry@zope.com (Barry A. Warsaw) writes:
Here's a small patch to setup.py which should fix things in a portable way, at least for *nix systems. It sets the envar LD_RUN_PATH to the location that it found the Berkeley library, but only if that envar isn't already set.
I dislike that change. Setting LD_RUN_PATH is the jobs of whoever is building the compiler, and should not be done by Python automatically. So far, the Python build process avoids adding any -R linker options, since it requires quite some insight into the specific installation to determine whether usage of that option is the right thing.
If setup.py fails to build an extension correctly, it is the adminstrator's job to specify a correct build procedure in Modules/Setup. For that reason, I rather recommend to remove the magic that setup.py looks in /usr/local/Berkeley*, instead of adding more magic.
-1 if it doesn't at least include an error message saying that we found dbm but couldn't use it. (That is, I agree with you that explicit is better than implicit -- but if we can provide info, we should.)
"MvL" == Martin v Loewis martin@v.loewis.de writes:
MvL> I dislike that change. Setting LD_RUN_PATH is the jobs of MvL> whoever is building the compiler, and should not be done by MvL> Python automatically. So far, the Python build process avoids MvL> adding any -R linker options, since it requires quite some MvL> insight into the specific installation to determine whether MvL> usage of that option is the right thing.
Really? You know the path for the -R/--rpath flag, so all you need is the magic compiler-specific incantation, and distutils already (or /should/ already) know that.
MvL> If setup.py fails to build an extension correctly, it is the MvL> adminstrator's job to specify a correct build procedure in MvL> Modules/Setup. For that reason, I rather recommend to remove MvL> the magic that setup.py looks in /usr/local/Berkeley*, MvL> instead of adding more magic.
I disagree. While the sysadmin should probably fiddle with /etc/ld.so.conf when he installs BerkeleyDB, it's not documented in the Sleepycat docs, so it's entirely possible that they haven't done it. That shouldn't stop Python from building a perfectly usable module, especially because it really can figure out all the necessary information.
Is there some specific fear you have about compiling in the run-path?
Note I'm not saying setting LD_RUN_PATH is the best approach, but it seemed like the most portable. I couldn't figure out if distutils knew what the right compiler-specific switches are (i.e. "-R dir" on Solaris cc if memory serves, and "-Xlinker -rpath -Xlinker dir" for gcc, and who knows what for other Unix or <gasp> Windows compilers).
-Barry
barry@zope.com (Barry A. Warsaw) writes:
Really? You know the path for the -R/--rpath flag, so all you need is the magic compiler-specific incantation, and distutils already (or /should/ already) know that.
Yes, but you don't know whether usage of -R is appropriate. If the installed library is static, -R won't be needed. If then the target directory recorded with -R happens to be on an unavailable NFS server at run-time (on a completely different network), you cannot import the library module anymore, which would otherwise work perfectly fine.
We had big problems with recorded library directories over the years; at some point, the administrators decided to take the machine that had /usr/local/lib/gcc-lib/sparc-sun-solaris2.3/2.5.8 on it offline. They did not knew that they would thus make vim inoperable, which happened to be compiled with LD_RUN_PATH pointing to that directory - even though no library was ever needed from that directory.
I disagree. While the sysadmin should probably fiddle with /etc/ld.so.conf when he installs BerkeleyDB, it's not documented in the Sleepycat docs, so it's entirely possible that they haven't done it.
I'm not asking for the administrator fiddle with ld.so.conf. Instead, I'm asking the administrator fiddle with Modules/Setup.
Is there some specific fear you have about compiling in the run-path?
Yes, see above.
Note I'm not saying setting LD_RUN_PATH is the best approach, but it seemed like the most portable. I couldn't figure out if distutils knew what the right compiler-specific switches are (i.e. "-R dir" on Solaris cc if memory serves, and "-Xlinker -rpath -Xlinker dir" for gcc, and who knows what for other Unix or <gasp> Windows compilers).
LD_LIBRARY_PATH won't work for Windows compilers, either. To my knowledge, there is nothign equivalent on Windows.
Regards, Martin
"MvL" == Martin v Loewis martin@v.loewis.de writes:
MvL> barry@zope.com (Barry A. Warsaw) writes:
>> Really? You know the path for the -R/--rpath flag, so all you >> need is the magic compiler-specific incantation, and distutils >> already (or /should/ already) know that.
MvL> Yes, but you don't know whether usage of -R is appropriate. MvL> If the installed library is static, -R won't be needed.
And shouldn't hurt.
MvL> If then the target directory recorded with -R happens to be MvL> on an unavailable NFS server at run-time (on a completely MvL> different network), you cannot import the library module MvL> anymore, which would otherwise work perfectly fine.
Do people still use NFS servers to share programs? I thought big cheap disks and RPMs did away with all that. :)
I believe that -R/-rpath adds directories to runtime search paths so if the NFS directory was unmounted, ld.so should still be able to locate the shared library through fallback means. That may fail too, but oh well.
One issue on Solaris may be that -- according to the GNU ld docs -- the runtime search path will be built from the -L options which we're already passing, /unless/ -rpath is given, and this seems to be added to help with NFS mounted directories on the -L specified path. But since I'm proposing that the -rpath directory be the same as the -L path, I don't think it will make matters worse.
MvL> We had big problems with recorded library directories over MvL> the years; at some point, the administrators decided to take MvL> the machine that had MvL> /usr/local/lib/gcc-lib/sparc-sun-solaris2.3/2.5.8 on it MvL> offline. They did not knew that they would thus make vim MvL> inoperable, which happened to be compiled with LD_RUN_PATH MvL> pointing to that directory - even though no library was ever MvL> needed from that directory.
Hmm. Was the problem that the NFS server was unresponsive, or that the directory was unmounted, but still searched? If the former, then maybe you do have a problem. I've experienced hangs over the years when NFS servers have been unresponsive (because the host was down and the nfs mounts options weren't given to make this a soft error). I haven't used NFS in years though so my memory is rusty on the details.
>> I disagree. While the sysadmin should probably fiddle with >> /etc/ld.so.conf when he installs BerkeleyDB, it's not >> documented in the Sleepycat docs, so it's entirely possible >> that they haven't done it.
MvL> I'm not asking for the administrator fiddle with MvL> ld.so.conf. Instead, I'm asking the administrator fiddle with MvL> Modules/Setup.
We've made it so easy to build a batteries-included Python that I think it would be unfortunately not to do better just because we fear that things /might/ go wrong in some strange environments. I think it's largely unnecessary to edit Modules/Setup these days, and since we /know/ that BerkeleyDB is installed in a funky location not usually on your ld.so path, I think we can take advantage of that to not require editing Modules/Setup in this case too.
Our failure mode for bsddbmodule is so cryptic that it's very difficult to figure out why it's not available. I think this simple change to setup.py[1] would improve the life for the average Python programmer. I'd be happy with a command line switch or envar to disable the -R/--rpath addition.
Here's a compromise. If LD_RUN_PATH is set at all (regardless of value), don't add -R/--rpath. Or add a --without-rpath switch to configure.
>> Note I'm not saying setting LD_RUN_PATH is the best approach, >> but it seemed like the most portable. I couldn't figure out if >> distutils knew what the right compiler-specific switches are >> (i.e. "-R dir" on Solaris cc if memory serves, and "-Xlinker >> -rpath -Xlinker dir" for gcc, and who knows what for other Unix >> or <gasp> Windows compilers).
MvL> LD_LIBRARY_PATH won't work for Windows compilers, either. To MvL> my knowledge, there is nothign equivalent on Windows.
Someone else will have to figure out the problems for Windows source builders <wink>. I'd like to make life just a little easier for Linux and Unix users. I think this change will do that.
-Barry
[1]
Index: setup.py =================================================================== RCS file: /cvsroot/python/python/dist/src/setup.py,v retrieving revision 1.95 diff -u -r1.95 setup.py --- setup.py 21 Jun 2002 14:48:38 -0000 1.95 +++ setup.py 24 Jun 2002 18:03:06 -0000 @@ -510,12 +510,14 @@ if dbinc == 'db_185.h': exts.append(Extension('bsddb', ['bsddbmodule.c'], library_dirs=[dblib_dir], + runtime_library_dirs=[dblib_dir], include_dirs=db_incs, define_macros=[('HAVE_DB_185_H',1)], libraries=[dblib])) else: exts.append(Extension('bsddb', ['bsddbmodule.c'], library_dirs=[dblib_dir], + runtime_library_dirs=[dblib_dir], include_dirs=db_incs, libraries=[dblib])) else:
barry@zope.com (Barry A. Warsaw) writes:
MvL> If then the target directory recorded with -R happens to be MvL> on an unavailable NFS server at run-time (on a completely MvL> different network), you cannot import the library module MvL> anymore, which would otherwise work perfectly fine.Do people still use NFS servers to share programs? I thought big cheap disks and RPMs did away with all that. :)
This was on Solaris, so no RPMs.
I believe that -R/-rpath adds directories to runtime search paths so if the NFS directory was unmounted, ld.so should still be able to locate the shared library through fallback means. That may fail too, but oh well.
Yes, but the startup time for the program increases dramatically - it has to wait for the dead NFS server to timeout.
One issue on Solaris may be that -- according to the GNU ld docs -- the runtime search path will be built from the -L options which we're already passing, /unless/ -rpath is given, and this seems to be added
Where do the docs say that? I don't think this is the case, or ever was ...
to help with NFS mounted directories on the -L specified path. But since I'm proposing that the -rpath directory be the same as the -L path, I don't think it will make matters worse.
Indeed, it wouldn't.
Hmm. Was the problem that the NFS server was unresponsive, or that the directory was unmounted, but still searched? If the former, then maybe you do have a problem.
Yes, that was the problem. Even with soft mounting, it will still take time to timeout.
We've made it so easy to build a batteries-included Python that I think it would be unfortunately not to do better just because we fear that things /might/ go wrong in some strange environments.
That is a reasonable argument, and I've been giving similar arguments in other cases, too, so I guess I should just stop complaining.
Here's a compromise. If LD_RUN_PATH is set at all (regardless of value), don't add -R/--rpath. Or add a --without-rpath switch to configure.
I guess we don't need to compromise, and approach is *very* cryptic, so I'd rather avoid it.
It looks like the current bsddb module is going to go away, anyway, so there is no need to tweak the current configuration that much. I don't know what the bsddb3 build procedure is, but any approach you come up with now probably needs to be redone after pybsddb3 integration.
Regards, Martin
"MvL" == Martin v Loewis martin@v.loewis.de writes:
>> Do people still use NFS servers to share programs? I thought >> big cheap disks and RPMs did away with all that. :)
MvL> This was on Solaris, so no RPMs.
I know, I was kind of joking. But even Solaris has pkg, though I don't know if it's in nearly as widespread use as Linux packages.
>> I believe that -R/-rpath adds directories to runtime search >> paths so if the NFS directory was unmounted, ld.so should still >> be able to locate the shared library through fallback means. >> That may fail too, but oh well.
MvL> Yes, but the startup time for the program increases MvL> dramatically - it has to wait for the dead NFS server to MvL> timeout.
Yeah that would suck. I wonder if that would only affect imports of bsddb though since the Python executable itself wouldn't be linked w/-R.
>> One issue on Solaris may be that -- according to the GNU ld >> docs -- the runtime search path will be built from the -L >> options which we're already passing, /unless/ -rpath is given, >> and this seems to be added
MvL> Where do the docs say that? I don't think this is the case, MvL> or ever was ...
It's in the GNU ld info page under Options:
`-rpath DIR' [...]
The `-rpath' option may also be used on SunOS. By default, on SunOS, the linker will form a runtime search patch out of all the `-L' options it is given. If a `-rpath' option is used, the runtime search path will be formed exclusively using the `-rpath' options, ignoring the `-L' options. This can be useful when using gcc, which adds many `-L' options which may be on NFS mounted filesystems.
Reading it again now, it's not clear if "SunOS" also means "Solaris".
>> We've made it so easy to build a batteries-included Python that >> I think it would be unfortunately not to do better just because >> we fear that things /might/ go wrong in some strange >> environments.
MvL> That is a reasonable argument, and I've been giving similar MvL> arguments in other cases, too, so I guess I should just stop MvL> complaining.
>> Here's a compromise. If LD_RUN_PATH is set at all (regardless >> of value), don't add -R/--rpath. Or add a --without-rpath >> switch to configure.
MvL> I guess we don't need to compromise, and approach is *very* MvL> cryptic, so I'd rather avoid it.
Cool. I'll commit the change.
MvL> It looks like the current bsddb module is going to go away, MvL> anyway, so there is no need to tweak the current MvL> configuration that much. I don't know what the bsddb3 build MvL> procedure is, but any approach you come up with now probably MvL> needs to be redone after pybsddb3 integration.
I suspect we'll need /something/ like this once pybsddb's integrated, but I'll definitely be testing it once Greg does the integration. I doubt pybsddb's build process is going to just drop into place, and I suspect it'll actually be easier.
Thanks, -Barry
barry@zope.com (Barry A. Warsaw) writes:
`-rpath DIR' [...]
The `-rpath' option may also be used on SunOS. By default, on SunOS, the linker will form a runtime search patch out of all the `-L' options it is given. If a `-rpath' option is used, the runtime search path will be formed exclusively using the `-rpath' options, ignoring the `-L' options. This can be useful when using gcc, which adds many `-L' options which may be on NFS mounted filesystems.Reading it again now, it's not clear if "SunOS" also means "Solaris".
I see. This is indeed SunOS 4 only.
Regards, Martin
Just a quick note to let you all know you've completely lost me with all this -R stuff. If someone would like to implement this, the now-closed patch is at
Just reopen it and assign it to yourself. A quick summary of this thread added to the bug report would probably be a good idea.
Skip
"SM" == Skip Montanaro skip@pobox.com writes:
SM> Just a quick note to let you all know you've completely lost SM> me with all this -R stuff. If someone would like to implement SM> this, the now-closed patch is at
SM> http://python.org/sf/553108
SM> Just reopen it and assign it to yourself. A quick summary of SM> this thread added to the bug report would probably be a good SM> idea.
Actually, I think we're now good to go, although we'll need to revisit this once Greg starts w/ the integration of pybsddb.
Thanks Skip, -Barry
Barry> Here's a small patch to setup.py which should fix things in a Barry> portable way, at least for *nix systems. It sets the envar Barry> LD_RUN_PATH to the location that it found the Berkeley library, Barry> but only if that envar isn't already set.
Martin> I dislike that change. Setting LD_RUN_PATH is the jobs of Martin> whoever is building the compiler, and should not be done by Martin> Python automatically.
Agreed. Also, is LD_RUN_PATH widely available?
Martin> If setup.py fails to build an extension correctly, it is the Martin> adminstrator's job to specify a correct build procedure in Martin> Modules/Setup. For that reason, I rather recommend to remove the Martin> magic that setup.py looks in /usr/local/Berkeley*, instead of Martin> adding more magic.
I'm happy with the current setup. While the /usr/local/BerkeleyN.M location is a bit odd, Sleepycat is pretty consistent in this regard. (At least versions 3 and 4 install this way.) I'd rather require sysadmins to run ldconfig or its equivalent.
Most of the time people install packages using the default locations. In those situations where they don't, distutils accepts a couple environment variables which specific alternate search directories for libraries and include files. Their names escape me at the moment, and I'm not sure they accept the usual colon-separated list of directories. If they don't, they should be suitably modified. It should probably be easy to specify these through some configure command line args.
Skip
BAW> I'm still having build trouble on my RH6.1 system, but maybe it's BAW> just too old to worry about (I /really/ need to upgrade one of BAW> these days BAW> ;).
BAW> -------------------- snip snip -------------------- BAW> building 'bsddb' extension BAW> gcc -g -Wall -Wstrict-prototypes -fPIC -DHAVE_DB_185_H=1 -I/usr/local/BerkeleyDB.3.3/include -I. -I/home/barry/projects/python/./Include -I/usr/local/include -I/home/barry/projects/python/Include -I/home/barry/projects/python -c /home/barry/projects/python/Modules/bsddbmodule.c -o build/temp.linux-i686-2.3/bsddbmodule.o BAW> In file included from /home/barry/projects/python/Modules/bsddbmodule.c:25: BAW> /usr/local/BerkeleyDB.3.3/include/db_185.h:171: parse error before `*' BAW> /usr/local/BerkeleyDB.3.3/include/db_185.h:171: warning: type defaults to `int' in declaration of `__db185_open' BAW> /usr/local/BerkeleyDB.3.3/include/db_185.h:171: warning: data definition has no type or storage class BAW> /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbhashobject': BAW> /home/barry/projects/python/Modules/bsddbmodule.c:74: warning: assignment from incompatible pointer type BAW> /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbbtobject': BAW> /home/barry/projects/python/Modules/bsddbmodule.c:124: warning: assignment from incompatible pointer type BAW> /home/barry/projects/python/Modules/bsddbmodule.c: In function `newdbrnobject': BAW> /home/barry/projects/python/Modules/bsddbmodule.c:182: warning: assignment from incompatible pointer type BAW> -------------------- snip snip --------------------
I think you might have to define another CPP macro. In my post from last night about building dbmmodule.c I included
define_macros=[('HAVE_BERKDB_H',None), ('DB_DBM_HSEARCH',None)],
in the Extension constructor. Maybe DB_DBM_HSEARCH is also needed for older bsddb? I have no trouble building though.
Skip
"SM" == Skip Montanaro skip@pobox.com writes:
BAW> I'm still having build trouble on my RH6.1 system, but maybe BAW> it's just too old to worry about (I /really/ need to upgrade BAW> one of these days ;).
[errors snipped]
SM> I think you might have to define another CPP macro. In my SM> post from last night about building dbmmodule.c I included
| define_macros=[('HAVE_BERKDB_H',None), | ('DB_DBM_HSEARCH',None)],
SM> in the Extension constructor. Maybe DB_DBM_HSEARCH is also SM> needed for older bsddb? I have no trouble building though.
It must be an issue with this ancient RH6.1 system. It builds fine on Mandrake 8.1, and the time to dig into this would probably be better spent upgrading to a more modern Linux distro.
But thanks, -Barry
participants (11)
-
Aahz -
barry＠zope.com -
Gerhard Häring -
Gerhard Häring -
Greg Ward -
Gregory P. Smith -
Guido van Rossum -
martin＠v.loewis.de -
Oleg Broytmann -
Skip Montanaro -
Thomas Heller