Committing bsddb 4.6.4, and where can I put testsuite temp files?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just committed pybsddb 4.6.4 to python svn. My next step (after a successfull buildbot cycle, I hope!) is to commit the new testsuite. First I need to review any changes there since I maintain pybsddb. The testsuite creates a lot of files/directories while working. Fine, since it cleans later, unless some test crashes hard. My testcode defines a variable to the path I want to use to keep those files/dirs. Currently it is "/tmp/z-BerkeleyDB", or something like that. Since my code allows for testcases to be run in multiple threads and/or processes, where would be a sensible path choice for temporal files?. Current working directory would be fine, if "somebody" is cleaning after running all tests. Ramdisk ("/tmp" in some systems, for example Solaris) would be nice, since some test uses transactions, and transactions are write-synchronous. In my system, runnning all tests serially (you can run all of them in parallel, if you wish) takes about 15-20 seconds. Far faster than current tests in python svn, and that can be improved even more. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSCnngZlgi5GaxT1NAQLhPwP9GMMyNMgz6mfmfHXVOV8bDJGOQRok95uL 14A+K9zXW3/wlp1rhvoPmCHYqvRoCLVkPZ/7pLEQlo1ZksOlHy6BH3MDeDJEjVax 69XlzUUeuqplGbTiMdX8qd0dPi2Jp+Akg7U6ZmBdADhF7D21umU474OalKO2eIoL ba/wnlMvens= =1lai -----END PGP SIGNATURE-----
On Tue, May 13, 2008 at 9:09 PM, Jesus Cea <jcea@jcea.es> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I just committed pybsddb 4.6.4 to python svn. My next step (after a successfull buildbot cycle, I hope!) is to commit the new testsuite. First I need to review any changes there since I maintain pybsddb.
The testsuite creates a lot of files/directories while working. Fine, since it cleans later, unless some test crashes hard. My testcode defines a variable to the path I want to use to keep those files/dirs. Currently it is "/tmp/z-BerkeleyDB", or something like that.
Since my code allows for testcases to be run in multiple threads and/or processes, where would be a sensible path choice for temporal files?. Current working directory would be fine, if "somebody" is cleaning after running all tests.
It is up to the tests to clean up after themselves since they can be executed directly, so there is not magical location that gets cleaned up in case a test screws up. You can use test.test_support.TESTFN if you want a location. Or something from tempfile. -Brett
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brett Cannon wrote: | It is up to the tests to clean up after themselves since they can be | executed directly, so there is not magical location that gets cleaned | up in case a test screws up. You can use test.test_support.TESTFN if | you want a location. Or something from tempfile. I'm worried about the case when a test crashes and leave garbage behind... that will crash future test runs. I guess I would delete the directory before using it, but I guess somebody already had this problem and solved it, somehow. :-? - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSCnvA5lgi5GaxT1NAQIM7wP9EEOsr+WYpQtA+0tbHTv247kplxzNDo7T WTIWU+pfs575gMSbK5NUCrr6TXuFsqlgybOaFqA3wTKHiKslcRhkBrmjtd/Wj4iF /bdv8C/SnLdM443ZnakWFnPAH+Y/horuRxKvsx01lNdaNtbADZG9bClOCfVBfoWM j/fUSb5RtAU= =GkQb -----END PGP SIGNATURE-----
On Tue, May 13, 2008 at 12:41 PM, Jesus Cea <jcea@jcea.es> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brett Cannon wrote: | It is up to the tests to clean up after themselves since they can be | executed directly, so there is not magical location that gets cleaned | up in case a test screws up. You can use test.test_support.TESTFN if | you want a location. Or something from tempfile.
I'm worried about the case when a test crashes and leave garbage behind... that will crash future test runs.
Then don't crash on tests. =)
I guess I would delete the directory before using it, but I guess somebody already had this problem and solved it, somehow. :-?
That's usually the other way to go about it. In your test_main() or your setUp() methods you make sure everything is clean before you start. -Brett
Brett Cannon wrote:
On Tue, May 13, 2008 at 12:41 PM, Jesus Cea <jcea@jcea.es> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brett Cannon wrote: | It is up to the tests to clean up after themselves since they can be | executed directly, so there is not magical location that gets cleaned | up in case a test screws up. You can use test.test_support.TESTFN if | you want a location. Or something from tempfile.
I'm worried about the case when a test crashes and leave garbage behind... that will crash future test runs.
Then don't crash on tests. =)
I guess I would delete the directory before using it, but I guess somebody already had this problem and solved it, somehow. :-?
That's usually the other way to go about it. In your test_main() or your setUp() methods you make sure everything is clean before you start.
Personally, I use the tempfile module to get a temporary directory and do anything I need in there (see test_runpy.py or test_cmdline_script.py for examples). That way, even if the test does manage to crash in a way that bypasses the shutil.rmtree cleanup call (which is actually fairly hard to do when the cleanup call is in a finally clause or a tearDown method), it won't affect subsequent executions (although it may accumulate clutter in the buildbot machine's temporary directories over time). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
btw, I fixed the Lib/test/test_bsddb3.py file for the updated Lib/bsddb/test/ modules. Thats how the test suite gets run by the buildbots (run the test suite from a python trunk sandbox using "./python Lib/test/regrtest.py -v -u bsddb test_bsddb test_bsddb3" to reproduce exactly how it is run yourself). I'm seeing what appears to be a memory corruption error when using a debug build of trunk (configure --with-pydebug). It causes a crash while deallocating the argument tuple passed to DBEnv.lock_get on the second call in bsddb.test.test_lock's test03_lock_timeout() test. Platform tested on: OS X 10.4, gcc 4.0.1 x86. BerkeleyDB 4.5. I haven't tested on other platforms but I expect other buildbots will also have issues now that the test suite isn't being skipped. -gps On Tue, May 13, 2008 at 12:09 PM, Jesus Cea <jcea@jcea.es> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I just committed pybsddb 4.6.4 to python svn. My next step (after a successfull buildbot cycle, I hope!) is to commit the new testsuite. First I need to review any changes there since I maintain pybsddb.
The testsuite creates a lot of files/directories while working. Fine, since it cleans later, unless some test crashes hard. My testcode defines a variable to the path I want to use to keep those files/dirs. Currently it is "/tmp/z-BerkeleyDB", or something like that.
Since my code allows for testcases to be run in multiple threads and/or processes, where would be a sensible path choice for temporal files?. Current working directory would be fine, if "somebody" is cleaning after running all tests.
Ramdisk ("/tmp" in some systems, for example Solaris) would be nice, since some test uses transactions, and transactions are write-synchronous.
In my system, runnning all tests serially (you can run all of them in parallel, if you wish) takes about 15-20 seconds. Far faster than current tests in python svn, and that can be improved even more.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iQCVAwUBSCnngZlgi5GaxT1NAQLhPwP9GMMyNMgz6mfmfHXVOV8bDJGOQRok95uL 14A+K9zXW3/wlp1rhvoPmCHYqvRoCLVkPZ/7pLEQlo1ZksOlHy6BH3MDeDJEjVax 69XlzUUeuqplGbTiMdX8qd0dPi2Jp+Akg7U6ZmBdADhF7D21umU474OalKO2eIoL ba/wnlMvens= =1lai -----END PGP SIGNATURE----- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gregory P. Smith wrote: | btw, I fixed the Lib/test/test_bsddb3.py file for the updated | Lib/bsddb/test/ modules. Thats how the test suite gets run by the | buildbots (run the test suite from a python trunk sandbox using | "./python Lib/test/regrtest.py -v -u bsddb test_bsddb test_bsddb3" to | reproduce exactly how it is run yourself). What about changing that file to read the testsuite from "Lib/bsddb/test", directly, to avoid skipping new tests again?. I would like to print the Berkeley DB release used, also. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSDW93Jlgi5GaxT1NAQKUcwP/ZOu3jOGEpGHmDGZYZbChowHnDYnpwh0E DgVbHnP3hWRneo5UESNSg6yu481jJ4eLHF662zAYbLlJx1J15yF8jMXHxWuW4fH1 QNVU4DwtIVGU9FdAKHcDqKglgvTtKy/0rzDPBcjkgmbjsM9MkxtOu8lZjBw1RgaB T4ojB+pr7Ps= =JU9d -----END PGP SIGNATURE-----
On Thu, May 22, 2008 at 1:32 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I would like to print the Berkeley DB release used, also.
Could that be done at build time as well?
Martin
I just added that to python's setup.py as well as made test_bsddb3 print the version to stderr when it gets run. -gps
Hi Jesus, Regarding where to place test files, you might want to look at how the current bsddb test suite in Lib/bsddb/test handles the need to create temporary files and such. A bit of work has gone into this particular aspect to improve reliability and robustness on buildbots -- particularily when it comes to Windows. (I believe the current test suite comes up with a unique directory based on process ID..) Trent. -----Original Message----- From: python-dev-bounces+tnelson=onresolve.com@python.org [mailto:python-dev-bounces+tnelson=onresolve.com@python.org] On Behalf Of Jesus Cea Sent: Tuesday, May 13, 2008 8:10 PM To: Python DEV Subject: [Python-Dev] Committing bsddb 4.6.4, and where can I put testsuite temp files? -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just committed pybsddb 4.6.4 to python svn. My next step (after a successfull buildbot cycle, I hope!) is to commit the new testsuite. First I need to review any changes there since I maintain pybsddb. The testsuite creates a lot of files/directories while working. Fine, since it cleans later, unless some test crashes hard. My testcode defines a variable to the path I want to use to keep those files/dirs. Currently it is "/tmp/z-BerkeleyDB", or something like that. Since my code allows for testcases to be run in multiple threads and/or processes, where would be a sensible path choice for temporal files?. Current working directory would be fine, if "somebody" is cleaning after running all tests. Ramdisk ("/tmp" in some systems, for example Solaris) would be nice, since some test uses transactions, and transactions are write-synchronous. In my system, runnning all tests serially (you can run all of them in parallel, if you wish) takes about 15-20 seconds. Far faster than current tests in python svn, and that can be improved even more. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSCnngZlgi5GaxT1NAQLhPwP9GMMyNMgz6mfmfHXVOV8bDJGOQRok95uL 14A+K9zXW3/wlp1rhvoPmCHYqvRoCLVkPZ/7pLEQlo1ZksOlHy6BH3MDeDJEjVax 69XlzUUeuqplGbTiMdX8qd0dPi2Jp+Akg7U6ZmBdADhF7D21umU474OalKO2eIoL ba/wnlMvens= =1lai -----END PGP SIGNATURE----- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com
participants (6)
-
"Martin v. Löwis" -
Brett Cannon -
Gregory P. Smith -
Jesus Cea -
Nick Coghlan -
Trent Nelson